Incorporating cybersecurity into water utility master planning — Part 1

Cybersecurity threats are ubiquitous and far-reaching. But the stakes are highest when the threats impact critical infrastructure, including water systems.

Yahoo. The Democratic National Committee. Ukraine’s power grid. Significant cybersecurity breaches unwittingly kept each of these organisations in the headlines for weeks at a time in recent years. And after the headlines had faded, these companies faced a damaged reputation and an aftermath marked by fading consumer confidence and more than a few class action lawsuits.

While these breaches have put consumers on alert, tactical campaigns targeting industrial applications and critical infrastructure such as Stuxnet, BlackEnergy and Havex have heightened governmental attention and resulted in additional pressure to protect vital systems.

For those safeguarding and distributing the public’s water supply, these massive security breaches and threats are yet another reminder of the potential vulnerability of their systems. Due to limited budgets, uptime requirements and talent shortfalls, many utilities struggle to apply even basic security measures. The latest call to strengthen system security to meet the most advanced threats has added a new level of confusion and concern.

Historically, water utilities embraced the IT landscape later than many industries. Many are just now beginning to realise significant benefits. Most have incorporated consumer online access to account information. Others have added smart metering technologies, including advanced metering infrastructures (AMIs). Still others have built SCADA systems that can potentially leverage cloud and mobile technology.

But while the benefits of new technological capabilities are readily apparent, the associated risks often remain undetected and unchecked until a major breach occurs. Recognising the catastrophic impact such a breach could have within critical infrastructure, governments and security communities around the world have stepped up research, leadership, training and guidance.

In the US, Executive Order 13636 Improving Critical Infrastructure Cybersecurity¹, issued in 2013, was followed by the National Institute for Standards and Technology (NIST) Cybersecurity Framework² in 2014. This voluntary framework consists of referenced standards, guidelines and practices to promote the protection of critical infrastructure. Simultaneously, the American Water Works Association (AWWA) released its own Process Control System Security Guidance for the Water Sector³ to provide a sector-specific approach to adopting the NIST framework.

Simply put, there is no lack of practical guidance and tools — and independent experts and suppliers eager to help. As with any security framework, program or solution, implementation can be daunting for water system professionals, especially those in public utilities with limited IT staff and resources. This article outlines the most dangerous cyber threats to utility control systems and why an active defence strategy is often the most practical and effective response. It also includes steps utilities can take now to prepare for the inevitable day when they are faced with a new security challenge or regulatory requirement.

Unique water sector challenges: mission, infrastructure and expertise

Water utilities face unique challenges when addressing cybersecurity issues. Unlike many organisations, water utilities are usually publicly funded and accountable to the community at a very local level. Their mission is to provide a clean, uninterrupted water supply to their state, city or municipality — and to do so within budgets that are often politically charged.

Ageing distribution networks

It’s no secret that the water infrastructure in many places around the world is both deteriorating and in need of expansion to match population shifts. Upgrades are costly and time-intensive.

For municipalities of all sizes, repairing and updating water mains, reservoirs and pumping stations is an imperative — and a growing, oftentimes unpredictable, expense. Given limited budgets and resources, utilities must make tough choices. And in many cases, that means devoting the majority of available funds and personnel hours to maintaining the existing water distribution system.

Conversely, emerging technologies and methods are available that provide new capabilities to improve and extend the life and service level of the infrastructure. But there is often limited budget and staff available to evaluate and deploy them.

Disparate industrial control and information systems

Of course, utilities still must attend to their industrial control systems (ICS) and IT infrastructures to meet other critical industry requirements:

• Maintain and optimise increasingly complex process control and SCADA systems — and develop ways to minimise vendor and control system sprawl.
• Establish proper risk and use evaluation for new technology to minimise negative exposure and impact to future budgets.
• Improve consumer trust — and satisfy growing demand for more transparency and access to account information.
• Meet the inevitable demands of regulatory compliance.
   

But with internal resources committed to maintaining system availability, many utilities are forced to postpone comprehensive upgrades. More typically, utilities evolve their systems slowly — and rely increasingly on multiple service level agreements (SLAs), system integrators and contractors for control system expertise.

Outsourcing has its advantages. However, often one unintended consequence is a ‘silo’ approach to the ICS and IT infrastructure and disparate systems. Without appropriate oversight, this can result in a fragmented environment that, by its very nature, is more susceptible to cyber risk.

Escalating threats, increasing sophistication

Across all industries, cyber threats are escalating, both in number and sophistication.

While the private sector accounts for the majority of threats — and power generation receives most of the attention in the utility space — the water sector is equally vulnerable. Although many data breaches are the result of accidental insider activity, the source of the initial breach provides little consolation if it opens the door to malicious actions.

More sophisticated. More persistent. More dangerous. Perhaps of even more concern than the accelerating number of cyber threats is the nature of those threats. Today, hostile entities are applying sophisticated, orchestrated methods and multiple technologies to stay one step ahead of the IT professionals who implement safeguards to foil them. Among the most dangerous attacks are those orchestrated by advanced persistent threats (APTs). Once nearly the exclusive purvey of nation states seeking data for political or other strategic gain, evidence suggests that APTs are now being found in critical systems on which citizens depend. Most important to the success of an APT or any sophisticated threat actor is its ability to remain undetected for as long as possible. Therefore, a successful breach does not begin with, nor may it ever culminate in, mass destruction. Instead, it relies on a covert progression of activities which can be masked by the common noise of a typical network environment. Sophisticated malicious actors often begin innocuously enough — with information gathering or ‘Google hacking’ via public-facing websites and social media. While the initial activity appears benign, this passive reconnaissance phase quietly identifies possible vulnerabilities in the system.

How can cyber events affect water systems? Cyber events can affect water system operations in a variety of ways, some with potentially significant adverse effects on public health. For example: Make unauthorised changes to programmed instructions to take control of water distribution or wastewater collection systems — resulting in disabled service, reduced pressure or overflow of untreated sewage. Block data or send false information to operators. Change alarm thresholds or disable them. Prevent access to account information. Interfere with treatment equipment and potentially impact all downstream who are dependent on clean water — from private citizens to industry.

Next, the attack moves to active reconnaissance. Now, the attacker deploys a variety of external probing and scanning activities — perhaps including phishing and social media mingling — to acquire sensitive information, such as usernames and passwords. Once inside the system, the attack exploits vulnerabilities utilising sophisticated software tools. These attack vectors take many forms and are custom tailored to the targeted environment. In some cases, these tools may identify the presence of zero-day vulnerabilities. These unknown software weaknesses enable further infiltration until detected and patched. The goal of exploitation is to establish some level of command and control. Once a beachhead is established, additional reconnaissance activity focuses on locating sensitive data within the system — and then transferring it out of the network for malicious purposes. Since the exfiltration of data can resemble normal network traffic, it’s very difficult to detect. At this point, the damage is done.

Impervious to traditional countermeasures

For years, cybersecurity programs have centred on network isolation and segmentation — and passive defence activities — designed to mitigate system vulnerability. Simply put, passive defences are systems that do not require human intervention. These standard countermeasures are important components of any network security program and include antivirus software, security patches, signature-based intrusion detection systems, email filters and firewalls.

In recent years, water utilities have recognised the importance of installing, improving and keeping these systems up to date. In addition, governments and the security community have supported this approach by instituting standards written to help ensure minimal levels of security in various business sectors. As essential as these countermeasures are, they often leave systems susceptible — or even defenceless — against APTs and other sophisticated, targeted attacks. Here are a few reasons why:

“Our current efforts, geared towards ‘passive’ cyber defence, are fixated on continuously monitoring and patching systems. Passive defence does not work and will never work against serious cyber threats.”4 — Steven Chabinsky, Former Deputy Assistant Director, FBI Cyber Division

• Anti-virus software only protects systems from malware signatures it recognises.
• Malicious attackers use encryption, DNS tunnelling, email and other covert techniques to avoid detection by intrusion detection systems.
• Email filters struggle to stop correspondence that in every way appears legitimate.
   

Given today’s climate, any organisation that claims traditional countermeasures are enough to keep their systems secure is naive at best. Advanced threats are real and occurring regularly across every business sector — including water and wastewater.

Moving towards active cyber defence

While passive defences still have a place in warding off low-level attacks, an agile and active defence strategy is required to stay ahead of the most advanced adversaries. At the highest level, an active defence strategy⁵ uses sophisticated forensics and intelligence sharing — across industries and governments — to identify and counter cyber threats.

Of course, not every utility faces the same level of cyber risk or requires the same type of program to achieve an appropriate level of security. For the largest utilities, transforming their ICS Security Program into a comprehensive security operations centre (SOC) may be merited. For many others, an enhanced ICS Security Program that incorporates an appropriate level of external partners is sufficient.

All utilities must perform a business impact analysis or risk assessment to establish the appropriate governance and level of security for their ICS Security Program.

Where to start

It’s a process, not a project

With internal IT and ICS security expertise in finite supply — and outsourcing common — there is an understandable tendency for utilities to view any initiative related to information infrastructure, metering or process control systems as a ‘project’. By definition, projects are limited in scope and have well-defined objectives, timelines and budgets.

For example, a water utility might initiate a project to change over their metering system to AMI technology. It may select an outside vendor for the project, based on an open bid process. Expenditures are relatively finite and predictable — and expected outcomes can be easily communicated to the public.

Projects are focused on completing a task. Projects begin and end — and may even co-exist at cross-purposes with other projects. But when it comes to safeguarding a utility’s industrial control system, a ‘set-it-and-forget-it’ project mentality can be dangerously limiting. To be truly effective, cybersecurity must embrace a cohesive strategy that extends through every project in parallel with all business operations throughout a utility’s life cycle. A breach at any point can put the entire system at risk.

Simply put, cybersecurity is a critical business objective. As such, it must be approached as an ongoing process, albeit one where budget projections and public relations are challenging — and where success is measured by what doesn’t happen, rather than what does.

In Part 2

In Part 2 of this article a foundational methodology for building a cybersecurity program will be introduced.

References

1. US Department of Homeland Security 2013, Fact Sheet, <http://www.dhs.gov/sites/ default/files/publications/EO-13636-PPD-21-Fact-Sheet-508.pdf>
2. US Department of Commerce, National Institute for Standards and Technology (NIST) 2015, Cybersecurity Framework, <http://www.nist.gov/cyberframework/>
3. American Water Works Association (AWWA) 2014, Process Control System Security Guidance for the Water Sector, <http://www.awwa.org/portals/0/files/legreg/documents/ awwacybersecurityguide.pdf>
4. Chabinsky S 2013, Passive Cyber Defense: The Laws of Diminishing and Negative Returns, American Center for Democracy <http://econwarfare.org/passive-cyber-defensethe- laws-of-diminishing-and-negative-returns/>
5. Lee R M 2015, Threat Intelligence in an Active Cyber Defense, Part I, Recorded Future, <https://www.recordedfuture.com/active-cyber-defense-part-1/>

Top image: ©stock.adobe.com/au/Greg Brave