The FBI has released an advisory on the Oldsmar water treatment facility incident. In an advisory (shared on Twitter by POLITICO’s Eric Geller) the Bureau said the attack “likely” exploited an old Windows 7 operating system and weak password security as they (or he, or she) gained access to the TeamViewer software in use at the facility. The Bureau and the US Secret Service have joined state and local law enforcement in the investigation. No suspects have so far been named, or arrested.
The Tampa Bay Times notes that the attack could have been far worse than it turned out to be. The paper also quotes TeamViewer as saying that, while it had no evidence that its software had been compromised, it was monitoring the situation closely. (Most speculation holds that the attacker gained access to TeamViewer through compromised credentials.) The Miami Herald says that other regional water utilities have assured them that they have safeguards in place that would have prevented the sort of incident Oldsmar sustained.
Who did it remains an open question. ComputerWeekly goes to a canonical source, the 2005 film Batman Begins, which opens with the villains poisoning Gotham City’s water supply. Imaginative hacks can inspire real-world imitators. If this turns out to be the work of some lone skid, said skid may indeed take DC villains as his or her moral lodestar.
Attribution of a “sophisticated” cyberattack on a water supply.
“Sophisticated” has become the cyber equivalent of Lake Woebegone’s “above average.” Where the children of Lake Woebegone were all “above average,” so too the media are in the habit of calling every attack “sophisticated.” Maybe yes and maybe no, but it’s worth pointing out that the sort of attack Oldsmar’s water system sustained was within the range of many threat actors, from the lone twisted creep in a basement to a national intelligence service. And don’t by any means rule out the lone twisted creep, in it for the sick lulz.
Nozomi Network’s Andrea Carcano, sent us some comments on the degree of sophistication on display in Florida:
“Based on the information available at this moment, this attack seems to lack any sophistication that could trigger more profound reactions. The fact that the perpetrator didn’t conceal his visual presence to the personnel monitoring the water treatment operation is the first signal that suggests the relatively low complexity of the attack. Furthermore, according to the reports of the incident, the attacker increased the levels of sodium hydroxide by a significant amount, typically monitored by automated systems, which likely suggests that the threat actor didn’t possess a specific background knowledge of the water treatment process.
“Nevertheless, this incident is important because it reflects the status of too many industrial control system (ICS) installations, especially those with smaller budgets and a smaller size, where security is often overlooked. Remote access, in particular, when not designed with security in mind, is often the beachhead used by remote attackers to infiltrate an ICS network. In this very case, the water treatment plant of Oldsmar has been using a Teamviewer instance, which apparently was accessible from the Internet. While it is not known at this stage how the attackers obtained the credentials required, this incident, like many that we’ve documented in recent years, didn’t seem to rely on sophisticated zero day exploit for its execution.”
Tim Erlin, Tripwire’s VP of product management and strategy, also doesn’t see a lot of sophistication in play:
“While this incident will rightfully cause concern, it appears that the likelihood of real damage was minimal due to the fail safes in place. There are real impacts to be worried about, and actions to be taken, but this doesn’t appear to be a sophisticated or novel attack.
“From a cybersecurity standpoint, we should be particularly concerned about how the attacker was able to authenticate into the remote access software. That entry point should be very well protected, given that it provides access to such obviously sensitive capabilities. Protecting remote access into industrial systems where these types of changes can be made should be a high priority for any industrial environment.”
It’s also worth noting that critical infrastructure can be hit in a variety of ways. Paradoxically, the very modernization of some sectors has exposed them to new risk. Where long-lived, legacy systems by their very age afforded a degree of resistance to cyberattack, with many controls remaining manual and many automated systems being by their nature air-gapped, that’s changing, and the risk has risen accordingly.
Chloé Messdaghi, VP of Strategy, Point3 Security also emphasized that sophistication isn’t a prerequisite for conducting this sort of attack:
“The thing we need to understand is that you don’t have to be a highly skilled attacker to be able to successfully breach a system like this. Although alarms would’ve been triggered before any dangerous water reached anyone’s taps, this plant was very lucky that the worker noticed his mouse moving and was able to address it quickly. Water plants are not known for their security resources, and between budget cuts and COVID keeping people working remotely, they’re even more vulnerable. It’s becoming more and more easy to access systems like these by people who have hardly any experience at all. The area this happened in has a high population of children, and it’s disturbing to think someone would attempt to do harm like this.”
Managing risk in critical infrastructure.
Eddie Habibi, Founder of PAS (now part of Hexagon) also commented. He points out that enabling remote access has become critical to many, probably most, organizations over the course of the current pandemic. That access has to be given due consideration in risk management:
“The news that a hacker infiltrated a water treatment facility in Florida and changed a configuration setting to increase the volume of a dangerous chemical (lye) has rightly been greeted with concern by the media and cybersecurity community. The cyber threat to critical infrastructure has been increasing steadily as hackers, whether nation-state actors, criminal enterprises, or lone individuals better understand how to exploit operational technology (OT) in addition to IT systems. While much of the coverage of the cyber risk to critical infrastructure to date has focused on the age of many industrial control systems and the fact that they were not designed and deployed with security in mind, in this case, the attack vector appears to have been the increased level of remote access enabled by the Florida county.
“In the rush to support remote operations during the global pandemic, there are very likely many organizations who have increased remote access to industrial engineering workstations and operator consoles. Fortunately, in this case, there was a vigilant operator who noticed the 111x increase in the chemical (from 100ppm to 11,100ppm) and was able to take quick corrective action to return the configuration setting to its prior level. While industrial espionage remains a significant threat (not all cyber attacks are focused on disruption), the worst fears of many in the OT cybersecurity community were realized in this episode; namely, changing a configuration setting to harm the community served by the facility. It is a poignant reminder that the best foundation for effective OT cybersecurity is a detailed and broad asset inventory that includes relationships and dependencies among OT systems and a baseline of configuration settings. With this in place, risk assessment is far more informed, enabling organizations to more effectively assign and limit remote access at both the system and account levels. Indeed, the combination of an up-to-date asset inventory and risk-based remote access management policies is more critical now than ever before, as it enables both reduced risk as well as faster recovery in the event of an unauthorized change.”
Christian Espinosa, Managing Director at Cerberus Sentinel reminds us that the consequence of an incident is one of the factors operators should use in managing risk:
“Critical infrastructure, such as water treatment plants, need to be treated as such. Normally, critical systems, such as this water treatment system, do not allow remote access. Risk is the impact if something bad happens times the likelihood of it happening. In this case, the impact (poisoning, possible death) to the population using the water from this facility is quite severe. The overall risk is normally manageable though because controls, such as disallowing remote access, are put in place to make the likelihood of something bad happening very unlikely. The challenge we are facing with these types of scenarios is that most organizations do not understand cybersecurity risk. In fact, convenience is often the primary driver for decisions with cybersecurity a mere afterthought.”
Chris Grove, technology evangelist at Nozomi Networks, offered some thoughts on how organizations should respond to risks of this kind:
“As evidenced in this cyber attack, typical cyber security activities may not have mitigated this risk, including vulnerability management, network segmentation, system hardening, identity and access management, firewalling, etc. In many cases, and especially during this pandemic, remote administration solutions have been thrown into the mix, sometimes haphazardly. In some cases the due diligence and compensating security controls haven’t been recognized. In other cases it has. Either way, facilities should stop thinking like they will prevent cyber attacks and start thinking like they’re already happening. They may not see it, so they should be in a constant state of recovery.
“That said, concepts such as zero trust start making sense. Once the operator realizes that nothing is to be trusted, they move towards monitoring the process itself, and the parameters being sent from all of the devices in the control room to the equipment. If the water facility in Oldsmar had this level of cyber security, alarms would have gone off the moment the values were set to anomalous numbers.
“Typical cyber security monitoring would not have really helped in this case if the attacker came from an IP address in the neighborhood. Maybe, If the attacker was not located domestically. The firewall could have alarmed about the strange external connection. However, today its Teamviewer, tomorrow it’s an Android phone, the day after its SolarWinds or VMware. There are too many lives at stake to blanket trust all of the vendors to be safe and secure within their products, and combined with cyber safe products being abused and misused by attackers, it becomes clear that the monitoring needs to go wide and deep.
“Unfortunately most of today’s facilities are only protected a little bit by wide monitoring which doesn’t go deep into the industrial control protocols themselves. Any facility where human lives are at risk, particularly so many, should monitor the industrial control process using artificial intelligence and anomaly detection to monitor, alert and stop anomalies within the process that aren’t a part of regular operations. By doing so, the facility would mitigate many risks including malicious or negligent insiders that may accidentally type a few digits too many, as well as external attackers looking to pull off an act of terrorism.
“By monitoring deep inside the process for anomalies, it wouldn’t make a difference if the attacker took over the HMI (human machine interface) used by the control room to send commands, the attacker would only be allowed them to send previously used safe values without raising flags.”
Steps critical infrastructure and other industrial operators can take to reduce the risk of cyberattack.
Gary Kinghorn (of Tempered Networks) emailed to offer one immediate takeaway for infrastructure operators: think long and hard about the remote access you’re prepared to allow. He wrote:
“Yesterday’s hack of the Oldsmar, Florida water treatment plant again highlights the importance of maintaining critical infrastructure with a virtual air-gap (being off the network) from remote access. These systems should not be reachable by unauthorized attackers because of the sophistication of modern penetration tools and the complexity of these systems to make them completely free of vulnerabilities. Traditional firewalls and other remote access or VPN solutions are proving inadequate against these threats. We need to block any unauthorized access from ever reaching these critical and life maintaining systems while still allowing authorized, fully identified users remote access through secure tunnels using military-grade encryption. We have many such water districts using our solution for just these types of scenarios.”
The FBI’s recommendations for securing infrastructure are:
- “Use multiple factor authentication;
- “Use strong passwords to protect Remote Desktop Protocol (RDP) connections;
- “Ensure anti-virus, spam filters, and firewalls are up to date, properly configured, and secure;
- “Audit network configurations and isolate computer systems that cannot be updated;
- “Audit your network for systems using RDP, closing unused RDP ports; applying two-factor authentication wherever possible, and logging RDP login attempts;
- “Audit logs for all remote connection protocol;
- “Train users to identify and report attempts at social engineering;
- “Identify and suspend access of users exhibiting unusual activity;
- “Keep software updated.”
The FBI’s recommendations amount to useful reminders of sound cyber hygiene, and there’s no suggestion that the Oldsmar water system was afflicted by all the failures the Bureau’s list suggests. We mentioned yesterday that Dragos had published a quick set of recommendations useful in securing any industrial environment. They’re brief and worth sharing in full, as a complement to the FBI’s suggestions:
- “Manually identify software installed on hosts, particularly those critical to the industrial environment such as operator workstations- such as TeamViewer or VNC. Accessing this on a host-by-host basis may not be practical but it is comprehensive.
- “Beyond host data, there are a variety of network traffic sources to help identify TeamViewer. Most environments are not configured where centralized logging is occurring and can be a manual process. We recommend:
- “Use DNS logging to identify outbound DNS resolution to *.teamviewer.com
- “Encrypted communications to teamviewer.com will have a X509 certificate for *.teamviewer.com
- “Use perimeter logging or other network logging to identify external communications via TCP/5938 and UDP/5938.
- “Talk to the operations staff or IT staff at the site to determine if other remote software tools such as virtual private networks are used. If so, perform searches for those tools and where possible utilize multi-factor authentication on remote connections.
- “From a prevention perspective, blocking these communications, and all egress communications that are not explicitly approved, will prevent remote access solutions like TeamViewer. However, ensure that you talk with plant personnel before doing this and after blocking any connections be available to reverse the changes if something was necessary that they did not know about.”
A final disturbing thought: the attack was noticed and stopped by a watchstander who noticed something going on, briefly, on his workstation that didn’t seem right. As Nozomi’s Grove put it, “Had a facility operator not noticed the moving mouse on the screen, this attack would have gone much further. That level of attention should have been automated.” And Chris Grove should know: we’re told he lives down there. One hopes that there’s more redundancy in such safety systems than a single watchstander, however skilled and alert that watchstander might be. And for heaven’s sake, Oldsmar, give that operator a big raise.
