Florida Water Treatment Plant Attack

April 22, 2021

Since there are plenty of threat monitoring sources throughout our industry, we thought it best not to repeat those concerning the Florida water treatment plant attack. ThreatGEN believes it is worthwhile to have a lessons learned after the the dust settles though. The reason we didn’t release immediately after the event, like everyone else, is because we take a retroactive look at data and facts rather than a knee jerk reaction and trying to use it as ambulance chasing marketing…

The Florida water treatment plant attack

An attacker managed to gain access to the public exposed TeamViewer remote access software of a Florida water treatment facility. The TeamViewer access allowed him or her to interact with an operation station, which in turn allowed the attacker to manipulate the setpoint for a chemical dosing control of the Industrial Control System. If gone unnoticed, this setpoint change would have increased the level of sodium hydroxide (lye) in the drinking water to levels that are considered corrosive, which if not picked up by a secondary safety mechanism (pH sensor/alarm or regular manual checks), could have made the drinking water harmful when ingested.

Takeaways from news sources and recommendations

First takeaway from the Florida water treatment plant attack

According to several news sources, the attacker during the Florida water treatment plant attack connected to facility’s TeamViewer software at least twice, briefly on Friday morning, and later that same day. Also, according to news sources this initially did not alarm the operator as his supervisor does this regularly. This is a red flag. Remote access to critical controls should not be commonplace! At a minimum there should be a procedure where the supervisor informs operators, he or she is accessing the system, ideally having the operator enable/allow that connection on the plant’s side.

Second takeaway

The TeamViewer software allowed access directly from the internet. This is the same as connecting the ICS directly to the internet. This should never be the case. Remote access to the industrial environment should be carefully considered and only allowing those individuals with the need to do so through properly defined and strictly controlled and monitored methods. The figure below shows how with properly architecting a remote access solution we can implement 4 layers of access control and monitoring throughout the entire connection process:

A remote user (vendor or employee) would connect to a remote access solution that is connected to the business side of the plant, using Multi Factor Authentication (MFA). The remote access solution should implement some sort of virtual machine technology that effectively separates the remote user from the company network and has that user instead use company owned assets to interact with the plant’s resources (as opposed to a VPN connection that just connects the remote user’s network to the company network).
Securely logged onto the remote access VM in the IT DMZ, the user can now connect through the Industrial Demilitarized Zone’s (IDMZ) remote desktop gateway (RDP-GW), optionally requiring a second MFA verification. The RDP-GW allows restrictions on:
1. What computer initiates the connection request (remote access VM)
2. What user can connect to what industrial resource (target restrictions)
3. What the user can do over the connection (restrict on shared resources such as clipboard, printers, etc.)
4. When the user can connect and for how long
After extensive authentication and authorization checks, the connection is brokered to the target system on the industrial network. Built into the design of an IDMZ, by switching of protocols during this process (RDP-GW uses HTTPS (443) in and RDP (3389) out), prevents a vulnerability on the enterprise side, complete and free access into the industrial zone.
The target in the industrial zone is a virtual desktop session where the user has just enough access rights to perform his or her job. Additional restrictions in the form of Windows domain policies, and OT vendor security protocols such as Rockwell Automation’s FactoryTalk security can be applied.

Properly architected remote access solution

Third takeaway

The remote access allowed changes to the process controls. This scenario should be prevented at all costs. If any “interactive” access with the control system is warranted, a remote access solution as described above should be implemented. However, more often than not, a remote user needs his or her access to merely view some values, run a report or keep an eye on the production process. We don’t need to allow access all the way into the industrial network to allows this type of data sharing. A Reverse Proxy broker service in the IDMZ can relay production reports from a Historian data server and expose that data securely on the Business network (enterprise zone). A secure way to share production data with business users (or remote users that connect to the business network via a remote access solution) is shown in the figure below:

Data is collected from production systems by the Historian server/solution on the industrial zone.
The historian server/solution publishes production data to a web server, composing a webpage with relevant data, statistics, information, etc.
The reverse web proxy in the IDMZ is configured to relay (broker) web access from enterprise (business) users to the web page exposed by the Historian server. A potential attacker would try to compromise the reverse web proxy in this scenario, and even if successful, would not allow them access to the Historian server or the industrial network.

A secure way to share production data with business users (or remote users that connect to the business network via a remote access solution)

Fourth takeaway

The attacker in the Florida water treatment plant attack managed to up the setpoint from 100 to 11,100. This kind of increase should not be allowed by the control software. If the setpoint is typically around 100, a 100x increase is abnormal and should be prevented or at least alarmed upon. Additionally, one should restrict changing critical setpoints with passwords.

Fifth and last takeaway from the Florida water treatment plant attack

The final takeaway is based on speculation, reading all these news stories about the Florida water treatment plant attack but I am going to go out on a limb and say the water treatment facility did not have proper cyber security monitoring in place. Without data such as connection logs into the TeamViewer software, access logs for the operator station, network logs, event logs or any other type of security logs it will be really hard to trace down what happened, how it happened and who did it. Make sure you are properly recording, logging, and alerting on your security posture.

For further reading

And finally the base sources about the Florida water treatment plant attack….

About Pascal Ackerman

Pascal Ackerman is a seasoned industrial security professional with a degree in electrical engineering and with 18 years of experience in industrial network design and support, information and network security, risk assessments, pentesting, threat hunting and forensics. After almost two decades of hands-on, in-the-field and consulting experience, he joined ThreatGEN in 2019 and is currently employed as Principal Analyst in Industrial Threat Intelligence & Forensics. His passion lays in analyzing new and existing threats to ICS environments and he fights cyber adversaries both from his home base and while traveling the world with his family as a digital nomad.

About ThreatGEN

Founded in Sugar Land, Texas in 2017, ThreatGEN delivers a solution to bridge “the Operations Technology (OT) Cybersecurity skills gap” utilizing its ThreatGEN® Red vs. Blue and ThreatGEN Services.

ThreatGEN® Red vs. Blue training uses cutting-edge computer gamification to provide an exciting & modernized approach to OT cybersecurity training, both practical and cost effective!

ThreatGEN Services are delivered worldwide by world-renowned OT cybersecurity experts (we literally wrote the books industry uses) using strategically chosen partnerships to create a holistic service offering.

Books written by Clint Bodungen, Aaron Shbeeb, and Pascal Ackerman.