Operational Security Moving Beyond the Firewall Argonne National Laboratory

1. Operational Security Moving Beyond the FirewallArgonne National Laboratory Michael A. Skwarek, Deputy CIO & Cyber Security Program Manager Christopher Poetzel, Computer Network Engineer Argonne National Laboratory 2009 DOE Cyber Security Conference May 13, 2009

2. About the Presenters • Michael A. Skwarek, Deputy CIO and Cyber Security Program Manager • Responsible for the effective balance between Cyber Security and Science • Strong supporter of risk based cyber security systems that integrate and provide efficiencies and effective communications to those in the trenches • Christopher Poetzel, Computer Network Engineer • Responsible for the management and integration of the Laboratory firewalls • Strong code and analysis capabilities in anomalous Intrusion detection

3. Diverse population: 2500 employees 10,000+ visitors annually Off-site computer users Foreign national employees, users, and collaborators Diverse funding: Not every computer is a DOE computer. IT is funded in many ways. Our goal: a consistent and comprehensively secure environment that supports the diversity of IT and requirements. Argonne National Laboratory Laboratory IT Environment Argonne is managed by the UChicago Argonne LLC for the Department of Energy.

4. Emphasis on the Synergies of Multi-Program Science, Engineering & Applications FundamentalPhysics AcceleratorResearch InfrastructureAnalysis ComputationalScience MaterialsCharacterization Catalysis Science TransportationScience NuclearFuel Cycle User Facilities StructuralBiology .. and much more.

5. Operational Security – Moving Beyond the Firewall • Operational Cyber Security is very reactionary in nature • The “Dashboard view” drives the day – “Green lights vs. Red lights”. • Intel from outside sources only provides a catalyst to the actions. • Questions continue to remain on the table • How do we do more with less? • How do we learn from our incidents and those of others? • How do we leverage a risk based approach for cyber security? • How do we architect a cyber defense system that is not going to get “top heavy”? • Incident review and root cause analysis • Learning from your “mistakes” and the pain felt of others can be a healthy process. • Think like a hacker • Results of analysis and a clear understanding of the threat and risks can build new and effective defense in-depth cyber systems. • Realization that there will never be a silver bullet to solve all of our problems.

6. Group Exercise: An incident in slow motion • Walk through a hypothetical cyber security incident that carries many trademarks of today’s reality. • Review the root cause elements that allowed for the incident to manifest and continue. • Through reflection, we will describe a number of mitigations in place today at Argonne that can be leveraged across the complex to mitigate similar and future attacks.

7. Phase I: Creation and Delivery of Infection • E-Mail addresses are harvested via an online phone book of employees within the Organization. • E-mail messages are crafted along with a Microsoft Word attachment that contains a malicious Zero-day exploit found within Office 2007. • Microsoft and AV vendors have not provided patches/virus signatures. • Local desktop administrative permissions are not required for exploitation. • Successful exploitation will result in the permission environment of the user with whom executed the file.

8. Cyber Incident in Review Firewall AV/SPAM CLEAN IDS

9. Phase II: Infection A • Recipient “A” is a member within the HR division • The employee has the following IT environment • Desktop: Fully patched Windows XP running Office 2003 and a member of the domain. • Virus Protection: Fully up-to-date. • Access Permission: Non-Administrator • Recipient “B” is a Post Doc within a Programmatic division • The employee has the following IT environment • Desktop: Fully patched Windows XP running Office 2003 and a member of the domain. • Virus Protection: Fully up-to-date. • Access Permission: Administrator B

10. Phase III: Command and Control • Recipient “B” system detects and reports that the user is local administrator -> Attack successful • Recipient “B” system has established a command and control session with “the mothership” over a non-SSH protocol TCP/22 “VPN like” connection. • The local system is modified to created a new local service to ensure that the command and control can be established after a reboot. • Antivirus is disabled on the local system to prevent detection of certain tools. • The remote attacker installs a virtual machine on the infected system, stealing an open IP address on the subnet. • Horizontal movement across the organization is now in mind

11. Communications are Established Firewall IDS CnC: 1010011001*Admin*01101001 CnC: 1010011001Non-Admin01101001 B

12. CnC: host1.rotate.org*tcp/22*01101001 Communications are Established Firewall IDS B

13. Communications are Established Firewall IDS Outbound TCP/22 B

14. Phase IV: Horizontal Movement • With the intention of moving to other systems within the organization, the attacker will need valid credentials for successful movement. • Option 1: Crack the desktop local passwords • Attacker is hoping the cracked password is used on all systems • Often seen for ease in large administration environments • Option 2: Use the “Pass the Hash toolkit” • Within seconds, can have the password hashes of the last 10 logons to the system. • Mostly likely, an admin has logged on once… if not SMS • With credentials, it is time to scan the network for where the attacker can go.

15. Horizontal Discovery and Spread Firewall IDS Outbound TCP/22 Outbound TCP/22 iam.exe -h administrator:mydomain:0102030405060708090A0B0C0D0E0F 10:0102030405060708090A0B0C0D0E0F10 A B PII

16. Root Cause Analysis and Associated Impacts

17. Phase I: Exploit Creation and Delivery

18. Phase I: Exploit Creation and DeliveryArgonne Mitigations and Actions • Root Cause: Public facing website provides valid email address of all employees. • Argonne has removed the capability to harvest email addresses from the public facing website. • Initial Email communications are established via a website that proxies the requested email to the recipient thus hiding the valid email address from attackers. • Root Cause: Attacker has knowledge of zero day exploit • Attempting to stay up-to-date on zero day exploits can be difficult but not impossible with strong communication and awareness. • Root Cause: Vendors do not have patches or signatures • This happens a number of times throughout the year, and in some cases there are workarounds to mitigate some of the risk. • User Education and Awareness are your first line of defense. Over communicate where possible (Email, Daily Newsletters, etc) and patch ASAP when available. • Root Cause: Attacker has knowledge, time and intent

19. Phase II: Infection Recap

20. Phase II: InfectionArgonne Mitigations and Actions • Root Cause: Recipients are enticed to open and execute email attachment • User education is key as the employee is the last line of defense. • Conducted a number of social engineering assessments. For those that “went to far”, immediate training is provided. • Plans to conduct future social engineering tests leveraging Core Impact. • Root Cause: Recipients are a user of their local desktops • This is a best case scenario. The exploit if run will assume the permission level of the recipient. In most cases, a non-administrative system is not interesting to attackers. • Root Cause: Recipients are administrator of their local desktop • This is a worst case scenario. Administrative permissions in the wrong hands can spell disaster. • Argonne has taken a hard stance on administrative permissions, and has required that employee accounts be provided the least user access required to fulfill their job requirements. 20

21. Phase II: Least User Access (LUA)Argonne Mitigations and Actions • Least User Access rollout has both cultural and technical hurdles. • Interestingly, many employees are not aware of what it means to be “admin” • Not every scenario works in a true LUA environment. • Exceptions to the rule are a fact of life since older applications were never written for LUA compliance. • Argonne exceptions to LUA are requested and vetted through the Cyber Security Office. • In the event that administrative permissions are required: • BeyondTrust is attempted to bridge the gap • If BeyondTrust does not meet the need, the user is provided a local administrative account and educated on the use thereof. • In near real-time, systems are monitored with Snare and Splunk for account membership changes. • On a nightly basis, each system is interrogated for local administrative membership and compared against the approved list. • Rogue accounts are removed

22. Phase III: Command and Control Recap

23. Phase III: Command and ControlArgonne Mitigations and Actions • Root Cause: Infected systems have unfettered access to Internet. • Installed divisional firewalls with both ingress and egress rulesets • Egress firewall deny logs are analyzed for anomalous behavior via splunk and in house written code to strengthen the signal to noise. • Beacon knocking is easily detected • Leverage web filtering product to block identified malicious websites • Instituted “DNS Blackhole” capability to “deny” DNS resolution to known malicious domains (currently ~32K domains). • Created inline IDS rules to detect when traffic on a given service port does not match the protocol’s characteristics. • Ex: Identified traffic on TCP/22 does not look like SSH • Root Cause: Attacker alters local desktop mitigations • Installed Snare on clients to forward to splunk anomalous events. • Account Management, Group Management, Logon Failures • Script created to generate event log for service creation • Script created to monitor antivirus process and restart if a STOP is detected. 23

24. Phase III: Command and ControlArgonne Mitigations and Actions Continued • Root Cause: Attacker able to gather and install toolkits without detection • Note: In some cases, attackers have installed Virtual Machines on the infected systems to ensure that tools are present and working • Completed an integration between authoritative host warehouse and router ARP cache. • New MAC/IP pairs are detected through ARP are compared against the host db • If a MAC/IP is detected to not be known, a shun is installed into the divisional firewall to halt traffic • Future natural extension is to drop the port, which has some issues revolving around hubs/switches. 24

25. Phase IV: Horizontal Discovery and Spread

26. Phase IV: Horizontal Discovery and SpreadArgonne Mitigations and Actions • Root Cause: Ability to easily harvest valid credentials from system and domain. • Due to advent of “Pass the Hash” windows environments are at great risk. • An aged configuration which enables logon capabilities in the event of loss of network or on remote, has the capability to gain domain admin within seconds • Altered the Cached Credential values on all Windows systems • Desktop and Servers = 0 • Laptops = 1 • Script created to randomize the administrator password on all windows systems after reboot. • This password is not known by anyone, and is not the same on any two systems. • Root Cause:Unfettered internal network access. • Network vlan boundaries are defined at the division level, and in some cases broken down even further. • Ingress firewalls are installed to hamper horizontal network access 26

27. Phase IV: Horizontal Discovery and SpreadArgonne Mitigations and Actions Continued • Root Cause: Anomalous internal network behavior not detected. • Leveraging network netflow records, traffic patterns can be analyzed against what is expected/normal traffic. • In-house written scripts are in place watching for anomalous traffic patterns throughout the core of the network. • Ex: It is not normal for a system to chat with >50 hosts in a minute on a given port(s). • “Noisy” network reconnaissance is easily identified • Root Cause: Anomalous amounts of traffic payloads not detected. • Leveraging network netflow records, traffic patterns can be analyzed for large payload flows leaving the network. • Creating a baseline of “norms” by hour and day can identify anomalies. • Special attention is paid towards systems of known sensitivity 27

28. Takeaways and Lessons Learned….

29. Presentation Top Takeaways and Lessons Learned • No two incidents will behave the same • Build general defenses to compensate for deviations in the attack vector. • There is no silver bullet answer • Sadly, the answer to all of our problems is not hiding out there. • Build systems that integrate • Integration of cyber defense systems can build new defense systems. • Keep in mind the “signal to noise” of defense systems • Red lights that are really not “Red” will only cause them all to be ignored. • Communicate and educate in everyway possible • Find ways to reach your employees, management and peers. • Build strong systems from the “ground up” • Preach and follow strong configuration management, enough said.

30. Contact Information and Questions • Please feel free to contact us with any questions or comments that you may have regarding the systems and capabilities mentioned within the presentation • Michael A. Skwarek (mskwarek at anl dot gov) • Chris Poetzel (cpoetzel at anl dot gov) • Questions?