Network Aspects of the SLAC Cyber Security Improvement Program

1. NLIT Summit, San Francisco June 29-July 2, 2014 Network Aspects of the SLAC Cyber Security Improvement Program Dr. R. Les Cottrell, Hadas Niv, & Ben Calvert SLAC

2. SLAC Cyber Security Improvement Program (CSIP) • Secure Mobility • Secure wireless • Mobile Device Management • IP Address Management (IPAM) • Network Security (Firewall) • Network intelligence Network Part of an overall SLAC CSIP includes: Collaboration between Network Engineering & Cyber Security

3. Technical Overview

4. Secure Mobility – Secure Wireless Guillaume Cessieux Network Engineer

5. Current state of Wireless • 271 Access points, • 65% not 802.11n capable • Only uses 2.4GHz band • Grew from “nice to have” (best effort) to critical • Very popular • Access via open visitor net • No authentication, no data encryption • “Starbucks” model • Not scalable/manageable 4K unique devices/mon Business day: ~1500 devices Weekend: ~ 200 devices

6. What do we need? • A corporate like network for SLAC employees • Providing authentication &data encryption • Grant more privileges to wireless users • Seamless access to SLAC services (intranet, Lync…) • Same privileges as in office for mobile users • Reduce VPN needs • Improved roaming, robustness, availability, coverage, speed • Move towards strategic vision, reduce need for wired copper • Forrester predicts 59% of all data traffic will move from wired to wireless by 2017 • User impact – Ease of use critical • Visitor wi-fi? Yes, it will still be there. • Add always-on secure network connection to SLAC internal • No sponsors • No manual login

7. What does it provide? – Scope • Value Add: Grant privileges to SLAC wireless users to seamlessly access internal services • Secure roaming • Visitor & secure wireless deliver by same infrastructure • migrate to central/robust/easier to manage solution • Speed • Replace old, slower 802.11a/b/g Access Points with 802.11n • Add capacity in conference rooms and dense buildings • Using 802.11ac, get experience

8. Encryption/Authentication • Encryption: WPA2 with AES • Using EAP-TLS as the authentication mechanism • Certificates are attested by the Microsoft Certification Authority (CA) • Certificates are delivered to clients via Windows Active Directory (AD) Group Policy Objects (GPO) AD 2 Wireless Controllers Cisco 5508 Radius Client Access Points Kerberos

9. Current State of Secure Wireless • Only supports managed Windows devices • Working on smartphones tablets (see MDM next section) • Add MacOS support March 2015 • Controllers are logging to Splunk. • Next mine for analytics • Rolled out to 2 major buildings • Already popular: • ~ 40 users / work day using secure wireless • Simplifies sign on, • Easy roaming from office • Site rollout July 9th, 2014

10. Secure Mobility - Mobile Device Management Edgar Estabanez Cyber Analyst Guillaume Cessieux Network Engineer Rodney Wong Windows Engineer

11. What is the Mobile Device Management Project (MDM/EMM) • Secure & support smart phones and tablets • Current gap in our security posture • Audit finding from Stanford Information Security • A requirement of the Stanford Mandate (2014) SLAC joined Stanford MDM Affiliates • Consortium of four Stanford entities • Stanford University (SU) • Stanford Hospital and Clinics (SHC) • Lucile Packard Children's Hospital (LPCH) • SLAC National Accelerator Laboratory • Chartered to deploy a solution with a feature set and platform coverage acceptable to all members AirWatch chosen • Cloud solution did not require a large infrastructure by any one entity • Scalability is very flexible by having the infrastructure hosted by AirWatch

12. What is Mobile Device Management • Device set up • Policy acceptance • Profile distribution • Device compliance Enforce Device tracking Applications catalog Monitoring & reporting Manage Assets Secure information Passcode enforcement Remote lock Remote wipe • Applies to company owned and employee owned devices (BYOD)

13. Airwatch: lifecycle example J. Doe Role: Account Manager Corporate Services VPN Wi-Fi (New) Exchange 2010 Securely enroll device using AirWatch Stolen Device All Access Denied J. Doe Role: Account Manager Corporate Services None Device stolen! AirWatch removes corporate data, apps, access to corporate services and remotely wipes the device 5 3 Company upgrades Exchange and rotates Wi-Fi certificates AirWatch automatically upgrades device configuration J. Doe Role: Business Director Corporate Services VPN Wi-Fi Exchange 2010 Corporate Apps 1 J. Doe Role: Account Manager Corporate Services VPN Wi-Fi Exchange 2007 J. Doe is promoted to Business Director AirWatch configures device based on new role for higher access to corporate services 4 AirWatch configures device to access corporate services 2 AirWatch Corporate Resources Applications Content VPN Certificate Services Directory Services Mail Services Wi-Fi 13

14. Why at SLAC • Security • 1300+ mobile devices connecting to Exchange using Active Sync • Currently unsecured and unmanaged • Stanford mandate requires encryption & passcode for PII & health data • iOSand Android are the widely used platforms at SLAC • MDM is a basic foundation for mobility program • Computing initiatives • Security guards checking in, emergencies, construction managers • SLAC departments are looking at developing apps in-house

15. Policies being worked out • SLAC does not provide smart phones or carrier service • Enrollment requirements • All SLAC-owned devices must be installed with AirWatch • Installation of an agent on device • Will be a Passcode requirement • Wipe of lost devices • Wipe of enterprise data on employment termination • Careful consideration needed when affecting “birthright” • How do rules apply to employee owned devices (BYOD)? • Consider enforcing MDM as a requirement to access SLAC email? • Alternately, enforce a similar security policy on unmanaged devices using ActiveSync?

16. Communication Plan is Critical • Road show: Stanford enrolled 7,000 devices by selling the personal protection aspects • Engagement with key groups (Business Managers, Sys Admin Forum, Governance group) • Web and email communications • Marketing message • Access to internal resources via Secure Wireless (limited to managed devices) • Personal benefits of device security • Convenience & productivity (one step setup for Secure Wireless, VPN, Lync, etc.) • Future benefit of enterprise applications • Foundation for future

17. Network Security – IP Address Management (IPAM) Open source: From Stanford Yee Ting Li, network engineer Kent Reuber, network engineer

18. What is “IP Address Management”? – Scope, assumptions • System of record for DNS names to IP addresses for all IP assets on SLAC internal networks • Contains information of physical locations • Map IP to physical interface, host type and user/administrator • Integrates with DNS servers (BIND, Windows) and DHCP servers • Related to asset management • Current system (1980’s vintage) not meeting current needs • E.g. does not support IPv6, Media Access Controller (MAC) addresses, no granularity of authorization • Replace the IPAM part of current system with NetDB from Stanford

19. What is NetDB? • Stanford University Open Source IP address management system. In use for over 20 years. • Web GUI interface. Also includes a Command line interface & interfaces for power users and scripting. • Over 1,000 Stanford NetDBregistered admins.

20. What does NetDB provide? • Authorization of admin allowing granular per-admin and per-group access privileges. • “Full Search” capabilities: find all devices in a certain building, maintained by a certain administrator, … • Automatically feeds DNS and DHCP services • IPv4 and IPv6 • Supports static (Media Access Control (MAC) address -> fixed IP) and dynamic (“Roaming”) addresses or a mix of the two. • Easy to learn. Stanford estimates that the GUI can be taught in 15 minutes. • Extensive Help that is actually useful.

21. Migration of data: Current Record of Records • Example: • Get ARPs from LANMON • Correlate to IP address in the old IPAM • Match IP addresses in Goliath and Taylor (get user data, serial numbers) • Match Serial Numbers to SLAC PC (get purchase information) • All non-matching entries in old IPAM can be deemed ‘stale’ • But still requires human to validate each entry…

22. Cleaning /prune data • ‘bad’ CANDO entries: ~9,000 out of ~25,000 • did not match to a network ARP within last 6 months • Mostly old clusters (decommisioned equipment) or temporary DHCP records • ‘non-matching’ records: ~2,000 out of 16,000 • Entries with conflicting information about IP or MAC addresses • Stale data on ‘people’ • Map to groups and or (company) hierarchy. • Import into NetDB, coordinate with ServiceNow Assets • Part of a federated set of databases

23. Conclusion • SLAC is replacing its old IPAM with Stanford’s NetDB. • Goals: • Better ability to find & track down systems & contacts • Enable self service: Distributed administration with limited privileges • Create permission groups and user records • Support for IPv6 • Increased ability to use DHCP, automate address assignment etc. • Feed SLAC DNS & DHCP • Ongoing work to clean & convert old data into NetDB. • Train admins on NetDB.

24. Network Security – Border firewall Antonio Ceseracciu, network engineer

25. Objective • Create a Next Generation SLAC Border Firewall - a security device that sits logically between the "SLAC Business network" and outside networks.

26. Proposed State • The basic changes involved with this project • Addition of Next Generation border firewalls with content inspection • High speed bypass to support Scientific Computing data transfers Internet SLAC Border • Retirement of existingweb proxies Visitor Impact: Fewer viruses & attacks get through Mostly transparent to user SLAC Core Enterprise Science DMZ Buildings Controls

27. Firewall Features

28. Firewall Features

29. Deployment Strategy and Approach • The firewalls shall not constrain or disrupt legitimate science activities and operations related to the SLAC mission. • The firewall deployment introduces no Single Points of Failure in SLAC Core Network. • Enable the firewall protection without adverse impact on users' computing experience. • As much testing as possible will be done in the staging phase • Rollout will be gradual: a small group of testers first, then the Computing Division building, then the entire site • Each iteration includes time to correct issues and provides assurance • This approach is reflected in the project deployment schedule

30. Firewall Project Status • The firewall vendor has been selected: Palo Alto Networks • Purchased: • A pair of PAN 5060 firewalls • Threat Protection • Wildfire (sandboxing) • URL filtering • The firewalls are being configured and tested in a test bed network

31. Network Security – Network Intelligence and Forensics Bro Dr. Yee-Ting Li, network engineer

32. Goals • Design and implement scalable cyber infrastructure to monitor and alert cyber incidents (network taps etc.) • Complement Next-Gen Firewall activities • Firewall will not protect scientific computing network • Implement Bro cluster to monitor science traffic • Evaluate and scope the need for full/partial packet captures • Help drive meaningful policies based on empirical information from Bro and network monitoring system(s)

33. Where does Bro come from? • International Computer Science Institute (ICSI)/ UC Berkeley / LBNL • Developed >15 years ago at LBL • Open Source (BSD) • Funded mostly by R&D grants (NSF, DoE) • Large online community and annual conference • Used at major universities, labs, supercomputing centers, corporations, researchers Sponsors

34. What is Bro? • Clustered software that runs on *nix • Takes in a network ‘feed’ • Does ‘super’ tcpdump • Can feed in syslogs (or any other streaming log) • Analyses the packets ‘live’ using ‘bro scripts’ • Uses a security domain specific scripting language • Bro scripts effectively notify Bro that: • should there be an event of a type we define, • then let us have the information about the connection • so we can perform some function on it. • Can write our own! • Creates log files from `bro scripts’

35. Why do we need Bro? • Many cyber attacks per day, constant change • Spam, phishing, brute force, DoS, virus, Trojans, worms etc. • Traditional IDS/IPS very costly (or do not exist) at high network speeds/volumes • SLAC plans to deploy 100Gbps network for science • Commercial IDS/IPS products cost prohibitive at such speeds • Bro scales horizontally by adding more commodity servers • Requires frontend ‘load-balancing’ solution • Out-of-the-box will only monitor traffic • Transparent to users • Necessary to help protect SLAC science • Not protected by Next Gen Firewall

36. What can Bro do? • Provide audit trail for forensic analysis • Who else did a host infect? • When was a machine last seen on the network? • Who’s trying (and succeeding?) to attack us? • What types of attacks are we seeing? • Provides insight into what’s happening on the net • Ensure only science applications are installed on science networks • Top 10’s of application traffic volume, top talkers • Plus anything you can think of via Bro Scripts • Take action based on Bro alerts • e.g. send email; add a null route for suspicious IPs etc.

37. Value Proposition of Bro • Monitor and alert on network activity ‘as it happens’ • Most of our systems are ‘near real-time’ (scans etc.) • Provide insight into application level data • Netflow only provides ‘meta’ data • Customizable to internal procedures and processes • e.g. use router block as opposed to having inline IDS • $ per GB of traffic analysis very cost effective • Plenty of community support and direct application to our (lab) environment • Easy to create reports: • Top application traffic • Unpatched software

38. Bro at SLAC Implementation High performance compute & data clusters for science 1 dedicated manager/proxy 4TB storage Bro Cluster e.g. Del R620 SPAN or Optical split Currently 6 Bro instances (scalable) Front end Bro switch e.g. Cisco 3172TQ or Arista 7150 Monitoring network2x 10Gbps Cu/host Gen purpose net: Patching, sshetc 1Gbp/host Packet Capture Network: 1x40Gbpsor 2* 10Gbps Time machine, e.g. Dell R710 + 16TByte storage

39. Time Machine Packet Capture • Keep copies of data packets on SLAC network • tcpdump with snaplen of the entire packet • Exclude storing ‘uninteresting’ data • ISO images, encrypted traffic etc. • Useful for forensics and replay • Full system with decent retention very expensive • Will repurpose existing hardware as trial • 16TB storage • Target important networks • Defense in depth

40. Summary

41. Summary: Mobility • Secure Wireless • MDM – AirWatch design 41 AD 2 Wireless Controllers Cisco 5508 Radius APs Kerberos

42. IP Adress management, DHCP • IPAM –NetDB features and Capabilities • IPAM –NetDB data aggregation

43. NGFW & network intelligence • Border Firewall – features and deployment strategy • Network Intelligence - Bro design

44. The end: Questions? • Contacts: • Network manager: • cottrell@slac.stanford.edu, • PM: • hadas@slac.stanford.edu • CSIO: • ben.calvert@slac.stanford.edu • SLAC National Accelerator Lab, • 2575 Sand Hill Road • Menlo Park, CA 94025 • Slides available at: https://confluence.slac.stanford.edu/download/attachments/17164/nlit-2014-net-csip.pptx