UW Medicine Networking Update

1. UW Medicine Networking Update Terry Gray Associate Vice President, IT Infrastructure University of Washington 16 April 2004

2. Changed: C&C now responsible for... In-building network implementation andoperational support for med ctrs, clinics Med center network design “for real” Not Changed: C&C still responsible for... Network backbone, routers Regional and Internet connectivity SoM and Health Sciences networking Key Elements of the Partnership

3. Consistency, interoperability, manageability Leverage C&C networking expertise Clinical/research hi-performance network needs 24x7 Network Operations Center (NOC) Advanced network management tools Avoid design/build organizational conflicts Beyond the network...hope to share distributed system architecture and network computing expertise Why the Partnership Makes Sense

4. Created “Top 10” list --now up to Top 20 :) Agreement on standard maintenance window Static addressing work-around (sDHCP) FDDI, VLAN elimination Subnet splits/upgrades (1500 computers) Equipment upgrades Router consolidation, dedicated subnets, separate med center backbone Equipment, outlet location database updates Initial wireless deployment NetVersant and Cisco external studies Near-term Progress and Plans

5. The Challenge Create a network computing environment • with excellent security • excellent supportability • that users find reliable and responsive

6. Increased dependency on network apps Decreased tolerance for outages Decades of deferred maintenance... Inadequate infrastructure investment Some old/unfortunate design decisions Some fragile applications Fragmented host management Increasingly hostile security environment Increasing legal/regulatory liability Increasing importance of research/clinical leverage Context: A Perfect Storm

7. Context: Some Numbers

8. Network Device Growth Note: Most dips reflect lower summer use; last one is a measurement anomaly

9. Network Traffic Growth (linear)

10. Network Traffic Growth(log)

11. Environmentals (Power, A/C, Physical Security) Network Client Workstations Servers Applications Personnel, Procedures, Policy, and ArchitectureFailures at one level can trigger problems at another level; need Total System perspective System Elements

12. Old infrastructure (e.g cat 3 wire) Non-supportable technologies (e.g. FDDI) Non-supportable (non-geographic) topology Expensive shortcuts (e.g. cat5 mis-terminated) Security based on individual IP addresses Subnets with clients and critical servers Documentation deficiency Contact database Device location database Critical device registry Systemic Network Problems(some of these go back decades)

13. Ever-increasing system complexity, dependencies Ever-increasing threats, liabilities Departmental autonomy Un-controlled hosts Un-reliable power and A/C in equipment rooms No net-oriented application procurement standards Are HA and DRBR expectations realistic? Are backup plans workable? Systemic General Problems

14. Key Operational Objectives • simplicity • lower cost • higher MTBF (modulo redundancy) • lower MTTR (quicker diagnosis) • consistency • deterministic outlet behavior (Network Utility Model) • connection transparency (open/deterministic Internet) • easier problem diagnosis • These objectives conflict with other goals

15. Networks = Connectivity; Security = Isolation Fault Zone size vs. Economy/Simplicity Reliability vs. Complexity Prevention vs. (Fast) Remediation Security vs. Supportability vs. FunctionalityDifferences in NetSec approaches relate to: Balancing priorities (security vs. ops vs. function) Local technical and institutional feasibility Design Tradeoffs

16. Tradeoff Examples • Defense-in-depth conjecture (for N layers) • Security: MTTE (exploit)  N**2 • Functionality: MTTI (innovation)  N**2 • Supportability: MTTR (repair)  N**2 • Perimeter Protection Paradox (for D devices) • Firewall value/efficiency  D • Firewall effectiveness  1 / D • Border blocking criteria • Threat can’t reasonably be addressed at edge • Won’t harm network (performance, stateless block) • Widespread consensus to do it • Security by IP address

17. 1990: Five anti-interoperable networks 1994: Nebula shows network utility model viable 1998: Defined border blocking policy 2000: Published Network Security Credo 2000: Added source address spoof filters 2000: Proposed med ctr network zone 2000: Proposed server sanctuaries 2001: Ban clear-text passwords on C&C systems 2001: Proposed pervasive host firewalls 2001: Developed logical firewall solution 2002: Developed Project-172 solution 2003: Slammer, Blaster… death of the Internet 2003: Developed flex-net architecture Network Security Chronology

18. Parallel networks; more redundancy Supportable (geographic) topology Med center subnets = separate backbone zone Perimeter, sanctuary, and end-point defense Higher performance High-availability strategies Workstations spread across independent nets Redundant routers Dual-homed servers Next-Gen Network Architecture

19. Tom’s Nobody gets hurt Nobody goes to jail Steve’s Four Nines or bust! High ROI (Return On Investment) Terry’s Low ROI (Risk Of Interruption) Low MTTR (Quick to Fix) High predictability (No surprises) Success Metrics

20. Net reliability & host security are inextricably linked Five 9s is hard (unless we only attach phones?) $ for $, best security investment is central host management Nebula existence proof: security in an open network Watch out for unfair cost shifting The cost of static IP configuration is very high Controlling net access is hard --hublets, wireless Even host firewalls don’t guarantee safety Perimeter firewalls may increase user confusion, MTTR It only takes one compromise inside to defeat a firewall Next-generation threats: firewalls won’t help Even so… defense-in-depth is a Good Thing Lessons

21. Questions? Comments?

22. Network Security Addendum

23. Recent Events • attacks • slammer (Jan 2003) • blaster (Aug 2003) • sobig (Sep 2003) • mydoom (Feb 2004) • witty (Mar 2004) • impact • demise of the open/transparent/deterministic Internet • demise of the network utility model • demise of the unmanaged/autonomous PC • demise of reliable email

24. Seven Security Axioms 1. Network security is maximized when we assume there is no such thing. 2. Large security perimeters mean large vulnerability zones. 3. Firewalls are such a good idea, every computer should have one. Seriously. 4. Remote access is fraught with peril, just like local access. 5. One person's security perimeter is another's broken network. 6. Private networks won't help (Limits of isolation). 7. Network security is about psychology as well as technology.

25. Network Security Credo • Focus first on the edge(Perimeter Protection Paradox) • Add defense-in-depth as needed • Keep it simple (e.g. Network Utility Model) • But not too simple (e.g. offer some policy choice) • Avoid • one-size-fits-all policies • cost-shifting from “guilty” to “innocent” • confusing users and techs (“broken by design”)

26. What is it? Why important? Incompatible with perimeter security? Too late to save? NUM-preserving perimeter defense Logical Firewalls Project 172 Foiled by static IP addressing… Requires all hosts be reconfigured Preserving the Net Utility Model

27. Conflicting Perspectives • System administrator view • some prefer local control/responsibility • some prefer central/big-perimeter defense • some underestimate cost impact on others • User view • want just enough openness to run apps • prefer “unlisted numbers”? • Network operator view • concerned about increased support costs and repair times due to growing complexity and unpredictability • concerned about loss of network functionality

28. Generic Security Toolkit • host choice: truly thin clients; species diversity • host configuration management • conventional firewalls • logical firewalls • private addressing (e.g. project 172) • IDS, IPS, ADS • vulnerability scanning, anti-virus tools • QoS (to protect critical traffic types) • isolated networks (physical, VLAN, VPN) • non-technical: policies, education, staff

29. network isolation for critical services host integrity (Make the OS net-safe) host perimeter (integral ACLs/firewalling) cluster/lab perimeter (sanctuary, FW, LFW) network zone perimeter (P172, FW) real-time attack detection and containment user education Lines of Defense

30. Perimeter Firewalls • increase time-to-infection • increase time-to-repair • provide defense-in-depth • may look like a broken network to users • are defeated by a single hacked host • are defeated by tunneling/encryption • often give a false sense of security • encourage backdoors • may be a performance bottleneck • may inhibit legitimate activities, innovation • create a vulnerability zone that is hard to protect: • vpns, laptops, wifi, usb drives, social engr attacks • the more you depend on perimeter defense, the more you must invest in defending the perimeter

31. Operational Impact by firewall type • host -- best case; user interaction w/FW possible • cluster -- no impact on net diagnosis “beyond” • logical -- low impact on basic net diagnosis • subnet -- impacts almost all diagnosis • zone -- impacts inter-zone diagnosis • border --impacts inter-enterprise diagnosisNB: cost of maintaining firewall config depends on who is doing it, and how many rules/exceptions there are.

32. hosts connected to two different networks can become attack gateways between the two example: home PCs with VPN connection to protected network safer remote access: SSH, SSL, K5, RDP, SSL VPNs Limits of Isolation:attack gateways

33. Med Center Zone Perimeter • purpose • time to defend against zero-day events • protect the otherwise unprotected • defense-in-depth • reduced annoyance/noise traffic • DOS attack mitigation • options • conventional inline firewall • private addressing + NAT or proxies • both

34. FDA-approved devices, printers, etc protection options (besides zone perimeter): private addressing individual firewall, VPN, or NAT box ($25 - $2500)--depending on performance requirements cluster/lab perimeter firewalls logical firewalls Protecting Non-fixable Devices

35. NOC view of Firewall Approaches EPFW = End-Point Firewall LFW = Logical Firewall w/masquerading NAT SFW = Subnet Firewall BZFW = Border or Zone Firewall P172 = Project 172-phase III (Private addresses with NAT) IDEAL EPFW LFW P172 SFW BZFW Policy Enforcement Point? Host Host Subnet Zone Subnet Zone Requires host reconfigure? No Yes Yes Yes No No Requires network reconfig? No No No No Yes Yes Destroys E2E transparency? No No No No Yes Yes Assured NOC access to switches? Yes Yes Yes Yes No* No* User sees why app failed? Yes Yes No No No No NOC-Predictable semantics? Yes No No Yes No No Inherent "unlisted number"? - No Yes Yes No No "unlisted number" possible? Yes Yes Yes Yes Yes Yes Adverse impact on internal network troubleshooting: Low Low Med Med High Low Adverse impact on external network troubleshooting: Low Low Med Med High High Size of vulnerability zone: Small Small Med Large Med Large * Can be mitigated by proper access lists and/or OOB connectivity

36. 1980 1985 1990 1995 2000 Source: Network Security Trends “stealth” / advanced scanning techniques Blendedattacks High denial of service DDOS attacks sniffers www attacks automated probes/scans back doors packet spoofing Attack Sophistication disabling audits hijacking sessions burglaries exploiting known vulnerabilities password cracking self-replicating code password guessing Low

37. Impact of Recent Security Events • more perimeter firewalls (demise of open Internet, NUM) • more VPNs • more tunneling (“firewall friendly” apps) • more encryption (thanks to RIAA) • more collateral damage (from attacks & remedies) • worse MTTR (complexity, broken tools) • constrained innovation (e.g. p2p, voip) • cost shifted from “guilty” to “innocent” • pressure to fix computer security problems in network • pressure for private nets • pressure to make network topology match org boundaries • blaster: triggered more perimeter defense, but showed weakness of conventional perimeter defense