ESNet Update ESCC, July 21, 2004

1. ESNet UpdateESCC, July 21, 2004 William E. Johnston, ESnet Dept. Head and Senior Scientist R. P. Singh, Federal Project Manager Michael S. Collins, Stan Kluz,Joseph Burrescia, and James V. Gagliardi, ESnet Leads and the ESnet Team Lawrence Berkeley National Laboratory

2. Asia-Pacific Europe Chicago (CHI) DOE sites New York (AOA) ESnetCore/Backbone Washington, DC (DC) Sunnyvale (SNV) Atlanta (ATL) El Paso (ELP) ESnet New Architecture Goal • MAN rings provide dual site and hub connectivity • A second backbone ring will multiply connect the MAN rings to protect against hub failure

3. First Step: SF Bay Area ESnet MAN Ring Seattle and Chicago • Increased reliability and site connection bandwidth • Phase 1 • Connects the primary Office of Science Labs in a MAN ring • Phase 2 • LLNL, SNL, andUC Merced • Ring must connect directly into SNV Level 3 hub (still working on physical routing for this) • Have not yet identified both legs of the mini ring Chicago Joint Genome Institute LBNL NERSC SF BA MAN ring topology – phase 1 SF Bay Area mini ring SLAC Level 3hub Qwest /ESnet hub NLR / UltraScienceNet Existing ESnet Core Ring LA and San Diego El Paso

4. Traffic Growth Continues ESnet Monthly Accepted Traffic ESnet is currently transporting about 250 terabytes/mo. TBytes/Month Annual growth in the past five years has increased from 1.7x annually to just over 2.0x annually.

5. Who Generates Traffic, and Where Does it Go? ESnet Inter-Sector Traffic Summary,Jan 2003 / Feb 2004 (1.7X overall traffic increase, 1.9X OSC increase)(the international traffic is increasing due to BABAR at SLAC and the LHC tier 1 centers at FNAL and BNL) 72/68% 21/14% Commercial DOE is a net supplier of data because DOE facilities are used by universities and commercial entities, as well as by DOE researchers 14/12% ESnet ~25/18% 17/10% R&E (mostlyuniversities) DOE sites 10/13% Peering Points 53/49% 9/26% International DOE collaborator traffic, inc.data 4/6% Note that more that 90% of the ESnet traffic is OSC traffic ESnet Appropriate Use Policy (AUP) All ESnet traffic must originate and/or terminate on an ESnet an site (no transit traffic is allowed) Traffic coming into ESnet = Green Traffic leaving ESnet = Blue Traffic between sites % = of total ingress or egress traffic

6. ESnet Top 20 Data Flows, 24 hrs., 2004-04-20 A small number of science users account for a significant fraction of all ESnet traffic SLAC (US)  IN2P3 (FR) 1 terabyte/day Fermilab (US) CERN SLAC (US) INFN Padva (IT) Fermilab (US)  U. Chicago (US) U. Toronto (CA)  Fermilab (US) CEBAF (US)  IN2P3 (FR) INFN Padva (IT)  SLAC (US) DFN-WiN (DE)  SLAC (US) Fermilab (US)  JANET (UK) SLAC (US)  JANET (UK) DOE Lab  DOE Lab Argonne (US)  Level3 (US) DOE Lab  DOE Lab Fermilab (US)  INFN Padva (IT) Argonne  SURFnet (NL) IN2P3 (FR)  SLAC (US)

7. Top 50 Traffic Flows Monitoring – 24hr 2 Int’l and 2 Commercial Peering Points 10 flows> 100 GBy/day More than 50 flows> 10 GBy/day

8. BNL LBNL TWC PPPL AMES ELP HUB SNV HUB CHI HUB NYC HUBS DC HUB ATL HUB SEA HUB ALB HUB Disaster Recovery and Stability • Engineers, 24x7 Network Operations Center, generator backed power • Spectrum (net mgmt system) • DNS (name – IP address translation) • Eng database • Load database • Config database • Public and private Web • E-mail (server and archive) • PKI cert. repository and revocation lists • collaboratory authorization service • Remote Engineer • partial duplicate infrastructure DNS Remote Engineer Duplicate Infrastructure Currently deploying full replication of the NOC databases and servers and Science Services databases in the NYC Qwest carrier hub • Remote Engineer • partial duplicate infrastructure • The network must be kept available even if, e.g., the West Coast is disabled by a massive earthquake, etc. • high physical security for all equipment • non-interruptible core - ESnet core operated without interruption through • N. Calif. Power blackout of 2000 • the 9/11/2001 attacks, and • the Sept., 2003 NE States power blackout • Reliable operation of the network involves • remote Network Operation Centers (3) • replicated support infrastructure • generator backed UPS power at all critical network and infrastructure locations

9. Disaster Recovery and Stability • Duplicate NOC infrastructure to AoA hub in two phases, complete by end of the year • 9 servers – dns, www, www-eng and noc5 (eng. databases), radius, aprisma (net monitoring), tts (trouble tickets), pki-ldp (certificates), mail

10. Maintaining Science Mission Critical Infrastructurein the Face of Cyberattack • A Phased Response to Cyberattack is being implemented to protects the network and the ESnet sites • The phased response ranges from blocking certain site traffic to a complete isolation of the network which allows the sites to continue communicating among themselves in the face of the most virulent attacks • Separates ESnet core routing functionality from external Internet connections by means of a “peering” router that can have a policy different from the core routers • Provide a rate limited path to the external Internet that will insure site-to-site communication during an external denial of service attack • Provide “lifeline” connectivity for downloading of patches, exchange of e-mail and viewing web pages (i.e.; e-mail, dns, http, https, ssh, etc.) with the external Internet prior to full isolation of the network

11. Phased Response to Cyberattack ESnet third response – shut down the main peering paths and provide only limited bandwidth paths for specific “lifeline” services ESnet first response – filters to assist a site ESnet second response – filter traffic from outside of ESnet peeringrouter X X router ESnet router LBNL attack traffic router X borderrouter • Lab first response – filter incoming traffic at their ESnet gateway router gatewayrouter peeringrouter border router Lab gatewayrouter Lab • Sapphire/Slammer worm infection created a Gb/s of traffic on the ESnet core until filters were put in place (both into and out of sites) to damp it out.

12. Phased Response to Cyberattack

13. Grid Middleware Services • ESnet is the natural provider for some “science services” – services that support the practice of science • ESnet is trusted, persistent, and has a large (almost comprehensive within DOE) user base • ESnet has the facilities to provide reliable access and high availability through assured network access to replicated services at geographically diverse locations • However, service must be scalable in the sense that as its user base grows, ESnet interaction with the users does not grow (otherwise not practical for a small organization like ESnet to operate)

14. Grid Middleware Requirements (DOE Workshop) • A DOE workshop examined science driven requirements for network and middleware and identified twelve high priority middleware services (see www.es.net/#research) • Some of these services have a central management component and some do not • Most of the services that have central management fit the criteria for ESnet support. These include, for example • Production, federated RADIUS authentication service • PKI federation services • Virtual Organization Management services to manage organization membership, member attributes and privileges • Long-term PKI key and proxy credential management • End-to-end monitoring for Grid / distributed application debugging and tuning • Some form of authorization service (e.g. based on RADIUS) • Knowledge management services that have the characteristics of an ESnet service are also likely to be important (future)

15. Science Services: PKI Support for Grids • Public Key Infrastructure supports cross-site, cross-organization, and international trust relationships that permit sharing computing and data resources and other Grid services • DOEGrids Certification Authority service provides X.509 identity certificates to support Grid authentication provides an example of this model • The service requires a highly trusted provider, and requires a high degree of availability • Federation: ESnet as service provider is a centralized agent for negotiating trust relationships, e.g. with European CAs • The service scales by adding site based or Virtual Organization based Registration Agents that interact directly with the users • See DOEGrids CA (www.doegrids.org)

16. ESnet PKI Project • DOEGrids Project Milestones • DOEGrids CA in production June, 2003 • Retirement of initial DOE Science Grid CA (Jan 2004) • “Black rack” installation completed for DOE Grids CA (Mar 2004) • New Registration Authorities • FNAL (Mar 2004) • LCG (LHC Computing Grid) catch-all: near completion • NCC-EPA: in progress • Deployment of NERSC “myProxy” CA • Grid Integrated RADIUS Authentication Fabric pilot

17. DOEGrids Security Bro Intrusion Detection RAs andcertificate applicants PKI Systems Fire Wall HSM Internet Secure racks Secure Data Center Vaulted Root CA Building Security LBNL Site security

18. Science Services: Public Key Infrastructure • The rapidly expanding customer base of this service will soon make it ESnet’s largest collaboration service by customer count Registration Authorities ANL LBNL ORNL DOESG (DOE Science Grid) ESG (Climate) FNAL PPDG (HEP) Fusion Grid iVDGL (NSF-DOE HEP collab.) NERSC PNNL

19. ESnet PKI Project (2) • New CA initiatives: • FusionGrid CA • ESnet SSL Server Certificate CA • Mozilla browser CA cert distribution • Script-based enrollment • Global Grid Forum documents • Policy Management Authority Charter • OCSP (Online Certificate Status Protocol) Requirements For Grids • CA Policy Profiles

20. Grid Integrated RADIUS Authentication Fabric • RADIUS routing of authentication requests • Support One-Time Password initiatives • Gateway Grid and collaborative uses: standard UI and API • Provide secure federation point with O(n) agreements • Support multiple vendor / site OTP implementations • One token per user (SSO-like solution) for OTP • Collaboration between ESnet, NERSC, a RADIUS appliance vendor, PNNL and ANL are also involved, others welcome • White paper/report ~ 01 Sep 2004 to support early implementers, proceed to pilot • Project pre-proposal:http://www.doegrids.org/CA/Research/GIRAF.pdf

21. Collaboration Service • H323 showing dramatic increase in usage • new audio and video ports to be added in next 6 mo.

22. Grid Network Services Requirements (GGF, GHPN) • Grid High Performance Networking Research Group, “Networking Issues of Grid Infrastructures” (draft-ggf-ghpn-netissues-3) – what networks should provide to Grids • High performance transport for bulk data transfer (over 1Gb/s per flow) • Performance controllability to provide ad hoc quality of service and traffic isolation. • Dynamic Network resource allocation and reservation • High availability when expensive computing or visualization resources have been reserved • Security controllability to provide a trusted and efficient communication environment when required • Multicast to efficiently distribute data to group of resources. • Integrated wireless network and sensor networks in Grid environment

23. Priority Service • So, practically, what can be done? • With available tools can provide a small number of provisioned, bandwidth guaranteed, circuits • secure and end-to-end (system to system) • various Quality of Service possible, including minimum latency • a certain amount of route reliability (if redundant paths exist in the network) • end systems can manage these circuits as single high bandwidth paths or multiple lower bandwidth paths of (with application level shapers) • non-interfering with production traffic, so aggressive protocols may be used

24. Guaranteed Bandwidth as an ESNet Service allocation will probably be relatively static and ad hoc • A DOE Network R&D funded project bandwidthbroker authorization resource manager policer usersystem1 site A resource manager usersystem2 • will probably be service level agreements among transit networks allowing for a fixed amount of priority traffic – so the resource manager does minimal checking and no authorization • will do policing, but only at the full bandwidth of the service agreement (for self protection) Phase 1 resource manager usersystem2 Phase 2 site B

25. Network Monitoring System • Alarms & Data Reduction • From June 2003 through April 2004 the total number of NMS up/down alarms was 16,342 or 48.8 per day. • Path based outage reporting automatically isolated 1,448 customer relevant events during this period or an average of 4.3 per day, more than a 10 fold reduction. • Based on total outage duration in 2004, approximately 63% of all customer relevant events have been categorized as either “Planned” or “Unplanned” and one of “ESnet”, “Site”, “Carrier” or “Peer” • Gives us a better handle on availability metric

26. 2004 Availability by Month Jan. – June, 2004 – Corrected for Planned Outages(More from Mike O’Connor) <99.9%available >99.9% available Unavailable Minutes

27. 3 ESnet Participants LBL FERMI BNL 3 Abilene Participants SDSC NCSU OSU ESnet Abilene Measurements • We want to ensure that the ESnet/Abilene cross connects are serving the needs of users in the science community who are accessing DOE facilities and resources from universities or accessing university facilities from DOE labs. • Measurement sites in place: • More from Joe Metzger

28. OWAMP One-Way Delay Tests Are Highly Sensitive • NCSU Metro DWDM reroute adds about 350 micro seconds ms Fiber Re-Route 42.0 41.9 41.8 41.7 41.6 41.5

29. ESnet Trouble Ticket System • TTS used to track problem reports for the Network, ECS, DOEGrids, Asset Management, NERSC, and other services. • Running Remedy ARsystem server and Oracle database on a Sun Ultra workstation. • Total external ticket = 11750 (1995-2004), approx. 1300/year • Total internal tickets = 1300 (1999-2004), approx. 250/year

30. Network Upgrades • Feb: Yucca Mt DS3 ATM to DS3 point-to-point • Mar: NYC (60 Hudson Hub) decommishioned Qwest DIA peering moved to AOA) • Mar: Bechtel DS3 point-to-point (now waiting on there connection) • May: ESnet-Abilene Peering on OC48 at the ATL-HUB • New Router Installations / Upgrades: • Mar: Bechtel-rt1 from 3640 to 7206 • Mar: Eqx-ash-rt1 new M10 • Apr: OSTI-rt1 from 7505 to 7507 • Apr: INEEL-RT1 from 7505 to 7507 • June: PAIX M5 consolidation (removed Fore SW and 7206)

31. Network Upgrades • Mar: Equinix San Jose private peering with TELIASONERA • Mar: Equinix Ashburn in production public (March29) and 1st private peering with TELIASONERA (Apr 15) • May: Level3 peering up at Equinix Ashburn • Firmware Upgrades: We continue to test and deploy the most appropriatesoftware loads for our network devices. Because of this rapid response, no ESnet router was able to be exploited by a denial of service attack that was possible before the upgrade.

32. Conclusions • ESnet is an infrastructure that is critical to DOE’s science mission and that serves all of DOE • Focused on the Office of Science Labs • ESnet is working on providing the DOE mission science networking requirements with several new initiatives and a new architecture • QoS service is hard – but we believe that we have enough experience to do pilot studies • Middleware services for large numbers of users are hard – but they can be provided if careful attention is paid to scaling