ESnet Planning for the LHC T0-T1 Networking

1. ESnet Planning for theLHC T0-T1 Networking William E. Johnston ESnet Manager and Senior Scientist Lawrence Berkeley National Laboratory

2. ESnet High-Speed Physical Connectivity toDOE Facilities and Collaborators, Summer 2005 SINet (Japan) Japan – Russia (BINP) Australia CA*net4 Taiwan (TANet2) Singaren CA*net4 France GLORIAD Kreonet2 MREN Netherlands StarTap TANet2 Taiwan (ASCC) CERN (LHCnet – partDOE funded) GEANT - Germany, France, Italy, UK, etc PNNL PNWGPoP NERSC SLAC BNL MIT ANL LIGO INEEL LLNL LBNL MAN LANAbilene SNLL TWC JGI Starlight 4xLAB-DC GTN&NNSA INEEL-DC ORAU-DC LLNL/LANL-DC Chi NAP PPPL FNAL AMES JLAB ORNL SRS SNV SDN HUB SNV SDN HUB LANL SNLA DOE-ALB PANTEX NOAA ORAU OSTI ARM YUCCA MT BECHTEL-NV GA Abilene Abilene Abilene Abilene MAXGPoP Allied Signal KCP SDSC HUB ELP HUB ALB HUB NYC HUB CHI HUB DC HUB ATL HUB SoXGPoP NREL ESnet Science Data Network (SDN) core SEA HUB ESnet IP core CHI-SL HUB QWEST ATM MAE-E SNV HUB Equinix PAIX-PA Equinix, etc. 42 end user sites Office Of Science Sponsored (22) NNSA Sponsored (12) International (high speed) 10 Gb/s SDN core 10G/s IP core 2.5 Gb/s IP core MAN rings (≥ 10 G/s) OC12 ATM (622 Mb/s) OC12 / GigEthernet OC3 (155 Mb/s) 45 Mb/s and less Joint Sponsored (3) Other Sponsored (NSF LIGO, NOAA) ESnet IP core: Packet over SONET Optical Ring and Hubs Laboratory Sponsored (6) peering points SND core hubs IP core hubs SNV HUB high-speed peering points

3. ESnet • The IP core is primarily a layer 3 infrastructure • However, supports layer 2 via MPLS • Directly connects sites • Provides global peering for sites • The SDN core is primarily a layer 2 infrastructure • Targeted at providing virtual circuit services

4. ESnet Target Architecture: IP Core + Science Data Network + MANs CERN Asia-Pacific GEANT (Europe) ESnet Science Data Network (2nd Core) Seattle (SEA) Aus. Chicago (CHI) New York(AOA) MetropolitanArea Rings Core loops Washington, DC (DC) Sunnyvale(SNV) ESnetIP Core Aus. Atlanta (ATL) Albuquerque (ALB) Existing IP core hubs El Paso (ELP) SDN hubs Production IP core Science Data Network core Metropolitan Area Networks Lab supplied International connections New hubs Primary DOE Labs Possible new hubs

5. ESnet MAN Architecture (e.g. Chicago) T320 T320 monitor monitor core router R&E peerings ESnet production IP core International peerings core router ESnet SDN core switches managingmultiple lambdas Qwest Starlight ESnet managedλ / circuit services ESnet managedλ / circuit services tunneled through the IP backbone ESnet management and monitoring 2-4 x 10 Gbps channels ESnet production IP service ANL FNAL site equip. Site gateway router site equip. Site gateway router Site LAN Site LAN

6. The First ESnet MAN: SF Bay Area Seattle and Chicago ESnet MAN ring Chicago • 2 λs (2 X 10 Gb/s channels) in a ring configuration, and delivered as 10 GigEther circuits • Dual site connection (independent “east” and “west” connections) to each site • Will be used as a 10 Gb/s production IP ring and2 X 10 Gb/s paths (for circuit services) to each site • Qwest contract signed for two lambdas 2/2005 with options on two more • Project completion date is 9/2005. λ4 future Joint Genome Institute LBNL λ3 future NERSC λ2 SDN/circuits λ1 production IP SF Bay Area LLNL SNLL SLAC Qwest /ESnet hub Level 3hub Qwest-ESnetnational core ring National Lambda Rail circuits LA andSan Diego El Paso

7. ESnet Near-Term Planning for FNAL T320 T320 CERN ORNL OC192 NRL,UltraScienceNet, etc. ESnet IP core ESnet SDN core IWire Qwest hub (NBC Bld.) Starlight Notes FNAL Qwest FNAL FNAL Shared w/ IWire All circuits are 10Gb/s ESnet ESnet Switch/RTR FNAL ESnet site gateway router ESnet/Qwest site equip. ESnet fiber

8. ESnet Planning for BNL (Long Island MAN Ring) Other connections Other connections Engineering Study for LI MAN

9. Proposed ESnet Lambda InfrastructureBased on National Lambda Rail – FY08 Seattle Boise Clev Chicago New York Denver Sunnyvale KC Pitts Wash DC Raleigh Tulsa LA Albuq. Phoenix San Diego Atlanta Dallas Jacksonville El Paso - Las Cruces Pensacola Baton Rouge Houston San Ant. NLR regeneration / OADM sites NLR wavegear sites

10. ESnet Goal – 2007/2008 Major DOE Office of Science Sites AsiaPac • 10 Gbps enterprise IP traffic • 40-60 Gbps circuit based transport SEA CERN Aus. Europe Europe ESnet Science Data Network (2nd Core – 30-50 Gbps,National Lambda Rail) Japan Japan CHI SNV NYC DEN DC MetropolitanAreaRings Aus. ESnet IP Core (≥10 Gbps) ALB ATL SDG ESnet hubs New ESnet hubs ELP Metropolitan Area Rings High-speed cross connects with Internet2/Abilene 10Gb/s 10Gb/s 30Gb/s40Gb/s Production IP ESnet core Science Data Network core Lab supplied Major international

11. OSCARS: Guaranteed Bandwidth Service • A high-priority R&D topic in “DOE Science Networking Challenge: Roadmap to 2008” • The MICS funded OSCARS project is implementing dynamically provisioned circuit switching in ESnet • On-demand Secure Circuits and Advance Reservation System (OSCARS) • End-to-end provisioning will be initially provided by a combination of Ethernet switch management of λ paths in the MAN and Ethernet VLANs and/or MPLS paths in the ESnet core • Provisioning will initially be provided by manual circuit configuration, on-demand in the future • A technology collaboration with Internet2/Abilene

12. OSCARS: Guaranteed Bandwidth Service • Progress • Testing OSCARS Label Switched Paths (MPLS based virtual circuits) • A static LSP between BNL and FNAL has been configured for the sites to test their infrastructure • Multiple static LSPs have been configured between GA and NERSC to test local and wide-area QoS • Collaboration – code is being jointly developed with Internet2's Bandwidth Reservation for User Work (BRUW) project

13. OSCARS: Guaranteed Bandwidth Service bandwidthbroker allocationmanager authorization resource manager policer usersystem1 shaper resource manager site A resource manager resource manager usersystem2 policer site B