1 / 23

ESnet Update ESnet/Internet2 Joint Techs Madison, Wisconsin July 17, 2007

Joe Burrescia ESnet General Manager Lawrence Berkeley National Laboratory. ESnet Update ESnet/Internet2 Joint Techs Madison, Wisconsin July 17, 2007. Outline. ESnet’s Role in DOE’s Office of Science ESnet’s Continuing Evolutionary Dimensions Capacity Reach Reliability

navid
Download Presentation

ESnet Update ESnet/Internet2 Joint Techs Madison, Wisconsin July 17, 2007

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Joe Burrescia ESnet General ManagerLawrence Berkeley National Laboratory ESnet UpdateESnet/Internet2 Joint Techs Madison, Wisconsin July 17, 2007

  2. Outline • ESnet’s Role in DOE’s Office of Science • ESnet’s Continuing Evolutionary Dimensions • Capacity • Reach • Reliability • Guaranteed Services

  3. DOE Office of Science and ESnet • “The Office of Science is the single largest supporter of basic research in the physical sciences in the United States, … providing more than 40 percent of total funding … for the Nation’s research programs in high-energy physics, nuclear physics, and fusion energy sciences.” (http://www.science.doe.gov) • The large-scale science that is the mission of the Office of Science is dependent on networks for • Sharing of massive amounts of data • Supporting thousands of collaborators world-wide • Distributed data processing • Distributed simulation, visualization, and computational steering • Distributed data management • ESnet’s mission is to enable those aspects of science that depend on networking and on certain types of large-scale collaboration

  4. The Office of Science U.S. Community Institutions supported by SC Major User Facilities DOE Specific-Mission Laboratories DOE Program-Dedicated Laboratories DOE Multiprogram Laboratories Pacific Northwest National Laboratory Idaho National Laboratory Ames Laboratory Argonne National Laboratory Brookhaven National Laboratory Fermi National Accelerator Laboratory Lawrence Berkeley National Laboratory Stanford Linear Accelerator Center Princeton Plasma Physics Laboratory Lawrence Livermore & Sandia National Laboratories Thomas Jefferson National Accelerator Facility General Atomics Oak Ridge National Laboratory Los Alamos National Laboratory Sandia National Laboratory National Renewable Energy Laboratory SC Program sites

  5. Footprint of SC Collaborators - Top 100 Traffic Generators Universities and research institutes that are the top 100 ESnet users • The top 100 data flows generate 30% of all ESnet traffic (ESnet handles about 3x109 flows/mo.) • 91 of the top 100 flows are from the Labs to other institutions (shown) (CY2005 data)

  6. Changing Science Environment  New Demands on Network • Increased capacity • Needed to accommodate a large and steadily increasing amount of data that must traverse the network • High-speed, highly reliable connectivity between Labs and US and international R&E institutions • To support the inherently collaborative, global nature of large-scale science • High network reliability • For interconnecting components of distributed large-scale science • New network services to provide bandwidth guarantees • For data transfer deadlines for remote data analysis, real-time interaction with instruments, coupled computational simulations, etc.

  7. Network Utilization ESnet Accepted Traffic (Bytes)Jan 1990 to Jun 2006 ESnet is Currently Transporting over 1.2 Petabytes/monthand this volume is increasing exponentially 1.20 Petabyte/mo June 2006 1.04 Petabyte/mo April 2006 TBytes/Month

  8. ESnet traffic has increased by10X every 47 months, on average, since 1990 Apr., 2006 1 PBy/mo. Nov., 2001 100 TBy/mo. 53 months Jul., 1998 10 TBy/mo. 40 months Oct., 1993 1 TBy/mo. 57 months Terabytes / month Aug., 1990 100 MBy/mo. 38 months Log Plot of ESnet Monthly Accepted Traffic, January, 1990 – June, 2006

  9. High Volume Science Traffic Continues to Grow • Top 100 flows are increasing as a percentage of total traffic volume • 99% to 100% of top 100 flows are science data (100% starting mid-2005) • A small number of large-scale science users account for a significant and growing fraction of total traffic volume 1/05 2 TB/month 6/05 2 TB/month 1/06 2 TB/month 7/06 2 TB/month

  10. Who Generates ESnet Traffic? ESnet Inter-Sector Traffic Summary for June 2006 76% 7% Commercial DOE is a net supplier of data because DOE facilities are used by universities and commercial entities, as well as by DOE researchers 5% ESnet ~7% 30% R&E (mostlyuniversities) DOE sites 5% Peering Points 31% 31% International(almost entirelyR&E sites) DOE collaborator traffic, inc. data 14% Traffic coming into ESnet = Green Traffic leaving ESnet = Blue Traffic between ESnet sites % = of total ingress or egress traffic Traffic notes • more than 90% of all traffic is Office of Science • less that 10% is inter-Lab

  11. ESnet’s Domestic, Commercial, and International Connectivity SINet (Japan) Russia (BINP) GÉANT - France, Germany, Italy, UK, etc PacificWave PNWGPoP/PacificWave SEA MAN LANAbilene CHI-SL Starlight Equinix Equinix UNM PAIX-PA Equinix MAE-West SNV NGIX-W NGIX-E USN USN Abilene Abilene Abilene Abilene MAXGPoP SoXGPoP High Speed International Connection Commercial and R&E peering points ESnet core hubs IP SDN High-speed peering points with Abilene Japan (SINet) Australia (AARNet) Canada (CA*net4 Taiwan (TANet2) Singaren CA*net4 France GLORIAD (Russia, China)Korea (Kreonet2) MREN Netherlands StarTapTaiwan (TANet2) UltraLight CERN (USLHCnetCERN+DOE funded) Australia NYC MAE-E CHI SNV DC ATL SDSC ALB Australia ESnet provides: • High-speed peerings with Abilene, CERN, and the international R&E networks • Management of the full complement of global Internet routes (about 180,000 unique IPv4 routes) in order to provide DOE scientists rich connectivity to all Internet sites AMPATH (S. America) AMPATH S. America

  12. ESnet’s Physical Connectivity (Summer 2006) SINet (Japan) Russia (BINP) GÉANT - France, Germany, Italy, UK, etc PNNL SEA SLAC NERSC BNL ANL MIT INEEL LIGO LBNL LLNL MAN LANAbilene SNLL CHI-SL LVK JGI Starlight OSC GTNNNSA LBNL DC LLNL/LANL DC Offices ORAU DC Chi NAP PPPL AMES FNAL JLAB ORNL SRS SNLA LANL DC USN USN DOE-ALB KCP-ALB NASAAmes PANTEX ORAU NOAA OSTI ARM IARC YUCCA MT BECHTEL-NV GA Abilene Abilene Abilene Abilene MAXGPoP KCP SoXGPoP NREL SNV Japan (SINet) Australia (AARNet) Canada (CA*net4 Taiwan (TANet2) Singaren CA*net4 France GLORIAD (Russia, China)Korea (Kreonet2 MREN Netherlands StarTapTaiwan (TANet2) UltraLight PNWGPoP/PAcificWave CERN (USLHCnetCERN+DOE funded) NLR Supplied Circuits AU ESnet IP core MAN rings NYC MAE-E SNV CHI Equinix PAIX-PA Equinix, etc. SNV SDN ATL SDSC AU ALB AMPATH UNM 42 end user sites AMPATH (S. America) ELP Office Of Science Sponsored (22) NNSA Sponsored (12) International (high speed) Lab Supplied 10 Gb/s SDN core 10G/s IP core 2.5 Gb/s IP core MAN rings (≥ 10 G/s) OC12 / GigEthernet OC3 (155 Mb/s) 45 Mb/s and less Joint Sponsored (4) Other Sponsored (NSF LIGO, NOAA) ESnet IP core: Packet over SONET Optical Ring and Hubs Laboratory Sponsored (6) commercial and R&E peering points ESnet core hubs IP high-speed peering points with Internet2/Abilene

  13. ESnet LIMAN with SDN Connections

  14. LIMAN and BNL • ATLAS (A Toroidal LHC ApparatuS), is one of four detectors located at the Large Hadron Collider (LHC) located at CERN • BNL is the largest of the ATLAS Tier 1 centers and the only one in the U.S, and so is responsible for archiving and processing approximately 20 percent of the ATLAS raw data • During a recent multi-week exercise, BNL was able to sustain an average transfer rate from CERN to their disk arrays of 191 MB/s (~1.5 Gb/s) compared to a target rate of 200 MB/s • This was in addition to “normal” BNL site traffic

  15. Chicago Area MAN with SDN Connections

  16. CHIMAN: FNAL and ANL • Fermi National Laboratory is the only US Tier1 center for the Compact Muon Solenoid (CMS) experiment at LHC • Argonne National Laboratory will house a 5-teraflop IBM BlueGene computer part of the National Leadership Computing Facility • Together with ESnet, FNAL and ANL will build the Chicago MAN (CHIMAN) to accommodate the vast amounts of data these facilities will generate and receive • Five 10GE circuits will go into FNAL • Three 10GE circuits will go into ANL • Ring connectivity to StarLight and to the Chicago ESnet POP

  17. Jefferson Laboratory Connectivity MATP Virginia Tech MAX GIGAPOP DC - MAX Giga-POP MATP NYC Lovitt ESnet core Bute St CO ODU Eastern LITE (E-LITE) Old Dominion University ESnet Router NASA Atlanta VMASC JLAB Site Switch 10GE OC192 OC48 W&M JLAB JTASC

  18. ESnet Target Architecture: IP Core+Science Data Network Core+Metro Area Rings Europe (GEANT) CERN Canada (CANARIE) Canada (CANARIE) Asia-Pacific CERN Europe (GEANT) Seattle SDN Core Chicago Australia New York Denver IP Core Sunnyvale Washington, DC MetropolitanArea Rings Atlanta Loop off Backbone LA Albuquerque Aus. South America (AMPATH) San Diego IP core hubs 10-50 Gb/s circuits Production IP core Science Data Network core Metropolitan Area Networks International connections South America (AMPATH) SDN hubs Primary DOE Labs possible hubs

  19. Reliability “5 nines” “4 nines” “3 nines” Dually connected sites

  20. Guaranteed Services Using Virtual Circuits • Traffic isolation and traffic engineering • Provides for high-performance, non-standard transport mechanisms that cannot co-exist with commodity TCP-based transport • Enables the engineering of explicit paths to meet specific requirements • e.g. bypass congested links, using lower bandwidth, lower latency paths • Guaranteed bandwidth [Quality of Service (QoS)] • Addresses deadline scheduling • Where fixed amounts of data have to reach sites on a fixed schedule, so that the processing does not fall far enough behind that it could never catch up – very important for experiment data analysis • Reduces cost of handling high bandwidth data flows • Highly capable routers are not necessary when every packet goes to the same place • Use lower cost (factor of 5x) switches to relatively route the packets • End-to-end connections are required between Labs and collaborator institutions

  21. OSCARS: Guaranteed Bandwidth VC Service For SC Science • ESnet On-demand Secured Circuits and Advanced Reservation System (OSCARS) • To ensure compatibility, the design and implementation is done in collaboration with the other major science R&E networks and end sites • Internet2: Bandwidth Reservation for User Work (BRUW) • Development of common code base • GEANT: Bandwidth on Demand (GN2-JRA3), Performance and Allocated Capacity for End-users (SA3-PACE) and Advance Multi-domain Provisioning System (AMPS) • Extends to NRENs • BNL: TeraPaths - A QoS Enabled Collaborative Data Sharing Infrastructure for Peta-scale Computing Research • GA: Network Quality of Service for Magnetic Fusion Research • SLAC: Internet End-to-end Performance Monitoring (IEPM) • USN: Experimental Ultra-Scale Network Testbed for Large-Scale Science • In its current phase this effort is being funded as a research project by the Office of Science, Mathematical, Information, and Computational Sciences (MICS) Network R&D Program • A prototype service has been deployed as a proof of concept • To date more then 20 accounts have been created for beta users, collaborators, and developers • More then 100 reservation requests have been processed

  22. OSCARS - BRUW Interdomain Interoperability Demonstration LSP LSP Sunnyvale CA Chicago IL Chicago IL Indianapolis IN 3 3 2 BRUW OSCARS 4 1 Source Sink • The first interdomain, automatically configured, virtual circuit between ESnet and Abilene was created on April 6, 2005

  23. A Few URLs • ESnet Home Page • http://www.es.net • National Labs and User Facilities • http://www.sc.doe.gov/sub/organization/organization.htm • ESnet Availability Reports • http://calendar.es.net/ • OSCARS Documentation • http://www.es.net/oscars/index.html

More Related