1 / 52

Networks for HENP and ICFA SCIC

Networks for HENP and ICFA SCIC. Harvey B. Newman California Institute of Technology CHEP2003, San Diego March 24, 2003. Global Networks for HENP Circa 2003. National and International Networks, with sufficient (and rapidly increasing) capacity and capability, are essential for

mahon
Download Presentation

Networks for HENP and ICFA SCIC

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Networks for HENP and ICFA SCIC Harvey B. Newman California Institute of TechnologyCHEP2003, San DiegoMarch 24, 2003

  2. Global Networks for HENPCirca 2003 • National and International Networks, with sufficient (and rapidly increasing) capacity and capability, are essential for • Data analysis, and the daily conduct of collaborative work in both experiment and theory, Involving physicists from all world regions • Detector development & construction on a global scale • The formation of worldwide collaborations • The conception, design and implementation of next generation facilities as “global networks” • “Collaborations on this scale would never have been attempted, if they could not rely on excellent networks” (L. Price)

  3. Next Generation Networks for Experiments: Goals and Needs • Providing rapid access to event samples and analyzed physics results drawn from massive data stores • From Petabytes by 2002, ~100 Petabytes by 2007, to ~1 Exabyte by ~2012. • Providing analyzed results with rapid turnaround, bycoordinating and managing large but LIMITED computing, data handling and NETWORKresources effectively • Enabling rapid access to the Data and the Collaboration • Across an ensemble of networks of varying capability • Advanced integrated applications, such as Data Grids, rely on seamless operation of our LANs and WANs • With reliable, monitored, quantifiable high performance Large data samples explored and analyzed by thousands of globally dispersed scientists, in hundreds of teams

  4. ICFA and International Networking • ICFA Statement on Communications in Int’l HEPCollaborations of October 17, 1996 See http://www.fnal.gov/directorate/icfa/icfa_communicaes.html “ICFA urges that all countries and institutions wishing to participate even more effectively and fully in international HEP Collaborations should: • Review their operating methods to ensure they are fully adapted to remote participation • Strive to provide the necessary communications facilities and adequate international bandwidth”

  5. ICFA Network Task Force: 1998 Bandwidth Req’ments Projection (Mbps) NTF 100–1000 X Bandwidth Increase Foreseen for 1998-2005 See the ICFA-NTF Requirements Report: http://l3www.cern.ch/~newman/icfareq98.html

  6. ICFA Standing Committee on Interregional Connectivity (SCIC) • Created by ICFA in July 1998 in Vancouver ; Following ICFA-NTF • CHARGE: Make recommendations to ICFA concerning the connectivity between the Americas, Asia and Europe (and network requirements of HENP) • As part of the process of developing theserecommendations, the committee should • Monitor traffic • Keep track of technology developments • Periodically review forecasts of future bandwidth needs, and • Provide early warning of potential problems • Create subcommittees when necessary to meet the charge • Representatives: Major labs, ECFA, ACFA, NA Users, S. America • The chair of the committee should report to ICFA once peryear, at its joint meeting with laboratory directors (Feb. 2003)

  7. Bandwidth Growth of Global HENP Networks • Rate of Progress >> Moore’s Law. (US-CERN Example) • 9.6 kbps Analog (1985) • 64-256 kbps Digital (1989 - 1994) [X 7 – 27] • 1.5 Mbps Shared (1990-3; IBM) [X 160] • 2 -4 Mbps (1996-1998) [X 200-400] • 12-20 Mbps (1999-2000) [X 1.2k-2k] • 155-310 Mbps (2001-2) [X 16k – 32k] • 622 Mbps (2002-3) [X 65k] • 2.5 Gbps  (2003-4) [X 250k] • 10 Gbps  (2005) [X 1M] • A factor of ~1M over a period of 1985-2005 (a factor of ~5k during 1995-2005) • HENP has become a leading applications driver, and also a co-developer of global networks

  8. CERN/Outside Resource Ratio ~1:2Tier0/( Tier1)/( Tier2) ~1:1:1 ~PByte/sec ~100-1500 MBytes/sec Online System Experiment CERN Center PBs of Disk; Tape Robot Tier 0 +1 Tier 1 ~2.5-10 Gbps FNAL Center IN2P3 Center INFN Center RAL Center 2.5-10 Gbps Tier 2 Tier2 Center Tier2 Center Tier2 Center Tier2 Center Tier2 Center ~2.5-10 Gbps Tier 3 Institute Institute Institute Institute Tens of Petabytes by 2007-8.An Exabyte ~5-7 Years later. Physics data cache 0.1 to 10 Gbps Tier 4 Workstations LHC Data Grid Hierarchy Emerging Vision: A Richly Structured, Global Dynamic System

  9. 2001 Transatlantic Net WG Bandwidth Requirements [*] [*] See http://gate.hep.anl.gov/lprice/TAN. The 2001LHC requirements outlook now looks Very Conservative in 2003

  10. History – One large Research Site Much of the Traffic:SLAC IN2P3/RAL/INFN;via ESnet+France;Abilene+CERN Current Traffic ~400 Mbps;Projections: 0.5 to 24 Tbps by ~2012

  11. LHC: Tier0-Tier1 Link Requirements Estimate: for Hoffmann Report 2000-1 • Tier1  Tier0 Data Flow for Analysis 0.5 - 1.0 Gbps • Tier2  Tier0 Data Flow for Analysis 0.2 - 0.5 Gbps • Interactive Collaborative Sessions [30 Peak] 0.1 - 0.3 Gbps • Remote Interactive Sessions [30 Peak] 0.1 - 0.2 Gbps • Individual (Tier3 or Tier4) data transfers 0.8 Gbps[Limit to 10 Flows of 5 Mbytes/sec each] • TOTAL Per Tier0 - Tier1 Link 1.7 - 2.8 Gbps • Does Not Include More Recent (e.g. ATLAS) Data Estimates • Rates 270-400 Hz, Event Size 2 MB/Event • Does Not Allow Fast Download to Tier3+4 of Many “Small” Object Collections • Example: Download 107 Events of AODs (104 Bytes)  100 Gbytes; at 5 Mbytes/sec per person that’s 6 Hours ! • This is a still a rough, bottoms-up, static, and hence Conservative Model. • A Dynamic Grid System may well require greater bandwidth

  12. 2003 NSF ITRs: Globally EnabledAnalysis Communities & Collaboratories • Develop and build Dynamic Workspaces • Construct Autonomous Communities Operating Within Global Collaborations • Build Private Grids to support scientific analysis communities • e.g. Using Agent Based Peer-to-peer Web Services • Drive the democratization of science via the deployment of new technologies • Empower small groups of scientists (Teachers and Students) to profit from and contribute to int’l big science

  13. Private Grids and Peer-to-PeerSub-Communities in Global HEP

  14. SCIC in 2002-3A Period of Intense Activity • Formed WGs in March 2002; 9 Meetings in 12 Months • Strong Focus on the Digital Divide • Presentations at Meetings and Workshops(e.g. LISHEP, APAN, AMPATH, ICTP and ICFA Seminars) • HENP more visible to governments: in the WSIS Process Five Reports; Presented to ICFA Feb. 13,2003See http://cern.ch/icfa-scic • Main Report: “Networking for HENP” [H. Newman et al.] • Monitoring WG Report [L. Cottrell] • Advanced Technologies WG Report [R. Hughes-Jones, O. Martin et al.] • Digital Divide Report [A. Santoro et al.] • Digital Divide in Russia Report [V. Ilyin]

  15. SCIC in 2002-3A Period of Intensive Activity Web Page http://cern.ch/ICFA-SCIC/ • Monitoring:Les Cottrell (SLAC) (http://www.slac.stanford.edu/xorg/icfa/scic-netmon) With Richard Hughes-Jones (Manchester), Sergio Novaes (Sao Paolo); Sergei Berezhnev (RUHEP), Fukuko Yuasa (KEK), Daniel Davids (CERN), Sylvain Ravot (Caltech), Shawn McKee (Michigan) • Advanced Technologies:R. Hughes-Jones, Olivier Martin (CERN) With Vladimir Korenkov (JINR, Dubna), H. Newman • The Digital Divide:Alberto Santoro (Rio, Brazil) • With V. Ilyin (MSU), Y. Karita(KEK), D.O. Williams (CERN) • Also V. White (FNAL), J. Ibarra and H. Alvarez (AMPATH),D. Son (Korea), H. Hoorani, S. Zaidi (Pakistan), S. Banerjee (India), • Key Requirements: Harvey Newman and Charlie Young (SLAC)

  16. HENP Networks: Status and Outlook: SCIC General Conclusions • The scale and capability of networks, their pervasiveness and range of applications in everyday life, and HENP’s dependence on networks for its research, are all increasing rapidly. • However, as the pace of network advances continues to accelerate, the gap between the economically “favored” regions and the rest of the world is in danger of widening. • We must therefore workto Close the Digital Divide • To make Physicists from All World Regions Full Partners in Their Experiments; and in the Process of Discovery • This is essential for the health of our global experimental collaborations, our plans for future projects, and our field.

  17. ICFA SCIC: R&E Backbone and International Link Progress • GEANT Pan-European Backbone (http://www.dante.net/geant) • Now interconnects >31 countries; many trunks 2.5 and 10 Gbps • UK: SuperJANET Core at 10 Gbps • 2.5 Gbps NY-London, with 622 Mbps to ESnet and Abilene • France (IN2P3): 2.5 Gbps RENATER3 backbone from October 2002 • Lyon-CERN Link Upgraded to 1 Gbps Ethernet • Plan for dark fiber to CERN by end 2003 • SuperSINET (Japan):10 Gbps IP and 10 Gbps Wavelength Core • Tokyo to NY Links: 2 X 2.5 Gbps started • CA*net4 (Canada): Interconnect customer-owned dark fiber nets across Canada at 10 Gbps, started July 2002 • “Lambda-Grids” by ~2004-5 • GWIN (Germany):2.5 Gbps Core; Connect to US at 2 X 2.5 Gbps;Support for Virtual SILK Hwy Project: Satellite links to FSU Republics • Russia: 155 Mbps Links to Moscow (Typ. 30-45 Mbps for Science) • Moscow-Starlight Link to 155 Mbps (US NSF + Russia Support) • Moscow-GEANT and Moscow-Stockholm Links 155 Mbps

  18. R&E Backbone and Int’l Link Progress • Abilene (Internet2) Upgrade from 2.5 to 10 Gbps in 2002-3 • Encourage high throughput use for targeted applications; FAST • ESNET: Upgrade: 2.5 and 10 Gbps Links • SLAC + IN2P3 (BaBar) • Typically ~400 Mbps throughput on US-CERN, Renater links • ~600 Mbps Throughput is BaBar Target for First Half of 2003 • FNAL: ESnet Link Upgraded to 622 Mbps • Plans for dark fiber to STARLIGHT, proceeding • US-CERN 622 Mbps in production from 8/022.5G to 10G Research Triangle STARLIGHT-CERN-SURFNet(NL); [10Gbps SNV-Starlight Link Loan from Level(3) 10/02-2/03] • IEEAF Donation from Tyco: NY-Amsterdam Completed 9/02; Transpacific Donation by Mid-2003. 622 Mbps+10G Wavelength • US Nat’l Light Rail and USAWaves (10 Gbps DWDM-based Fiber Infrastructures) Proceeding this Year

  19. SuperSINET Updated Map: October 2002 • 10 GbE + 10 Gbps IP Backbone • 10 15 SuperSINET Universities with GbE • Many SINET Nodes with 30-100 Mbps

  20. 2003: OC192 and OC48 Links Coming Into Service;Upgrade Links to US HENP Labs

  21. Network Challenges and Requirements for High Throughput • Low to Extremely Low Packet Loss (<< 0.01% for standard TCP) • Need to track down uncongested packet loss • No Local Infrastructure Bottlenecks or Quality Compromises • Gigabit Ethernet and eventually some 10 GbE “clear paths” between selected host pairs • TCP/IP stack configuration and tuning Absolutely Required • Large Windows (~BW*RTT); Possibly Multiple Streams • Also need to consider Fair Sharing with Other Traffic • Careful Configuration: Routers, Servers and Client Systems • Sufficient End-system CPU and Disk I/O; NIC performance • End-to-end monitoring and tracking of performance • Close collaboration with local and “regional” network staffs New TCP Protocol Stacks Engineered for Stable Fair Operation at 1-10 Gbps; Eventually to 100 Gbps

  22. HEP is Learning How to Use Gbps Networks Fully: Factor of 25-100 Gain in Max. Sustained TCP Thruput in 15 Months, On Some US+TransAtlantic Routes • 9/01 105 Mbps 30 Streams: SLAC-IN2P3; 102 Mbps 1 Stream CIT-CERN • 1/09/02 190 Mbps for One stream shared on Two 155 Mbps links • 3/11/02 120 Mbps Disk-to-Disk with One Stream on 155 Mbps link (Chicago-CERN) • 5/20/02 450-600 Mbps SLAC-Manchester on OC12 with ~100 Streams • 6/1/02 290 Mbps Chicago-CERN One Stream on OC12 (mod. Kernel) • 9/02 850, 1350, 1900 Mbps Chicago-CERN 1,2,3 GbE Streams, 2.5G Link • 11-12/02: 930 Mbps in 1 Stream California-CERN, and California-AMS FAST TCP 9.4 Gbps in 10 Flows California-Chicago • 2/03 2.38 Gbps in 1 Stream California-Geneva (99% Link Utilization) *

  23. FAST TCP:Baltimore/Sunnyvale • RTT estimation: fine-grain timer • Fast convergence to equilibrium • Delay monitoring in equilibrium • Pacing: reducing burstiness 88% 10G 90% 9G Measurements • Std Packet Size • Utilization averaged over > 1hr • 3000 km Path 90% Average utilization 92% 8.6 Gbps; 21.6 TB in 6 Hours 95% Fair SharingFast Recovery 1 flow 2 flows 7 flows 9 flows 10 flows

  24. 10GigE Data Transfer Trial On Feb. 27-28, a Terabyte of data was transferred in 3700 seconds by S. Ravot of Caltech between the Level3 PoP in Sunnyvale near SLAC and CERN through the TeraGrid router at StarLight from memory to memoryAs a single TCP/IP stream at average rate of 2.38 Gbps. (Using large windows and 9kB “Jumbo frames”)This beat the former record by a factor of ~2.5, and used the US-CERN link at 99% efficiency. European Commission 10GigE NIC

  25. HENP Major Links: Bandwidth Roadmap (Scenario) in Gbps Continuing the Trend: ~1000 Times Bandwidth Growth Per Decade;We are Rapidly Learning to Use Multi-Gbps Networks Dynamically

  26. HENP Lambda Grids:Fibers for Physics • Problem: Extract “Small” Data Subsets of 1 to 100 Terabytes from 1 to 1000 Petabyte Data Stores • Survivability of the HENP Global Grid System, with hundreds of such transactions per day (circa 2007)requires that each transaction be completed in a relatively short time. • Example: Take 800 secs to complete the transaction. Then Transaction Size (TB)Net Throughput (Gbps) 1 10 10 100 100 1000 (Capacity of Fiber Today) • Summary: Providing Switching of 10 Gbps wavelengthswithin ~3-5 years; and Terabit Switching within 5-8 yearswould enable “Petascale Grids with Terabyte transactions”,to fully realize the discovery potential of major HENP programs, as well as other data-intensive fields.

  27. National Light Rail Footprint SEA POR SAC BOS NYC CHI OGD DEN SVL CLE WDC PIT FRE KAN RAL NAS STR LAX PHO WAL ATL SDG OLG DAL JAC 15808 Terminal, Regen or OADM site Fiber route • NLR • Buildout Started November 2002 • Initially 4 10 Gb Wavelengths • To 40 10Gb Waves in Future • Transition beginning now to optical, multi-wavelength R&E networks. • Also Note: IEEAF/GEO plan for dark fiber in Europe

  28. 1 2 3 4 5 6 7 8 Optical Packet Routing Using  ConversionD. Blumenthal, UC Santa Barbara • Optical > Electronic Switching • Microprocessor Power • Per Fiber Capacity Increases Wavelength Router Fast Wavelength Converter Packet switched to wavelength 5 Packet switched to wavelength 2 Packets at Wavelength 1 and7 Control Signals Fast Tunable Laser • Circuit Switched Mode • Burst Mode • Packet Mode

  29. Incoming Packets Packets Routed to 1542.0 nm Packets Routed to 1556.0 nm Packets Routed to 1548.0 nm 1st Hop 2nd Hop 80 Gbps Optical Packet Routing with Label Swapping Results (UCSB)

  30. Rapid Network Advances and the Digital Divide • The current generation of 2.5-10 Gbps network backbones arrived in the last 15 Months in the US, Europe and Japan • Major transoceanic links also are reaching 2.5 - 10 Gbps • Capability Increased ~4 Times, i.e. 2-3 Times Moore’s Law • This is a direct result of the continued precipitous fall of network prices for 2.5 or 10 Gbps in these regions • Higher prices remain in the poorer regions • There are strong prospects for further advances that will cause the Divide to become a Chasm, Unless We ActFor the Rich Regions: • 10GigE in campus+metro backbones; GigE/10GigE to desktops • Advances in protocols (TCP) to use networks at 1-10 Gbps+ • DWDM: More 10G wavelengths and/or 40G speeds on a fiber • Owned or leased wavelengths: in the last mile, the region, and/or across the country

  31. PingER Monitoring Sites PingER (Also IEPM-BW) • Measurements from • 38 monitors in 12 countries • 790 remote hosts in 70 Countries; 3500 monitor-remote site pairs • Measurements go back to Jan-95 • Reports on link reliability, quality • Countries monitored • Contain 78% of world population • 99% of Internet users Remote Sites Need to Continue, Strengthen the IEPM+ICTP Monitoring Efforts

  32. History – Loss Quality (Cottrell) • Fewer sites have very poor to dreadful performance • More have good performance(< 1% Loss) • BUT <20% of the world’spopulation has Good orAcceptable performance

  33. History - Throughput Quality Improvements from US 80% annual improvement Factor ~100/8 yr Bandwidth of TCP < MSS/(RTT*Sqrt(Loss)) (1) Progress: but Digital Divide is Maintained (1) Macroscopic Behavior of the TCP Congestion Avoidance Algorithm, Matthis, Semke, Mahdavi, Ott, Computer Communication Review 27(3), July 1997

  34. NREN Core Network Size (Mbps-km):http://www.terena.nl/compendium/2002 100M Logarithmic Scale Leading Nl 10M Fi Cz Advanced Hu Es 1M Ch In Transition It Pl Gr 100k Ir Lagging 10k Ro 1k Ukr 100

  35. Work on the Digital Divide:Several Perspectives • Identify & Help Solve Technical Problems: Nat’l, Regional, Last 10/1/0.1 km; Peering. • SCIC Questionnaire to Experiment Managements Lab Directors • Strong Support for Monitoring Projects, such as IEPM • Inter-Regional Proposals (Example: Brazil) • US NSF Proposal (10/2002); EU @LIS Proposal • Work on Policies and/or Pricing: pk, in, br, cn, SE Europe, … • Find Ways to work with vendors, NRENs, and/or Gov’ts • Use Model Cases: Installation of new advanced fiber infrastructures; Convince Neighboring Countries • Slovakia; Czech Republic; Poland (to 5k km fiber) • Exploit One-off Solutions: E.g. the Virtual SILK Highway Project (DESY/FSU satellite links); Extend to a SE European site • Work with Other Cognizant Organizations: Terena, Internet2, AMPATH, IEEAF, UN, GGFm etc.; help with technical and/or political solns

  36. Digital Divide Sub-Committee: Questionnaire Response Extract:

  37. Dai Davies SERENATE Workshop Feb. 5, 2003

  38. Telecom monopolies have even higher prices in low income countries • Fewer Market Entrants. Less Competition • Lower Income  Less Penetration of New Technologies • Price cap regulation creates cross subsidies between costumer groups. • Large customers (inelastic demand) subsidize small costumers (elastic): High bandwidth services are very expensive • Inefficient Rights of Way (ROW) regulation • Inefficient spectrum allocation policies C. Casasus, CUDI (Mexico); W. St. Arnaud, CANARIE (Canada)

  39. GEANT 155Mbps 34Mbps 34Mbps 34Mbps 34Mbps 34Mbps Romania: 155 Mbps to GEANT and Bucharest;Inter-City Links of 2-6 Mbps; to 34 Mbps in 2003 Annual Cost Was > 1 MEuro Stranded Intercity Fiber Assets

  40. APAN Links in Asia January 2003 Typical Intra-Asia Int’l Links 0.5 – 45 Mbps Progress: Japan-Korea Link: 8 Mbps to 1 Gbps in Jan. 2003;IEEAF 10G + 0.6G Links by ~June 2003

  41. Inhomogeneous Bandwidth Distributionin Latin America. CAESAR Report (6/02) J. Ibarra, AMPATH Wkshp Need to Pay Attentionto End-point connections (e.g. UERJ Rio) In Progress:622 Mbps Miami–Rio; CLARA Project: Brazil, Mexico, Chile, Agentina Int’l Links4,236 Gbps Fiber Capacity Into Latin America; Only 0.071 Gbps Used

  42. Digital Divide Committee

  43. Gigabit Ethernet Backbone; 100 Mbps Link to GEANT

  44. STM 16 STM 4 STM 16 Virtual Silk Highway Project Managed by DESY and Partners Virtual SILK Highway Project (from 11/01): NATO ($ 2.5 M) and Partners ($ 1.1M) • Satellite Links to 8 FSU Republics in So. Caucasus and Central Asia • In 2001-2 (pre-SILK) BW 64-512 kbps • Proposed VSAT to get 10-50 X BW for same cost • See www.silkproject.org [*] Partners: DESY, GEANT, CISCO UNDP, US State Dept., Worldbank, UC London, Univ. Groenigen • SCIC: Extend to a SE Europe Site ? NATO Science for Peace Program

  45. “Cultivate and promote practical solutions to delivering scalable,universally available and equitable access to suitable bandwidth and necessary network resources in support of research and education collaborations.” http://www.ieeaf.org Groningen Carrier Hotel TransAtlantic, Transpacific, Intra-US and European Initiatives

  46. US-JP-KR-CN-SG Tokyo by ~6/03 NY-AMS Done 9/02 (Research)

  47. GlobalMedicalResearchExchangeInitiative Bio-Medicine and Health Sciences St. Petersburg Kazakhstan Uzbekistan NL MD CA Barcelona Greece CN Chenai Navi Mumbai SG GHANA Buenos Aires/San Paolo Layer 1 – Spoke & Hub Sites PERTH Layer 2 – Spoke & Hub Sites Layer 3 – Spoke & Hub Sites 2002-3: Beginning a Plan for a Global Research and Education Exchange for High Energy Physics Global Quilt Initiative – GMRE Initiative - 001

  48. Jensen, ICTP Typ. 0-7 bpsPer Person

  49. Progress in Africa ? Limited by many external systemic factors:Electricity; Import Duties; Education; Trade restrictions Jensen, ICTP

More Related