Internet2: Supporting Next Generation Networking

1. Internet2: Supporting Next Generation Networking NASA IT Summit 2010 Christian Todorov August 16, 2010

2. Session Agenda • Internet2 Overview • Network Architecture • Network Services • IP • Circuit Service • Use of Dynamic Circuits • Waves and Private Networking • Supporting Advanced Research • Discussion

3. INTERNET2 OVERVIEW

4. The Internet2 Network Overview • Layer 1: Managed wavelengths from Level(3) Communications • Level(3) owns and manages Infinera optical gear: responsible for software upgrades, equipment maintenance, remote hands, sparing, NOC services • Internet2NOC has total provisioning control • Layer 2: Internet2 owned and managed equipment • Using OSCARS and DRAGON GMPLS control plane • Layer 3: Internet2 owned and managed Juniper MX960s • Expanded Observatory • Platform for layer 1/3 network performance data collection, collocation, experimentation • perfSONAR integration for intra- & inter-network performance analysis • International connectivity • Layer 1 network extended to international exchange points in Seattle, Chicago and New York City • Peering points in Seattle, PAIX, Equinix Chicago

5. Internet2 Connected Community 217 Leading US research institutions ~50,000 K20, libraries, museums, zoos and aquariums 38 states covered by SEGP program Federal network peerings and International peerings reaching 80 countries Private networking provided to NOAA and Department of Energy (ESnet); Piloting with USFS Named participant in the FCC’s RHCPP; connecting rural healthcare networks Actively working with the VA JPL and Goddard are Affiliate Members Internet2 peers with NASA networks: NREN and NISN as well as others such as DREN and USGS

6. Internet2 Universities217 University Members as of June 2010

7. Sample Listing of Government Affiliates

8. Internet2 Affiliate Members • ACUTA: The Assn. for Information Communications Professionals in Higher Ed. • Altarum • Ames Laboratory, US Department of Energy • Argonne National Laboratory • Association of Universities for Research in Astronomy • Berklee College of Music • Brookhaven National Laboratory • CineGrid, Inc. • Coalition for NeCleveland Institute of Music • tworked Information (CNI) • Department of Veterans Affairs (VA) • Desert Research Institute • EDUCAUSE • ESnet • Federal Highway Administration, Turner-Fairbank Highway Research Center • Fermi National Accelerator Laboratory • HIMSS (Healthcare Information and Management Systems Society) • Howard Hughes Medical Institute • IEEAF (Internet Educational Equal Access Foundation) • Indian Health Services • Indiana Higher Education Telecommunication System (IHETS) • ISOC (Internet Society) • Jet Propulsion Lab • JSTOR • Laboratory for Telecommunications Sciences (LTS) • Lawrence Berkeley National Laboratory • Library of Congress • Los Alamos National Laboratory • Manhattan School of Music • NASA Goddard Space Flight Center • National Archives and Records Administration • National Emergency Number Association (NENA)

9. Internet2 Affiliate Members • National Geographic • National Institutes of Health • National Science Foundation • New World Symphony • NIST (National Institute of Standards and Technology) • NITLE (National Institute for Technology and Liberal Education) • NOAA (National Oceanic & Atmospheric Administration, Washington, D.C.) • Oak Ridge National Laboratory • OCLC Online Computer Library Center • Pacific Northwest National Laboratory • Princeton Plasma Physics Laboratory • Ruth Lilly Health Education Center • Stanford Linear Accelerator Center • SURA (Southeastern Universities Research Association) • The Children's Hospital of Philadelphia • The Philadelphia Orchestra Association • The World Bank • Thomas Jefferson National Accelerator • TOPIX (Torino E Piemonte Exchange Point) • UCAR/NCAR (University Corporation for Atmospheric Research) • United Nations System of Organizations (UN) • United States Antarctic Program (USAP) • United States Department of Agriculture, Forest Service (USFS) • United States Department of Commerce Boulder Labs • United States Department of State • United States Holocaust Memorial Museum • University of North Carolina General Administration

10. The Internet2 Network Network Design • Built on dedicated fiber from Level(3) Communications – 13,000+ mile footprint • Currently based on 10G waves; moving to 100G • Built on Infinera innovative optical technology • Simple and convenient add/drop technology • Simple and convenient wave setup • Demonstrated high reliability in initial period of operation on the Level(3) network • Economics of Infinera system are disruptive in the market place • Architecture has maximum flexibility • Every direct connector can access every wave on the system if needed • Adding add/drop points doesn’t require network redesign

12. Internet2 NetworkInternational Connectivity PacWave: APAN/TransPac2, AARnet, KOREN/KREONet2, CA*Net4, GEMNET, REANNZ, T640 T640 T640 T640 T640 T640 T640 T640 T640 MANLAN:TANet2/TWAREN, TENET, MCIT/ENERGI, LHCnet, QatarFN, CA*Net4, SURFNet, CERN, GEANT(2), SINET, NetherLight/IEEAF France Telecom (v6), HARNET, CERN StarLight: CERN, JGN2, ASNet, CERN, KOREN/KREONet2, CA*Net4, CERN, JGN2, SINET, GLORIAD, TANet2/TWAREN NGIX-Ames:AARnet PacWave:UNINET, SINET, QatarFN, APAN/TransPac2 TANet2/TWAREN, SingAREN NGIX-East/A-Wave: CLARA, GEANT, RNP2 AMPATH/SFGP:ANSP, REACCIUN-2, redCLARA redCLARA, CUDI via CALREN/PacWave CUDI via UTEP / UT

13. Last updated: April 2005 Internet2 NetworkInternational Connectivity Europe-Middle East Asia-Pacific Americas Austria (ACOnet) Belgium (BELNET) Croatia (CARNet) Czech Rep. (CESNET) Cyprus (CYNET) Denmark (Forskningsnettet) Estonia (EENet) Finland (Funet) France (Renater) Germany (G-WIN) Greece (GRNET) Hungary (HUNGARNET) Iceland (RHnet) Ireland (HEAnet) Israel (IUCC) Italy (GARR) Latvia (LATNET) Lithuania (LITNET) Australia (AARNET) China (CERNET, CSTNET, NSFCNET) Hong Kong (HARNET) Japan (SINET, WIDE, JGN2) Korea (KOREN, KREONET2) Singapore (SingAREN) Philippines (PREGINET) Taiwan (TANet2, ASNet) Thailand (UNINET, ThaiSARN) M Luxembourg (RESTENA) alta (Univ. Malta) Netherlands (SURFnet) Norway (UNINETT) Poland (POL34) Portugal (RCTS2) Qatar (Qatar FN) Romania (RoEduNet)Russia (RBnet) Slovakia (SANET) Slovenia (ARNES) Spain (RedIRIS) Sweden (SUNET) Switzerland (SWITCH) United Kingdom (JANET) Turkey (ULAKBYM) *CERN Argentina (RETINA) Brazil (RNP2/ANSP) Canada (CA*net) Chile (REUNA) Mexico (Red-CUDI) United States (Abilene)Peru (RAAP) Venezuela (REACCIUN-2) Africa Central Asia Algeria (CERIST) Egypt (EUN/ENSTIN) Morocco (CNRST) Tunisia (RFR) South Africa (TENET) Armenia (ARENA) Georgia (GRENA) Kazakhstan (KAZRENA) Tajikistan (TARENA) Uzbekistan (UZSCI)

14. Internet2 NetworkFederal Peering T640 T640 T640 T640 T640 T640 T640 T640 T640 PacWave:DREN, NREN, ESnet New York: ESnet StarLight:TeraGrid, NREN, DREN, NISN, USGS, ESNet NGIX-Ames: NREN, DREN, NISN, USGS DC: DRAGON NGIX:DREN, USGS, NISN, NREN, Wright Patt AFB, ESNet

15. Network Architecture

16. Objectives • The vision for the Internet2 Network is a seamless integrated network facility that allows for applications and users to transparently utilize the services and network layers that most appropriately serve their needs, when they need it, in a cost effective manner. • This network facility will allow users to focus on their work and not on the network.

17. Scalability and Operability The Internet2 Network is based on a unique arrangement with Level 3 that represents a hybrid approach to carrier provided resources. Internet2 has full control over the provisioning on the network but does not bear the responsibility of supporting and maintaining the physical infrastructure: fiber, amps, transport equipment, etc. Level 3’s support of the physical network frees Internet2 of having higher levels of specialized engineering resources dedicated to network support. The Internet2 NOC has a full view into the underlying transport equipment and works jointly with a dedicated NOC group within Level 3. The Internet2 network is constructed on a dedicated fiber pair and with dedicated transport equipment The Infinera, and Juniper equipment used in the network are 40G capable and each has 100G on their roadmaps

18. Collocation Facilities • 30 facilities across the US • 27x Level 3 • 1x Westin Building, Seattle • 1x 32 AoA, New York • 1x 710 N. Lakeshore Drive, Chicago • Dedicated suites or colo areas – secured • Supports Internet2, ESnet, NOAA and some Internet2 connector equipment

19. Optical Network • Infinera optical transmission equipment • Managed by Level 3 Communications • Internet2 has direct provisioning control • Internet2 has direct TL-1 monitoring • Infinera nodes • Redundant Add / Drop (28) • Regeneration (28) • Optical Amplification (193)

20. Internet2 NetworkIP Network Services • Carrier-class IP service • Natively supports IPv4 and IPv6, as well as multicast and jumbo frames • Carried over 10Gbps waves on the extremely reliable Infinera backbone • 40Gbps and 100Gbps are future enhancements • Backhaul to the nearest router is included in the IP connection • Redundancy options • One option is a geographically diverse backup connection. This connection is viewed as a backup and the aggregate traffic flow over the primary and redundant circuit must not exceed that of the primary circuit • Internet2 currently supports 2 x 5 Gig and 2 x 10 Gig connections as well. These can be in geographically diverse locations. • Protection against fiber cuts using the automatic restoration capabilities of the Juniper MX’s MPLS Fast Reroute as an optional service • Additional IP circuits are also available

21. Internet2 NetworkWaveCo Static Circuit Services • Are currently set up manually by the Internet2 NOC. • Can have any bandwidth up to a full 10Gbps wave or multiple 10G waves • Can be provisioned directly over the Infineras (waves) or through the Junipers (sub-rate circuits) • This provisioning can occur over the same physical interface that is used for the Dynamic Circuit Network • Circuits can be offered both as protected and unprotected • Can be ordered for a period of weeks up to years • Have a fee for service that is based on distance, bandwidth, duration and protection scheme.

22. Internet2 NetworkDynamic Circuit Network The dynamic circuit network (ION) automatically provisions circuits among participants deploying the dynamic networking control plane and protocol Each participant may connect to one or more circuits allowing them to customize the infrastructure to meet their application needs The ION service is currently based on connections into the Juniper MX960s. As the ION service grows and other R&E networks across the world “peer” with the ION Network, through exchange points, world wide reachability will grow.

23. ION Service • ION has moved from a SONET-based network on the Ciena’s to an MPLS-based service operating on the current IP network. • This transition took place during the summer • ION connectivity and provisioning options for Internet2 Network Participants will increase. • A Connector need only maintain a single connection that can provide both IP and ION services on the same port. The Connector is empowered to appropriate resources in a way that works with their regional model. • The shared MPLS model can increase the efficiency of the backbone. • Bandwidth reserved for a circuit instantiation is available for use by other network users when the circuit owner is not utilizing the circuit for transfer • There’s opportunity to provide circuits that can burst above their requested commit rate, if there is sufficient headroom available. • ION is a production service managed by the IU NOC. • ION circuits are provisioned using a simple and secure web based interface. • ION is available to any institution regardless of their membership status. • The goal is to make the service available to the entire connector community. • Connectors or sites that use ION should contact us to discuss the best way to utilize this new model. • Contact Dale Finkelson (dmf@internet2.edu) for further information.

24. I2 DCN Software Suite • OSCARS (IDC) • Originally an ESnet Project, now joint development by Internet2 and ESnet • Open source project maintained by Internet2 and ESNet • DRAGON (DC) • NSF-funded • Open source project maintained by USC ISI EAST and MAX

25. InterDomain Controller (IDC) Protocol (IDCP) Developed via collaboration with multiple organizations Internet2, ESnet, GEANT2, Nortel, University of Amsterdam, others The following organizations have implemented/deployed systems which are compatible with this IDCP Internet2 Dynamic Circuit Network (ION) ESNet Science Data Network (SDN) GÉANT2 AutoBahn System Ciena (Nortel) (via a wrapper on top of their commercial DRAC System) Surfnet (via use of above Nortel solution) LHCNet (use of I2 ION Software Suite) Nysernet (use of I2 ION Software Suite) University of Amsterdam (use of I2 ION Software Suite) DRAGON Network The following "higher level service applications" have adapted their existing systems to communicate via the user request side of the IDCP: LambdaStation (FermiLab) TeraPaths (Brookhaven) Phoebus

26. ION Control Plane

27. Multi-Service/Domain/Layer/Vendor Provisioning • Multi-Domain Provisioning • Interdomain ENNI (Web Service and OIF/GMPLS) • Multi-domain, multi-stage path computation process • AAA • Scheduling GEANT TDM GUI Internet2 Network Regional Network Regional Network XML API Dynamic Ethernet Dynamic Ethernet TDM Domain Controller ESNet Data Plane Ctrl Element Control Plane Adjacency Ethernet LSP IP Network (MPLS, L2VPN) SONET Switch Router Slide from Tom Lehman, ISI-East

28. Application Interfaces to ION • Phoebus is network middleware to facilitate high throughput for the non-wizard • Based on a “Session” Protocol (Layer 5) that talks to Phoebus Gateways • These Gateways do Transport protocol translation and tuning • Transparently allocates circuits over the Internet2 ION • TeraPaths is a system from BNL • Creates end-to-end virtual paths and guaranteed bandwidth for specific flows with DiffServ marking • Web interface for manual reservations and a Web Services interface for automatic reservation • Allocates resources on the ESNet SDN or Internet2 ION

29. Phoebus Gateways are placed at network access points and can be thought of as network “on-ramps • At a high level, the goal is to get network data to these gateways which can then intelligently move the data as efficiently as possible to the other edge of the network • Phoebus will automatically utilize ION resources as needed and as available • At the other edge, it again appears as legacy traffic does today, thus neither client or server need to be modified

30. Waves and Private Networking Internet2 makes dedicated waves available over its network to support researchers and organizations Internet2 has provided nation-wide dedicated network infrastructure to support the needs of advanced research networks like ESnet (DoE) and N-Wave (NOAA) allowing for private research networking that has access to the broader Internet2 community

31. Monitoring and Measurement

32. Internet2 NetworkObservatory • Dependence on the network is increasing • Distributed applications & moving larger data sets • Network is growing much more complex • Need to better understand the network • Transparent network operation • User must know what performance levels to expect • Network operators must be able to demonstrate that the network meets or exceeds those expectations. • Application developers must have access to tools that differentiate between network problems and application problems.

33. Observatory Services • Several tools and services are available for network troubleshooting: • Active testing to the middle to divide and conquer • OWAMP (latency), BWCTL (throughput), NDT (last-mile/end station) • perfSONAR (cross-domain tool results, link utilization, and test point locations) • BGP feeds to RouteViews • Link utilization at 10 second intervals • “visible backbone” gives router configuration snapshots • “Router” proxy for interactive query • http://www.internet2.edu/observatory/ (data views) • http://noc.net.internet2.edu/

35. perfSONAR A way to export measurement data from your network, and consume it from many networks (including your own) “Interoperable network measurement middleware” A set of schemas / “APIs” and protocols

36. perfSONAR Integrates Network measurement tools and archives (e.g. stored measurement results) Data manipulation Information Services Discovery Topology Authentication and authorization

37. Many collaborations are inherently multi-domain, so for an end-to-end monitoring tool to work everyone must participate in the monitoring infrastructure m1 m1 m1 m1 m1 m4 m4 m4 m4 m4 m3 m3 m3 m3 m3 user performance GUI Analysis tool measurement archive measurement archive measurement archive measurement archive measurement archive GEANT (AS20965) [Europe] DESY (AS1754) [Germany] FNAL (AS3152) [US] DFN (AS680) [Germany] 37 ESnet (AS293) [US]

38. Meeting Upcoming Application Demands

40. Demands on the Network • Entering the age of large scientific facilities • Large Hadron Collider at CERN • Very Long Baseline Arrays (radio astronomy) • Large Synoptic Survey Telescope (2010-13) – 30TB/night • An increasingly diverse set of demanding applications are utilizing network resources • Telemedicine: BIRN project, proteomics, tele-surgery, remote ICU, radiology: high-resolution 3D color fMRI brain scan = 4.5PB • Telepresence: master classes, virtual classrooms, tele-psychiatry • High performance video delivery: Uncompressed HD, Cinegrid • Disaster Recovery and distributed storage

41. Tier 1 to Tier 2 Traffic

42. The New Requirements High performance applications are dependent on high performance networks Networks must be fast, reliable, scalable, have flexible architectures, be cost effective, capable of delivering multiple services across multiple network layers, easy to operate and maintain, and have a view towards the future Enable the user – the network as a service

43. Questions?