Internet2 Network Design

1. Internet2 Network Design Rick Summerhill, Director Network Research, Architecture, and Technologies Internet2 Community Design Workshop 15, 16 June 2006 Indianapolis, IN

2. Contents • Strategic Objectives • Background • Requirements • Architecture • Network Usage Policy • Engineering • Network Research • Advisory Groups • Partnerships • Agreement Summary

3. Strategic Objectives Strategic Objectives • Ensure community control of the underlying network infrastructure • Control those elements that allow development of new network capabilities • Leverage the capabilities of a global telecommunications leader • Providing carrier class reliability and expanded breadth of services, along with a broad set of partnership options • Capitalize on the latest technological advancements in networking • Create an asset that benefits the entire community - researchers, universities, regional optical networks, industry, government, K-12, and the international community

4. Background Background • A defining goal of Internet2 is to support the US research universities’ need for scalable, sustainable, high-performance networking • In 1998, the Abilene network was created using 2.5 Gbps SONET circuits as part of a partnership with Qwest Communications, Cisco Systems, and Nortel Networks • The original agreement with Qwest was later extended to October of 2007 and the Abilene network was upgraded to a 10 Gbps network • A further extension for one year was possible if notification was made by March 31, 2006, an extension that was recently declined • Abilene is an IP packet based network similar in design to the original NSFnet, but with much greater capabilities and bandwidths • Today there is great Interest in Hybrid networks • Greater availability of circuit based capabilities • Greater interest world wide to bring circuit based services closer to the edge • Most RONs now have developed this capability already • The HOPI project has experimented with the basic ideas for the last two years

5. Background Process • With pending end of MoU with Qwest, Internet2 began complete examination of all the possibilities • Think out of the box - look at new ideas • Consider different network options - IP, hybrid networks, etc. • Considered a variety of different carrier and existing fiber options. For example, • How does one share wavelengths on an existing system? • Several viable options were available • In the end, one option stood out - the Dedicated Wave System (DWS) from Level3 • A Hybrid Network, not just an IP network and waves • Supports a wide variety of services and capabilities, from highly experimental to production level • Control at layer 1 • Uses equipment and fiber dedicated to Internet2 • Internet2 board has approved the DWS for the Internet2 network and the contract has been executed.

6. Requirements Requirements • Throughout the process, the requirements of the community were considered - the process did not occur in a vacuum • Requirements documents produced during the spring of 2005 were considered • The Group A report, produced by a joint engineering committee from NLR and describing a wide ranging set of network attributes • The Abilene Technical Advisory Committee (TAC) Report outlining a need to provide for new experimental services along with production services like IP • The MORPHnet document (1997) - Multi-Modal Organizational Research and Production Heterogeneous Network • Input from the Quilt Optical workshops and discussions with the RONs • Research examining a wide variety of different types of capabilities • DRAGON and CHEETAH - NSF funded projects examining dynamic provisioning • Science related projects such as UltraLight and eVLBI • OptIPuter and TeraGrid • HOPI Testbed • Input from the International community • GEANT2 providing hybrid types of capabilities • GLIF and lightpath capabilities for scientific work

7. Requirements Process Revisited • The actual process of negotiating with Level3 was done by a relatively small number of Internet2 staff • The negotiations could not have been done with a large community group • The process created a framework for the Internet2 network and enabling community involvement • The process used the knowledge gained from the above requirements documents and discussions, and was done in the spirit of developing a network to serve the community by providing a wide range of capabilities for advancing the state of the art of networking • Within the above framework, there are many details yet to be understood and ideas to be developed • That is the purpose of this meeting, starting the process and … • Leading to network deployment, future planning, and continuing community involvement

8. Architecture Architecture Design Goals • Develop an innovative optical system on a national footprint to serve the broad research and education community • The goal is to support all Internet2 members and to enhance the ability to serve a wider community • Develop a hybrid network capable of providing point-to-point services together with an IP network • Every connector (RON) connects a NewNet backbone ring across a metro location not requiring extensive backhaul • The community retains complete control of the layer 1 optical system including provisioning and switching of wavelengths • Internet2 does not have to concentrate on reliability and sparing - Level 3 is responsible for an SLA. • The community focus is on networking, not on managing devices like amplifiers • The system is capable of supporting network research in wide variety ways • Minimal Conditions of Use (CoU), allowing full participation from the entire community in providing new services and capabilities

9. Architecture The System • System uses dedicated fiber from the carrier • Significant financial advantage if owned (via IRU) footprint is used • Built on Infinera platform providing innovative optical technology • Simple and convenient add/drop technology • Simple and convenient wave setup • Demonstrated high reliability in initial period of operation on the Level3 network • Economics of Infinera system are disruptive in the market place • Architecture has maximum flexibility. Every RON can access every wave on the system if needed • System includes grooming capabilities - lightpaths can be built over Ethernet or SONET • Can take advantage of advanced SONET cpabilities like GFP, VCAT, and LCAS • Capable of lightpath provisioning to the campus

10. Architecture Lightpath Capabilities Rick Summerhill

11. Architecture System • RON connectivity expected to evolve to 2 x 10 Gbps connections • 10 Gbps IP connection • 10 Gbps point-to-point connection (capable of STS-1 granularity lightpaths provisioned in seconds), most likely provision using Ethernet (GFP based) • Hybrid capabilities • Expect 20 - 24 connectors • Simple and consistent connection scheme • Promoting aggregation • Need input and discussion on exceptional cases • The following diagrams represent preliminary thoughts on topology • Exact locations may vary, but fiber is expected to be on the Level3 footprint • One of the goals of this meeting is to clarify and optimize these locations

12. Architecture Layer 1 Topology PROVISIONAL TOPOLOGY – SUBJECT TO DISCUSSION Rick Summerhill

13. Architecture IP Network • IP network built on top of optical system • High reliability - architecture provides a variety of protection options • Commodity service offering - standard connection will include commodity services • Current plan is to continue to use highly reliable Juniper routers, but open to new technologies • Will likely use fewer routers, emphasizing point-to-point capabilities and hybrid networking • Potential near term option of 40 Gbps • Community input is important these issues

14. Architecture Layer 1 Topology with IP Network PROVISIONAL TOPOLOGY – SUBJECT TO DISCUSSION Rick Summerhill

15. Architecture Architecture Summary • Initially provisioned with ten 10-Gbps wavelengths (100 Gbps) • Capacity: up to 80 wavelengths (0.8 Tbps) • Scalability: Potential for 40 & 100 Gbps wavelength support • Reliability: Carrier provides standard service assurances for these wavelengths • Provide a variety of different types of reliability • Flexibility: Support for dynamic provisioning and wavelength switching

16. Architecture Node Architecture • Advanced optical DWDM equipment • Grooming capabilities to provide sub channels • Support for other networks having a national footprint • Simplified and standardized interface to connectors, exchange points, and other global research and education networks • 2 x 10 Gbps interfaces • Example displayed is the Denver optical node, showing connections required by the collocated IP router node and an additional network PROVISIONAL ARCHITECTURE – SUBJECT TO DISCUSSION Rick Summerhill

17. Architecture Node Architecture • Node without IP router • IP connection to remote router • Same basic architecture PROVISIONAL ARCHITECTURE – SUBJECT TO DISCUSSION Rick Summerhill

18. Architecture Node Architecture • Node without network connector • Support for Other Network • Same basic architecture PROVISIONAL ARCHITECTURE – SUBJECT TO DISCUSSION Rick Summerhill

19. Architecture Example Circuit Services • Services on NewNet • Short Term Dynamically Configured STS-1 Granularity Deterministic Circuits using two waves on the full footprint, with framing either SONET or GFP mapped Ethernet. There is no additional cost to use these capabilities. They are included in the base connectivity Costs. The may be call blocking in the use of these capabilities. • Long Term Circuits configured as above. There is an extra cost for these circuits, proportional to bandwidth, and charged on a day-by-day basis. The shortest term is one day, and there is no call blocking for these capabilities. • Long Term full waves configured and provisioned for a minimum of a year. • Additional Carrier Services • WaveCo services provisioned by Level3 as part of the agreement.

20. Architecture Example Projects on NewNet • Dynamic services • Creating lightpaths across multiple administrative domains in seconds • Development of “Domain Controllers” • Support for applications - setup of “Application SpecificTopologies” • Dynamic inclusion of optical paths into IP networks • TCP data transfers • File transfers with long segments not having to require congestion control • Network research projects deploying programmable Ethernet capable chipsets at each optical node using “NetFPGA” cards from Stanford • Service trial with GEANT2 on dynamic provisioning of 1 GigE circuits across Internet2 and GEANT2

21. Architecture Internet2/GEANT2 Service Trial • Develop inter-domain 1 GigE services between hosts or clusters of hosts in Europe and the US • Develop the technology to provide real services • Domain controllers and interoperability • Develop policies and cost models for providing such services • Interested in enlisting a couple of RONs and campuses willing to participate • The following diagram illustrates the problems

22. Internet2/GEANT2 Service Trial

23. Network Usage Policy Network Usage Policy • The use of the network facility shall be consistent with the broad set of non-profit objectives of the U.S. research and education community and its affiliates • Internet2 will seek to maximize the ability of the network to support the broad collaboration requirements of the Internet2 membership and their affiliates to interact with the broadest scope of domestic and international networks and collaborating organizations • Each organization utilizing the facility will be responsible for complying with all technical and operational standards developed for the network, and will avoid actions that adversely impact the performance or stability of the facility itself or those experienced by other users • The facility shall not be used for illegal, classified, life-safety, or unrelated commercial activities • Additionally, Internet2 will seek to maximize to its members the value of IP transit by expanding connectivity through peering with commercial networks including content providers (e.g., Google, Akamai) and other regional and national service providers (i.e., tier 2 ISPs)

24. Engineering Engineering • Building on the experience from projects such as Abilene, HOPI, and NLR, operational support is expected to fall into three broad categories: • Control plane development, deployment, and support. The Mid-Atlantic crossroads through the DRAGON project has focused on dynamic resource allocation. • Application support, enabled by targeting key applications for the research community, applications such as eVLBI. The North Carolina Research and Education Network has focused on this area for the past year through the HOPI project • Engineering, monitoring, and management has been provided by the Global Network Operations Center at Indiana University • These groups were originally selected as the HOPI testbed support center • Internet2 expects to continue these relationships for the new network

25. Network Research Network Research • The Internet2 network is an ideal platform for network research - the ability to support both highly experimental projects along with production based services is a key objective • The Abilene observatory will be expanded to include the new capabilities of the network • Data collection at all layers of the network, with datasets made available to network researchers • Support for collocation of equipment in optical nodes • Internet2 has supported individual network research projects whenever possible. Examples include • The router clamp • MPLS tunnels for researchers - these can now be supported at layer 1 • Support for the 100x100 project • Flexibility is key feature of the new network • Smaller projects at lower bandwidths can be supported on variable footprints • Projects can also be supported for variable lengths of time • Projects like VINI and GENI can be supported by the new network

26. Advisory Groups Advisory Groups • Internet2 currently has four councils: Network Planning and Policy Advisory Council (NPPAC), the Applications Strategy Council (ASC), the Industry Strategy Council (ISC), and the Network Research Liaison Council (NRLC). The latter in particular will play a major role because of the developmental nature of the network (Note that the Internet2 board is in the process of reviewing the Internet2 governance) • A network executive committee consisting of senior executives from Internet2, Level3, and others including Infinera • A technical advisory committee that reports on a regular basis to NPPAC • The advisory committee could have detailed working groups or other advisory groups (e.g., the HOPI design team) organized to provide advice on specific subjects • It could commission working groups to work on particular aspects of the network • Input from the community is crucial - a later session on this will seek further input

27. Partnerships Partnerships • The agreement with Level3 and the many capabilities of the network provide an opportunity for strong partnerships to form. For example, • The partnership with Level3 to examine new service capabilities • Joint work on control plane investigations • Examination of new optical capabilities with Infinera and others • International partnerships, for example related to service trials • Federal government partnerships, continuing and expanding on collaborations with other networks such as Esnet • Continued work with the original Internet2 partners, Qwest communications, Cisco Systems, and Nortel Networks

28. Agreement Summary Additional Agreement Information • Continuation of FiberCo • Introduction of WaveCo • Essentially, a “circuit” version of FiberCo • Provides a platform for collaboration with the corporate and academic research communities to examine new services and ideas • More information will be presented later