Abilene and Internet2 Engineering Update

1. Abilene and Internet2 Engineering Update Guy Almes <almes@internet2.edu> Terena Networking Conference 2002 Limerick, Ireland

2. Outline • Abilene Update • Engineering Update • Multicast • IPv6 • QoS • End-to-End Measurements • Transport for Bulk Data Flows

3. Internet2 Engineering Objectives • Provide our universities with superlative networking: • Performance • Functionality • Understanding • Make superlative networking strategic for university research and education

4. Abilene Update • Current 2.5 Gb/s Abilene Network • Plans for 10 Gb/s Upgrade

5. Abilene is a Partnership • To build/operate Abilene, Internet2 partners with: • Cisco Systems (routers, switches, and access) • Juniper Networks (routers) • Nortel Networks (SONET kit) • Qwest Communications (circuits and collocation) • Indiana University (network operations center) • Internet2 Test & Evaluation Centers (ITECs) • North Carolina • Ohio

6. Current Abilene Status • IP-over-SONET backbone (2.5 Gb/s) • 53 direct connections • 4 2.5 Gb/s connections • 1 Gigabit Ethernet trial • 23 will connect via at least 622 Mb/s by 1Q02 • Number of ATM-based connections decreasing • 215 participants: universities and labs • All 50 states, District of Columbia, and Puerto Rico • 15 regional gigaPoPs support ~70% of participants • Expanded access • 50 sponsored participants • 23 state education networks (SEGPs)

7. Abilene international connectivity • Transoceanic R&E bandwidths growing !? • GÉANT: 5 Gb/s between Europe and New York City • Key international exchange points: • StarTap and StarLight: Chicago (GigE) • AmPath: Miami (155 Mb/s ATM) • Pacific Wave: Seattle (GigE) • MAN LAN: New York City (GigE/10GigE planned) • CA*net3/4: Seattle, Chicago, and New York • CUDI: CENIC and Univ Texas El Paso • International transit service • Collaboration with CA*net3 and StarTap

8. 09 March 2002 Sacramento Washington Los Angeles Abilene International Peering STAR TAP/Star Light APAN/TransPAC, Ca*net3, CERN, CERnet, FASTnet, GEMnet, IUCC, KOREN/KREONET2, NORDUnet, RNP2, SURFnet, SingAREN, TAnet2 Pacific Wave AARNET, APAN/TransPAC, CA*net3, TANET2 NYCM BELNET, CA*net3, GEANT*, HEANET, JANET, NORDUnet SNVA GEMNET, SINET, SingAREN, WIDE LOSA UNINET OC3-OC12 San Diego (CALREN2) CUDI AMPATH REUNA, RNP2 RETINA, ANSP, (CRNet) El Paso (UACJ-UT El Paso) CUDI * ARNES, CARNET, CESnet, DFN, GRNET, RENATER, RESTENA, SWITCH, HUNGARNET, GARR-B, POL-34, RCST, RedIRIS

9. Packetized Raw HDTV • Raw HDTV/IP: single 1.5 Gb/s UDP flow • DARPA-funded project of USC/ISIe, Tektronix, and Univ Washington • 6 Jan 2002: Seattle to Washington DC via Abilene • 18 hours: no packets lost, 15 resequencing episodes • End-to-end network performance (includes P/NW & MAX) • Loss: <0.8 ppb (90% c.l.) • Reordering: 5 ppb • Transcontinental 1-Gb/s TCP requires loss of • <30 ppb (1.5 KB frames) • <1 ppm (9 KB jumbo)

10. End-to-End Performance:‘High bandwidth is not enough’ • Bulk TCP flows • Current median flow over Abilene: 1.9 Mb/s • 95th percentile: 7.0 Mb/s

11. Future of Abilene • Internet2/Qwest agreement amended, and extended to Oct-06 • Upgrade now underway to shift • from OC-48c (using Nortel OC-192 Sonet) • to 10-Gb/s lambda (unprotected) • x4 increase in core backbone bandwidth

13. Next generation Abilene router selection • Extensive router specification and testing • Tests focused on next gen advanced services • High performance TCP/IP throughput • High performance multicast • IPv6 functionality and throughput • Classification for QoS and measurement • 3 router platforms tested and commercial ISPs referenced • Juniper T640 platform selected

14. Deployment timing • Ongoing: Backbone router procurement, detailed deployment planning • July: Rack assembly (Indiana Univ.) • Aug/Sep: New rack deployment • Fall: First Wave lambdas commissioned • Fall meeting demonstration events • Internet2 Fall Member Meeting (Los Angeles): late Oct. • SC2002 (Baltimore): mid Nov. • 2003: Remaining lambdas commissioned

15. Key next-gen Abilene emphases: I • Native IPv6 • Motivations • Resolving IPv4 address exhaustion issues • Preserving original End-to-End Architecture model • International collaboration • Router and host OS capabilities • Run native IPv6, concurrent with IPv4 • Replicate Abilene’s 1999 multicast deployment strategy • Close collaboration with Internet2 IPv6 Working Group

16. Key next-gen Abilene emphases: II • Network resiliency • Abilene lambdas will not be ring protected • Increasing use of videoconferencing/VoIP impose tighter restoration requirements (<100 ms) • Options: • MPLS/TE fast reroute (initially) • IP-based IGP fast convergence (preferable)

17. Key next-gen Abilene emphases: III • Deeper measurement capabilities • Significant factor in NGA rack design • 4 dedicated servers at each nodes • Additional provisions for future servers • Local data collection to capture data at times of network instability • Enhance active probing • Now: Latency and jitter, loss, reachability (Surveyor) • Regular TCP/UDP throughput tests: ~1 Gbps • Separate server for E2E performance beacon • Enhance passive measurement • Now: SNMP (NOC) and traffic matrix/type (Netflow) • Routing (BGP and IGP) • Optical splitter taps on backbone links at select location(s)

18. Abilene Observatories • Currently a sketch of a program for better support of computer science research • 1) Improved and accessible data archive • Need coherent database design • Unify & correlate 4 separate data types • SNMP, active measurement data, routing, Netflow • 2) Provision for direct network measurement and experimentation • Resources reserved for two additional servers • Power (DC), rack space (2RU), router uplink ports (GigE) • archive/measurement/experiment

19. Abilene Upgrade Summary • Backbone upgrade project underway • Partnership with Qwest extended through 2006 • Juniper T640 routers selected for backbone • 10-Gb/s backbone lambda deployment starts this fall • Advanced service foci • Native, high-performance IPv6 • Enhanced, differentiated measurement • Network resiliency • Incremental, non-disruptive transition

20. Multicast • By 1998, • Routing protocols existed • Deployment of native IP multicast quite rare • Early MBone no longer scalable • Considered key to new conferencing and streaming applications • Current native multicast support • PIM-Sparse, MBGP, and MSDP • Emphases on • Deployment and support for operations • Applications • Working to make it scalable

22. Current Multicast Emphases • Pressing ahead on Deployment • What are the current inhibitors to progress? • Applications / Content • Make it useful for your campuses • Explore the role of multicast in the future Internet • Improve Scalability • Press deployment of SSM • Explore the role of SSM

23. Could SSM be Enough? • 'Classic' Multicast • Group <g> has global significance • A user creates, joins, sends to g • Others can join, then send to and/or listen to g • MBGP, PIM-SM, MSDP triad • Source Specific Multicast • Group <g> has local significance • A user 's' creates, sends to <s,g> • Others can subscribe to, then listen to <s,g> • No need for MSDP (or allocation of <g> values)

24. Implications of SSM • Simplify Multicast Routing / Addressing • No need for global class-D address allocation • No need for source discovery • Complicates 'few-to-few' applications • Define all the members of the application-level group • Both a burden and an opportunity • Allows better Security, Scalability • Requires new version of IGMP

25. IPv6 • Clarify motivation for IPv6 • End-to-end transparency and global addressability • Supports application innovation, e.g., peer-to-peer • Support deployment and engineering expertise on networks, especially on campus • Anticipate need for first-class support • E.g., 10 Gb/s Abilene upgrade • E.g., Linux, Windows XP

27. Current IPv6 Emphases • IPv6 Training Workshops • About 8-10 workshops this year • First: in Los Angeles, hosted by CENIC, in February • Get some IPv6 on each campus/gigaPoP • Prepare for native peering • Abilene to gigaPoP • gigaPoP to campus • continue within campuses to key departmental LANs • Explore applications, DNS, operational stability

28. QoS • Premium Service Retrospective • Inter-AS Premium Service proved too ambitious • Too great a demand on all routers to be able to police and to shape • Too great a demand on ability of peering networks to coordinate • Implicit claims • Over-provisioning plus removal of non-congestive loss • Adaptive applications

29. Non-Elevated Services • Scavenger • Less than best effort • Easy to deploy • Applications: • Massive file transfers • Marking non-performance-sensitive applications • Alternative Best Efforts • Active area of research • Avoid gaming by users, while avoiding need for policing

30. Localized Elevated Services • Nature of Congestion as a threat • Less on national/international backbones • More at hard-to-upgrade local/metro networks • Decentralized Experimentation • Coordinates with some Scavenger deployment • Active area of study by the working group

31. The Current Situation • Our universities have access to an infrastructure of considerable capacity • examples of multi-hour 1.6 Gb/s flows with no loss and very little reordering • End-to-end performance varies widely • but 40 Mb/s flows not always predictable • users don't know what their expectations should be • A well-known mismatch

32. What are our Aspirations? • Candidate Answer #1:Switched 100BaseT + Well-provisioned Internet2 networking at 80 Mb/s • But user expectations and experiences vary widely

33. What are our Aspirations? • Candidate Answer #2:Lower user expectations and minimize complaining phone calls • There is a certain appeal I suppose...

34. What are our Aspirations? • Candidate Answer #3:Raise expectations, encourage aggressive use, deliver on performance/functionality to key constituencies. • Not the easy way, but necessary for success

35. Threats toEnd to End Performance • Fiber problems • dirty fiber • dim lighting • 'not quite right' connectors

36. Threats toEnd to End Performance • Fiber problems • Switches • horsepower • full vs half-duplex • head-of-line blocking

37. Threats toEnd to End Performance • Fiber problems • Switches • Inadvertently stingy provisioning • mostly communication • happens also in international settings

38. Threats toEnd to End Performance • Fiber problems • Switches • Inadvertently stingy provisioning • Wrong Routing • asymmetric • best use of Internet2 • distance

39. Threats toEnd to End Performance • Fiber problems • Switches • Inadvertently stingy provisioning • Wrong Routing • Host issues • NIC • OS / TCP stack • CPU

40. Perverse Result • 'Users' think the network is congested or that the Internet2 infrastructure cannot help them • 'Planners' think the network is underutilized, no further investment needed, or that users don't need high performance networks

41. Transport Protocol Issues • Improved TCP Implementation • Web100 Project • SACK, Window Scaling • ECN • But, still subject to fundamental limits • Mathis et al. Theoretical result • TCP-throughput = (C x MTU) / (RTT x sqrt(loss)) • Prospects for raising MTU, reducing RTT, loss

42. Need to supplement TCP • Bulk Data Transfers • Absolute need to include congestion control • But include flow control in a more aggressive way • Related Ideas • Transport-level gateways? • Known-Capacity pipes vs Groping for available Capacity

43. www.internet2.edu