Resource Optimization in Hybrid Core Networks with 100G Links

1. Resource Optimization in Hybrid Core Networks with 100G Links Malathi Veeraraghavan University of Virginia Date: Feb. 4, 2010 (Collaborator: Admela Jukan) • Outline • What is a hybrid network? • Scheduled Dynamic Circuit Service (SDCS) • Single or separate hybrid networks? • SDCS usage • ASCR project software testing on DOE ANI network testbed • Summary Sponsored by DOE ASCR grant DE-SC0002350

2. What is a hybrid network? • A network with • packet switches and circuit switches? • Connection-oriented (CO) and connectionless (CL) modes of operation? • All-optical switches and electronic switches? • “Optical and IP”? [dissimilar items]

3. A classification of switches/gateways • A circuit switch is necessarily connection-oriented as positions have to be allocated to a communication session before data transfer can start

4. ESnet: IP-routed network(2006 network) • Is this a hybrid network? • Leased lines between routers pass through SONET circuit switches • Thus, packet and circuit switches • Leased-lines: CO mode • CO and CL modes • Some switches could be all-optical (metro rings) • Thus all-optical and electronic • Not quite! From ESnet document dated Oct. 08, 2006

5. Our understanding of“hybrid network” • Supports two types of services: • IP-routed service • Dynamic Circuit Service • To do so, what is required of the network? • Support for connection-oriented (CO) and connectionless modes • CO mode should support dynamic bandwidth sharing with control-plane software such as OSCARS schedulers, not just leased-line service • Circuit or virtual-circuit switches to support the dynamic circuit service • IP routers • As link speeds and switching capacities increase, all-optical switches may be required

6. Scheduled Dynamic Circuit Service (SDCS) • Instead of advance-reservation dynamic circuit service, we propose the name SDCS because • circuit requests can be for immediate usage • so cannot limit it to “advance-reservation” or book-ahead • scheduler (e.g., OSCARS) returns a start time for the circuit • hence “Scheduled Dynamic Circuit Service” • Two types of SDCS requests: • User-specified start times (USST): co-scheduling apps • with instrument or HPC or for a teleconference • Earliest start time (EST): immediate usage requested, but scheduler returns an assigned start time based on resource availability: suitable for large file transfers

7. How does SDCS fit in with other connectivity services? RDP = 5.9TB (T1*1 year) RDP = 1.4MB (DS0*3 min) Gap to fill RDP = 1500B RDP: Rate Duration Product

8. How does SDCS differ from leased line service? Leased line Circuit/VC switch Customer device e.g., IP router SDCS access link SDCS access link Dynamic circuit Customer device e.g., cluster computer Customer device e.g., LCD display

9. Differences • Difficult to articulate differences between a “leased line” and a “dynamic circuit” • BUT difference between leased-line service and dynamic-circuit service is clear

10. Outline check • Outline • What is a hybrid network? • Scheduled Dynamic Circuit Service (SDCS) • Single or separate hybrid networks? • SDCS usage • ASCR project software testing on DOE ANI network testbed • Summary

11. Single or separate networks? • The term “hybrid networking” is more general than “hybrid network” • Hybrid networking: Deploy two separate networks to support the two services • IP-routed service • Scheduled dynamic circuit service (SDCS) • Hybrid network: seems to imply a single network

12. ESnet4 (from 2006 talk by Johnston & Burrescia) 1625 miles / 2545 km Production IP core (10Gbps) SDN core (20-30-40-50 Gbps) MANs (20-60 Gbps) or backbone loops for site access International connections Primary DOE Labs High speed cross-connectswith Ineternet2/Abilene Possible hubs 2700 miles / 4300 km Core networks 50-60 Gbps by 2009-2010 (10Gb/s circuits),500-600 Gbps by 2011-2012 (100 Gb/s circuits) Canada (CANARIE) Europe (GEANT) CERN(30+ Gbps) Canada (CANARIE) Asia-Pacific Asia Pacific CERN(30+ Gbps) GLORIAD(Russia and China) USLHCNet Europe (GEANT) Asia-Pacific Science Data Network Core Seattle Boston Chicago IP Core Boise Australia New York Cleveland Kansas City Denver Washington DC Sunnyvale Atlanta Tulsa LA Albuquerque Australia South America (AMPATH) San Diego South America (AMPATH) IP core hubs Houston Jacksonville SDN hubs Core network fiber path is~ 14,000 miles / 24,000 km

13. “Single” hybrid network • What does a “single” network mean: • One node at each PoP • One set of inter-PoP links • One set of customer (e.g., site) access links • Today’s deployment: • One node at each PoP for ESnet (IP-routed service) and one node for SDN (dynamic circuit service) • Separate inter-PoP links • Separate site access links • Reasons: Costs of MX vs. M-series interfaces, funding, etc. • Expenditures saved: • Inter-PoP and site access link costs • Maintenance costs for nodes • What are the negatives?

14. What is required for a“single” hybrid network deployment? • Node equipment should support • IP routing capability • Circuit/VC capability (e.g. VLAN, MPLS, GMPLS) • To support IP-routed service and SDCS on a single interface • Circuit/VC based rate enforcement • e.g., VLAN rate policing/limiting • OSCARS IDC for circuit scheduling

15. Single node supports both IP-forwarding and virtual-circuit capabilities • Existing equipment already available • Juniper MX series switches • Cisco 6500 series • Carrier-grade Ethernet switches • Separate SDN was proof-of-concept for dynamic circuit service? • Since same type of node (MX) used in both IP and SDN networks, both services can be offered on one substrate IP (Layer 3) Ethernet/VLAN (L2) Leased-line for IP-routed service Access links to sites and peers Inter-PoP ESnet links Dynamic circuit

16. Part of the ESnet topology (potential single network offering both services) PE router Leased-line for IP-routed service DENV PNWG IP IP IP IP IP PNNL Ethernet/VLAN Ethernet/VLAN Ethernet/VLAN Ethernet/VLAN Ethernet/VLAN Dynamic circuit ALBU Spectrum NMS OSCARS IDC SUNN PE router LANL 100 GbE ELPA

17. Outline check • Outline • What is a hybrid network? • Scheduled Dynamic Circuit Service (SDCS) • Single or separate hybrid networks? • SDCS usage • ASCR project software testing on DOE ANI network testbed • Summary

18. SDCS deployment • Successful deployment in the core • important: inter-domain demonstrations • [Scheduled Circuit Routing Protocol: like BGP for SDCS] • Status on deployment in: • site networks? • With site deployments, development of new applications (remote instrument control, WAFS) can be encouraged • Without data source-to-data sink (end-to-end) reach of SDCS: • usage scenarios for core SDCS service

19. Core network SDCS usage scenarios • Usage I: Lambdastation • Applications on user hosts (dCache/SRM) communicate with Lamba Station Server, which runs at the “edge” of sites • Lambda Station Server communicates with core network IDC to reserve/provision dynamic circuit, and sets PBR in CE router to forward packets corresponding to that particular application flow to the circuit • Usage II: Automatic long flow detection at PE routers and rerouting to dynamic circuits • Usage III: Use SDCS instead of leased-line service for creating IP-routed topology, and resize/reroute these router-to-route circuits as aggregate IP loads on these circuits change • Spectrum monitors SNMP data and initiates changes by requesting these of the IDC

20. Two issues with Usage III • Current ESnet weathermap shows light loading (5%-20%) on links between IP routers • If SNMP loads indicate a change from 5% to 25%, should the Spectrum NMS react and increase the rate of the circuit between the two routers? • Commercial providers note: • Operations divisions typically have strong resistance to change the network topology because of the potential for: • route flaps, and • drastic changes in the end-end packet latency (e.g., > 10ms) • For these reasons, while theoretically management-plane traffic and network engineering is a potential user of core SDCS, we expect its use to be limited mainly to failure recovery.

21. Our ASCR project • Focus on Usage II • Long-flow identification methods: • 5-tuple rule based flow classification • payload based classification • machine learning algorithms

22. Software implementation and testing on DOE ANI network testbed • MFDB: Monitored Flow Data Base • Flows in MFDB mirrored by router to HYNES server • When one of these flows is detected, circuit setup is initiated and PBR set in router to route flow to circuit

23. Summary • Proposed name “Scheduled Dynamic Circuit Service (SDCS)” to cover both • advance-reservation • immediate-request • Hybrid networks support IP-routed service and SDCS • SDCS: comparison with leased-line service • Single hybrid network deployment • Three usage scenarios for SDCS • HYNES software testing on DOE ANI network testbed