Inter-Domain Services

1. Inter-Domain Services Presented to Quilt Spring Workshop Salt Lake City, UT By Linda Winkler winkler@mcs.anl.gov April 4, 2007

2. Motivation Emerging large-scale, globally distributed applications require more sophisticated network services than have previously been delivered. • Support applications that demand capabilities that are hard to support in a shared packet infrastructure • Large bandwidth applications • Applications don’t want to worry about their impact on other network users, or vice versa • Applications that benefit from circuit characteristics, and that may be low bandwidth in nature • Dedicated network resources • Deterministic performance • Repeatable and predictable • Very high performance • Multi gigabit flows, low latency, low loss, minimal jitter • User community expectations • Extreme bandwidth, dedicated and on-demand • Inter-regional and international connectivity requirements • Reserve and schedule network resources in advance • In conjunction with availability of non-network resources (e.g. instruments, clusters, viz devices) • Flexible and dynamic • Capability to acquire resources on short notice from many potential service/resource providers • E-Science services growth • Broad scope of applications • Petabyte demands • Globally distributed applications, resources and collaborators

3. Trends=>Set the Stage • Applications community desire to treat network resources as an integral Grid resource • R&E Optical Networks • Locally-managed fiber termination points • Locally-organized peering relationships • Locally-controlled layer-0/1/2/3 services • Community ownership and control of network infrastructure • Ubiquitous Ethernet • Most-requested client service interface • Both point-to-point and virtual-LAN topologies • Apps consuming bandwidth in Ethernet-sized increments • 10G has become a commodity • Application expectations for error-free connections keep rising. Apps with minimal error checking running on pipes with almost no error correction, across inter-RON pipes with no error reporting  a recipe for failure • Convergence Optical + Ethernet Layers • Ethernet client interface, Optical transport • Optical service established via GMPLS • Control plane technologies emerging to help meet user expectations in the current environment • FPGA technology • Allowing WDM transport hardware to be built with software defined functionality • Flexibility, fewer components, configurations, spares • Investment protection and future-proofing • Pluggable optics • Flexibility, lower costs • Newer capabilities

4. Technology Choices • Reasons you may want/need build circuits • Dedicated bandwidth connections for deterministic file transfers • Dedicated bandwidth & low jitter for instrument control or interactive applications • Connector backhaul to your IP Network • Traffic engineering of your IP Network • Dynamic router-to-router circuits for traffic cut through • Computer to Computer communications • Processor to memory or block data storage system access • Setting up application specific topologies to create & optimize distributed application or data storage systems • There are many technologies available over which to construct these circuits • IP router-based Multiprotocol Label Switching (MPLS) Label Switched Paths (LSPs) “circuits” • Ethernet VLAN based “circuits” • SONET/SDH TDM “circuits” • Wavelength Division Multiplexing (WDM) “circuits”

5. Mapping Technology to Services Network Technology Service IP Layer 3 IP Ethernet Layer 2 Ethernet VLAN SONET DM encap GFP Lightpath (sub-lambda) Layer 1 Lambda Wavelength Layer 0 Fiber Dark Fiber

6. iCAIR UvA 2 2 Starlight NetherLight, Amsterdam 1 1 Fujitsu NortelDX NortelDX Nortel OME 6500 Nortel 8600 Nortel 8600 2 OMNInet UvA ONS ONS 1 Nortel Research Labs Ottawa Nortel OME 6500 Glimmerglass CA*Net5 Topology for L1 Dynamic Optical Multicast Testbed OC192/STM64 10GE LAN 750-~980 Mbps • 750-~980+ Mbps Hi-Def Video • 3x1:2 Optical Multicast: StarLight OME 10 G WAN Nortel UvA Nortel HDXc iCAIR Nortel HDXc 3x750 M 3x750 M 3x1:2 Optical multicast UvA iCAIR iCAIR Nortel Nortel

7. OC-192 GFP/VCAT 10GE LAN 10G WAN 1.5 Gbps LSU Tampa Brno Starlight Netherlight Nortel HDXc Nortel HDXc CAT 6509 F10 E1200 CAT 6506 SL E1200 10GE switch Calient EnLIGHTened Wave Calient Catalyst 65xx Brno CaveWave ONS ONS McLEAN e300 SC NOC Nortel CAT 6506 Nortel MERS 8600 A-Wave Raptor LSU Topology for L2 Multicast Demo • 1.5 Gbps Hi-Def video • 3x1:2 Ethernet multicast: StarLight & SCinet • Layer1 and Layer2 networking Baton Rouge (LSU Class) Brno Tampa 2 1 1 OMNInet 2 2 OMNInet 4 3 x 1.5G 2 OPTERON 2 3x1.5G 3x1.5G 1 1 Brno 3 Tampa 1 LSU 2 2 1 LONI Tampa OPTERON AMD PC CAT 6504 Brno LONI

8. Unification of Data Plane Technology Across Layers 1/2/3—One Method Alignment of VLANS Circuit Switched IP Network MPLS tunnel VLAN over MPLS Layer-1/2 connection VLAN over Ethernet or SONET

9. Dynamic Circuit Setup/Allocation • Lots of work in the community on dynamic resource allocation (lightpaths) and how we’re going to get there from here, eg GLIF forum. • Soon we will be to able to rely on multiprovider/multivendor control plane signaling for circuit establishment. • Simplify or add complexity? • Dynamic resource mechanisms will increase our capabilities and enable new capabilites, like timesharing resources between large short-duration flows. • Cost TBD

10. Connection Brokering per Domain • Network Descriptions • Paths through network • Information about paths • Common Services Definition • Description of connectivity service • Predictable • Verifiable • Repeatable • End-to-end • Agreement between user and provider(s)

11. Control Plane Functions • Bandwidth required • Allow for reservation of future bandwidth • User authentication • Report status of infrastructure • Retain and share information on topology resources and reachability • Routing- topology discovery and path computation • Signaling- standard communications protocols between network elements for the establishment and maintenance of connections • Neighbor discovery- cross-domain sharing of connectivity details among neighbors • Domain (local) resource management- accounting of available resources

12. Interdomain Connection Setup Interdomain Control Plane TDM (SONET/SDH Layer) Control Plane Data Plane TDM or L2 substrate RON / NREN

13. Challenges • Engineering/configuration/setup • O&M complexity • Troubleshooting • Fault managment • Monitoring • Performance • Debugging tools • End-to-end measurement for monitoring and performance analysis • Policies • a way to enforce policies and filters at the domain boundaries to process the incoming inter-domain requests based on certain agreed trust and service levels/contracts between domains • Authentication, Authorization, Accounting (AAA) • Security • Scheduling • Service Level Agreements (SLAs) • User interface • Capacity planning • Many lightpaths traverse a number of transport boxes and domains • Number of domains participating on the rise (backbone providers, RONs, campuses). Are they prepared for the challenge?

14. Dynamic Resource Allocation Research • MAX DRAGON • Internet2 Circuit Services- based on extension of the DRAGON control plane architecture • Internet2 BRUW (Bandwidth Reservation for User Work) • ESNet Science Data Network and the OSCARS (On-demand Secure Circuits and Advance Reservation System) project • DANTE/GEANT JRA3 project • SURFnet and collaboration with Nortel on the DRAC project • University of Amsterdam, Network Description Language • European Union PHOSPHORUS Project • G-Lambda project (Japan) • CANARIE- UCLP • DOE UltraScience Network • DOE Hybrid Multi-Layer Network Control Project (ESnet, ORNL, I2, ISI East, Tenn Tech) • TeraPaths Brookhaven National Lab • Enlightened Computing Project (NCREN, MCNC, LSU, NCSU, RCI) • Cheetah (Univ of Virginia) • UltraLight • DICE (DANTE, Internet2, CANARIE, ESnet) • OptIPuter/iCAIR

15. Enlightened Computing Testbed San Diego L.A. Raleigh Baton Rouge Chicago To Asia To Canada To Europe SEA POR BOI CAVE wave EnLIGHTened wave (Cisco/NLR) PIT OGD DEN CHI KAN CLE SVL WDC Cisco/UltraLight wave LONI wave TUL DAL • International • Partners • LUCIFER - EC • G-Lambda - Japan • GLIF • Members: • MCNC GCNS • LSU CCT • NCSU • (Subcontract) RENCI HOU • Official Partners: • AT&T Research • SURA • NRL • Cisco Systems • Calient Networks • IBM • NSF Project Partners • OptIPuter • UltraLight • WAN-in-LAB • DRAGON • Cheetah

16. HD Media Testbed to support class on Computational Science

17. EnlightenedArchitecture

18. Internet2 Circuit Service Attributes • Physical connection • 1G or 10G Ethernet • OC-48 or OC-192 SONET • Guaranteed bandwidth (STS-1 to STS-192) • Circuit Service Type • Point-to-point Ethernet framed SONET Circuit • Point-to-point SONET Circuit • Deterministic behavior (bounded jitter, latency and packet loss) • Static or dynamic

19. NLR • FrameNet • Ethernet-based transport services over the nationwide NLR optical infrastructure • National Exchange Fabric • Dedicated FrameNet • Non-Dedicated FrameNet • Research with MCNC • Calient Optical Switch

20. References • dragon.maxgigapop.net • www.es.net/oscars • www.internet2.edu/network/library/internet2-network-service-descriptions-02272007-v1.1a.pdf • www.ist-phosphorus.eu/ • www.mfaforum.org/interop/GMPLSwhitepaper_Final1009021.pdf • www.glif.is • enlightenedcomputing.org • www.oiforum.com • IETF • ITU