Dynamic Circuit Network Hands-On Workshop

1. Dynamic Circuit Network Hands-On Workshop College Station, Texas January 31- February 1, 2009

2. Welcome! • Wireless and Wired connections available

3. Welcome! • This is the 10th DCN Workshop • Nysernet • MAX • NASA Ames • University of Houston • University of Hawaii (double header) • University of Nebraska - Lincoln • University of Houston • NORDUnet Copenhagen • Texas A&M University • Introductions • http://tinyurl.com/tamudcn • http://tinyurl.com/tamudcndoc • http://tinyurl.com/tamudcnppt

4. Welcome! • Key objectives of this workshop are: • Disseminate information to the R&E community regarding the emerging class of Hybrid Network and the associated techniques for Dynamic provisioning and configuration • Review in detail and provide instruction on how to use the control plane software currently in service on the Internet2 Dynamic Circuit Network (DCN), ESnet Science Data Network (SDN), and several regional networks. • Obtain feedback directly from the community on how to improve the technologies…Hopefully, to help guide future development and deployment priorities and speed adoption • Review the state of implementation and deployment of these types of dynamic networks throughout the R&E community.

5. Instructors • Tom Lehman (USC/ISI-East) • Andy Lake (Internet2) • Remote Support • Chris Tracy (MAX) • Xi Yang (USC/ISI-East) • Jarda Flidr (MAX) • These people are involved in numerous projects related to deploying dynamic control planes: • Internet2 Dynamic Circuit Network • ESnet OSCARS Project • NSF DRAGON • Internet2 HOPI Testbed • DICE (Dante, Internet2, Canarie, Esnet) – International development activities

6. Why do a workshop? • Dynamic Hybrid Networks are new… • The service concepts are still unfamiliar to many networker experts and users… What does one gain with DCN? • The software and hardware implementations are still evolving… • Even the standards are still evolving… • The networks that support these capabilities are few but growing. • The user base is small [for now]…. But will grow as the capabilities mature and become more ubiquitous, persistent, robust, and the utility of both connection oriented services and dynamic provisioning becomes more widely recognized and accepted. • Providing hands-on experience to design and deploy these architectures is one way to broaden and promote adoption.

7. Agenda • Day 1 • 9:00 am Overview of GMPLS and DRAGON • 10:00 am Exercise #1: Designing a GMPLS Control Plane for Ethernet Data Planes • 12noon Lunch • 1:00pm Continue working on Exercise #1 • 2:00pm Overview of Web Services and OSCARS • 3:00pm Exercise #2: IntraDomain provisioning with OSCARS • 5:00pm Adjourn Day 2 • 9:00am Overview of Inter-Domain implementation in OSCARS • 10:00am Exercise #3: Inter-domain Provisioning with OSCARS • 12noon Lunch • 1:00pm Continue working with Exercise #3 • 3:00pm Use of DCN and peering dynamic networks • 4pm Adjourn Breaks: 10am, 2pm Lunch: 12noon

8. Office of Science DOE OSCARS Workshop Perspective • In this workshop we focus on implementation • We will design and build a multi-domain GMPLS controlled ethernet network • We have a mobile GMPLS test and evaluation lab consisting of 24 PCs and 12 switches • We will be focused on the GMPLS intra-domain control plane issues • Specifically, OSPF and RSVP protocols and Path Computation • We will do a very brief and cursory review of RSVP and OSPF. • For detailed information on the protocols themselves see the IETF RFCs. • We will not deal with ISIS or CR/LDP or LMP • We will focus on the “DICE” Inter-domain architecture • Web Services based topology distribution and provisioning • We use open source software developed by the NSF DRAGON Project, the DOE OSCARS Project • Intra-domain: Adapted versions of KOM-RSVP and Zebra OSPF plus the NARB for path computing • This software is the only GMPLS software available to support dynamic ethernet services • Uses OSCARS (Dept of Energy) for book-ahead scheduling and AAA • Additional software and interfaces have been developed under auspices of the DICE effort (DANTE, Internet2, Canarie, ESnet) • The code has been adapted to support a wide variety of vendor equipment (e.g. Force10, Extreme, Dell, Ciena, Cisco, Raptor) DRAGON

9. Control Plane Data Plane DCN Workshop Architecture Core Dynamic Circuit Network Green Pod ASN4 Red Pod ASN1 Yellow Pod ASN3 Blue Pod ASN2

10. Pod Network Elements Control and Data Planes Control Plane PC (VLSR#-PC, NARB, IDC) Data Plane Ethernet Switch (VLSR#-SW) End System (ES#) Network Aware Resource Broker- “NARB” Inter-Domain Controller – “IDC” NARB / IDC gre6 Virtual Label Switching Router- “VLSR” VLSR2 gre2 gre4 VLSR3 VLSR1 VLSR3-PC gre3 VLSR1-PC D2 D4 VLSR3-SW VLSR1-SW D3 D5 D1 ES1 ES2

11. Dynamic NetworksOverview and Status • Objectives and of Dynamic Hybrid Networks • Hybrid Networking and the Global R&E Community • Standardization Efforts • Internet2 Dynamic Circuit Network (DCN) • Control Plane Software • Network Architecture

12. 10/15/2008 FMM New Orleans 12

13. Joint Techs - College Station, TexasDemonstration using Dynamic Circuit

14. SC2008 - Austin, TexasDynamic Circuit Utilization Summary

15. SC2008 - Austin, TexasDynamic Circuit Utilization Summary • 119 circuits were built through the SCinet IDC between Monday the 7th@6pm CST and Thursday the 20th@6pm CST . • This includes circuits between booths and circuits going to sites not on SCinet. • 111 additional circuits were built on Internet2's DCN that did not go through SCinet. This includes Terapaths circuits, circuits between UvA and Ann Arbor, and the Paris<->Boston eVLBI circuit among others • 230 circuits (the sum of the two numbers above) in total were successfully built over the course of SC.

16. SC2008 - Austin, TexasDynamic Circuit Utilization Summary • 9 circuits were marked as FAILED during SC. • The average duration of a circuit going through SCInet was about 94 minutes. • The average was about 187 minutes if you exclude the 5 minute "Pinger" circuits built from the Internet2 booth. • The average bandwidth allocated across SCInet was around 841Mbps. • The average was about 1.6Gbps if you exclude the 100Mbps "Pinger" circuits built from the Internet2 booth

17. Hybrid Networking • There has been interest from many communities for the development of network architectures and mechanisms that utilize lower layers of the protocol stack along with IP at layer 3 • This has become known as “hybrid networking” • It is motivated by applications from the research and education community that require greater capabilities • High bandwidth flows (for example, flows that come close to saturating links in the shared IP backbone) • Flows with special requirements related to quality of service, for example jitter requirements • Network and Application Virtualization

18. Hybrid Networks - Motivating Factors • Hybrid networks are intended to provide a flexible mix of IP routed service and “lower layer services” • “flexible” means the network can respond quickly to user/application/connector requirements and requests to access both the IP Routed and/or lower layer services • “lower layer services” means access to layer 2 and below paths which can be utilized in a multitude of ways by creative users. • Typical user requirements for these lower layer services are based on: • critical, large bandwidth flows which may require one of more of the following: deterministic network performance, dedicated network resources, guaranteed network capacity, freedom to use protocols other than (congestion control friendly) TCP, privacy/security requirements, scheduled services • User/application communities which desire to build entire topologies which integrate domain specific resources along with dedicated network resources (which have one or more of the above mentioned characteristics)

19. Hybrid NetworksHeterogeneous By Nature Hybrid networks are extremely heterogeneous at several levels DataPlane can be constructed from router based Multiprotocol Label Switching (MPLS) tunnels Ethernet VLAN based Circuits Synchronous Optical Network / Synchronous Digital Hierarchy (SONET/SDH) circuits Wavelength Division Multiplexing (WDM) connections Combinations of the above

20. Hybrid NetworksHeterogeneous By Nature Control Planes can be based on Multiprotocol Label Switching (MPLS) Generalized Multiprotocol Label Switching (GMPLS) Web Services Management Systems Combinations of the above Client (user) services or attachment points could be Ethernet SONET IP Router InfiniBand

21. Multi-Domain, Multi-Layer Control Planes Key Requirements The “Multi-Layer” is meant to identify several items regarding how hybrid networks may be built. In this context it includes the following: Multi-Technology - MPLS, Ethernet, Ethernet PBB-TE, SONET, NG-SONET, T-MPLS, WDM Multi-Level - domains or network regions may operate in different routing areas/regions, and maybe be presented in an abstracted manner across area/region boundaries Multi-Domain indicates that we want to allow hybrid network service instantiation across multiple domains And of course all this implies that this will be a Multi-Vendor environment. Multi-Control – mpls, gmpls, management, vendor proprietary

22. Multi-Domain, Multi-Layer Hybrid Networks Hybrid networks are intended to provide a flexible mix of IP routed service and dedicated capacity “circuits” The “Multi-Layer” is meant to identify several items regarding how hybrid networks may be built. In this context it includes the following: Multi-Technology - MPLS, Ethernet, Ethernet PBB-TE, SONET, NG-SONET, T-MPLS, WDM Multi-Level - domains or network regions may operate in different routing areas/regions, and maybe be presented in an abstracted manner across area/region boundaries Multi-Domain indicates that we want to allow hybrid network service instantiation across multiple domains But there are other "Multi-" parameters as well

23. Multi-Domain, Multi-Layer Hybrid Networks Multi-Service: This refers to the client experience when they connect to the edge of a dynamic network. Typical service definitions are characterized by the combination of the physical port type (e.g. Ethernet, SONET/SDH, Fibre Channel, etc), the network transport instance (e.g. IP Routed, Ethernet VLAN, SONET), and performance characteristics (e.g. bandwidth, QoS specifications). Multi-Vendor: This is a reflection that advanced networks will be constructed based on technologies from multiple vendors. A key challenge will be to develop technologies and mechanisms which allow integrated control and service provisioning in this multi-vendor environment. Multi-Policy: Access to and use of various network components, regions, or topologies may vary by user and/or community due to provider policies. Multi-X environment

24. Multi-Level, Multi-Technology, Multi-Vendor Infrastructures • Multiple Options, most will have vendor proprietary control and management mechanisms which only work across single vendor regions Ethernet Layer Routers Switched WDM Optical Layer Ethernet PBB-TE Switched WDM Optical Layer Ethernet Layer Switched SONET Layer (vcat, lcas)

25. Multi-Level, Multi-Technology, Multi-Vendor – Network Virtualization • Network Virtualization and Topology Building in Multi-Level, Multi-Technology, Multi-Vendor Infrastructures Bandwidth Request was smaller, so provision Ethernet, then router connection Same Result with Either Approach Provisioned Topologies Routers Ethernet PBB-TE Bandwidth Request was large enough to justify provisioning at WDM layer Switched WDM Optical Layer End Points might attach at different levels: How to flexibly provision at what ever level an end point might appear?

26. Multi-Level, Multi-Technology, Multi-Vendor Infrastructures • Current dynamic provisioning environment can be described as: • Static Topology, Dynamic Provisioning • Next we want to enable: • Dynamic Topology, Dynamic Provisioning

27. Dynamic Network ServicesIntraDomain • Source Address • Destination Address • Bandwidth • VLAN TAG (untagged | any | tagged | tunnel) • User Identification (certificate) • Schedule Circuit Request Dynamically Provisioned Dedicated Resource Path (“Circuit”) DRAGON Enabled Control Plane Internet2 IDC Client B XML Ethernet Mapped SONET or SONET Circuits USER API Client A Internet2 DCN Service • api can run on the client, or in a separate machine, or from a web browser Actual Network Path

28. Dynamic Network Services InterDomain No difference from a client (user) perspective for InterDomain vs IntraDomain 2 2 USER API A A 1 XML RON Dynamic Infrastructure Ethernet VLAN RON Dynamic Infrastructure Ethernet VLAN Internet2 DCN Ethernet Mapped SONET A. Abstracted topology exchange 1. Client Service Request 2. Resource Scheduling 5. Service Instantiation (as a result of Signaling) Multi-Domain Dynamically Provisioned Circuit

29. DCN Control Plane

30. DCN Control Plane Software • OSCARS (Web Service) • Started by ESnet, merged with Internet2’s BRUW project in 2006 • Web service architecture, interfaces to lower level network specific provisioning systems • Vendor based MPLS L2VPN (Martini Draft) • Internet2 DCS/HOPI • DRAGON (NSF funded project in development by USC/ISI EAST and MAX) • Uses GMPLS protocols to build layer 2 circuits

31. I2 DCN Software Suite • OSCARS (IDC) • Web service layer, InterDomain messaging, AAA, Scheduling • DRAGON (DC) • Control of domain network elements (Core Directors and/or Ethernet Switches) • Intra and Inter Domain Path Computation • RSVP based signaling • Version 0.4 of DCNSS deployed December, 2009 • https://wiki.internet2.edu/confluence/display/DCNSS

32. OSCARS-DRAGON Integration

33. DRAGON • Virtual Label Switched Router(VLSR) • PC based control plane software • Manages and provisions various network equipment such as ethernet switches, SDH/SONET • Signaling with RSVP packets • Network Aware Resource Broker (NARB) • Stores topology in OSPF-TE database • Performs inter/intradomain path calculation • Exchanges interdomain topology

34. IDC - Web Service Based Definition • Four Primary Web Services Areas: • Topology Exchange, Resource Scheduling, Signaling, User Request

35. Other AAA Models Possible MetaScheduler Topology Topology Scheduling Scheduling Signaling Signaling • Meta-Scheduler Approach • Same set of Web Services used for linear instantiation model can be used by a high level process to build services: • Topology Exchange, Resource Scheduling, Signaling, User Request • A key issue is that this requires a trust relationship between the “meta-scheduler” and all the domains with which it needs to talk

36. InterDomain Controller (IDC) Protocol (IDCP) Developed via collaboration with multiple organizations USC/ISI, ESnet, Internet2, GEANT2, Nortel, University of Amsterdam, others The following organizations have implemented/deployed systems which are compatible with this IDCP Internet2 Dynamic Circuit Network (DCN) (use of DCN Software Suite) ESNet Science Data Network (SDN) (use of OSCARS) GÉANT2 AutoBahn System (AutoBahn system implementation) Nortel (via a wrapper on top of their commercial DRAC System) Surfnet (via use of above Nortel solution) US LHCNet (use of DCN Software Suite) CalTech (use of DCN Software Suite) Nysernet (use of DCN Software Suite) Northrop Grumman Corp. (use of DCN Software Suite) LEARN (Lonestar Education and Research Network) (use of DCN Software Suite) LONI (Louisiana Optical Network Initiative) (use of DCN Software Suite) University of Amsterdam (use of DCN Software Suite) JGN2 (use of DCN Software Suite) MAX/DRAGON Network (use of DCN Software Suite) The following "higher level service applications" have adapted their existing systems to communicate via the user request side of the IDCP: LambdaStation (FermiLab) TeraPaths (Brookhaven) Phoebus http://www.controlplane.net/

37. 10/15/2008 FMM New Orleans 37

38. InterDomain Controller Protocol Standardization Activities Standardization process and increasing community involvement continues Optical Grid Forum (OGF) Network Markup Language (NML) Working Group Standardizing topology schemas (perfsonar and control plane) Network Services Interface (NIS-WG) Grid High Performance Networking (GHPN) Research Group Network Measurement (NM-WG) Network Measurement Control (NMC-WG) Information Services (IS-WG) GLIF Control Plane Subgroup working on normalizing between various interdomain protocols (IDCP, G-Lambda GNS-WSI, Phosphorus API) Also other GLIF subgroups in this and related space (global id format, PerfSonar)

39. Internet2 DCN Working Group • DCN WG has been formed under NTAC • Chair: Linda Winkler (Argonne National Laboratory) • DCN WG will drive directions and set agenda in this area • Mailing list and Wiki available • dcn-wg@internet2.edu • https://spaces.internet2.edu/display/DCN/Home

40. Internet2 DCN Infrastructure

41. Internet2 DCN Services 1-A-5-1-1 1-A-6-1-1 1-A-6-1-1

42. DCN Services - circuits Physical Connection: 1 or 10 Gigabit Ethernet SONET (Future) Circuit Service: Point to Point Ethernet (VLAN) Framed SONET Circuit Point to Point SONET Circuit (future) Bandwidth provisioning in 100 Mbps increments How do Clients Request? Client must specify [VLAN ID | ANY ID | Untagged | Tunnel], SRC Address, DST Address, Bandwidth Request mechanism options are Web Service API, Web Page, phone call, email What is the definition of a Client? Anyone who connects to an ethernet or SONET port on an Ciena Core Director; could be RON, other wide area networks, domain specific applications

43. DCN Services - topologies Individual circuits are the “atomic” service provided by the DCN and control plane These circuits could be intra or inter domain It is envisioned that higher level “services” may be developed which coordinate the instantiation of multiple individual circuits to develop entire “topologies” co-scheduling/allocation of other resources (compute, data storage) may also be desired Probably a task for individual science/application domains or someone developing middleware on their behalf

44. Workshop Details

45. Control Plane Data Plane DCN Workshop Architecture Internet2 Core Dynamic Circuit Network Green Pod ASN4 Red Pod ASN1 Yellow Pod ASN3 Blue Pod ASN2

46. Pod Network Elements Control Plane PC (VLSR#-PC, NARB, IDC) Data Plane Ethernet Switch (VLSR#-SW) End System (ES#) Inter-Domain Controller – “IDC” Network Aware Resource Broker- “NARB” NARB / IDC Virtual Label Switching Router- “VLSR” VLSR2 VLSR3 VLSR1 VLSR3-PC VLSR1-PC VLSR3-SW VLSR1-SW ES1 ES2

47. Basic Pod Data Plane Data Plane Ethernet Switch (VLSR#-SW) End System (ES#) VLSR2-SW Ethernet Switch D2 D4 VLSR3-SW VLSR1-SW D3 D5 D1 ES1 ES2 End System Data Plane via Cat5 Patch Cable Data Plane (D#)

48. Control Plane PC (VLSR#-PC, NARB, IDC) End System (ES#) Basic Pod Control Plane Network Aware Resource Broker- “NARB” Inter-Domain Controller – “IDC” NARB / IDC Virtual Label Switching Router- “VLSR” gre6 gre2 gre4 gre3 VLSR3-PC VLSR1-PC

49. Pod Network Elements Control and Data Planes Control Plane PC (VLSR#-PC, NARB, IDC) Data Plane Ethernet Switch (VLSR#-SW) End System (ES#) Network Aware Resource Broker- “NARB” Inter-Domain Controller – “IDC” NARB / IDC gre6 Virtual Label Switching Router- “VLSR” VLSR2 gre2 gre4 VLSR3 VLSR1 VLSR3-PC gre3 VLSR1-PC D2 D4 VLSR3-SW VLSR1-SW D3 D5 D1 ES1 ES2

50. Pod Management Addressing “Red” pod: ASN=1 “Blue” pod: ASN=2 “Yellow” pod: ASN=3 “Green” pod: ASN=4 Workshop Gateway Router 192.168.1.1 Management VLAN 192.168.<asn>.n/16 .10 NARB / IDC VLSR2 .6 VLSR1 VLSR3 .5 eth0 .4 .8 .7 .3 .9 eth0 eth1 eth1 eth0 .2 ES1 ES2 eth0 - Management Plane Interface and Control Channel (PCs) eth1 - Data Plane Interfaces (PCs)