CHEETAH: Circuit-switched High-speed End-to-End Transport ArcHitecture

1. CHEETAH: Circuit-switched High-speed End-to-End Transport ArcHitecture • Concept/analysis • Opticomm 2003 paper • CHEETAH: a scalable solution for wide usage • Focus: File transfers • Practical implementation • NSF EIN project • Focussed on eScience project - Supernova • NSF ANIR hardware signaling project • NSF ITR project • Commerical applications: financial and retail institutions transfer large files

2. CHEETAH concept • What is it? • Dynamic end-to-end circuits • Circuit: Ethernet –EoS – Ethernet • Leverages • Ethernet – King of LANs • SONET – King of MANs • Why this solution? • Ethernet and SONET already deployed • Add necessary upgrades to switches and software to end hosts to enable the use of these circuits

3. Add-on solution to Internet

4. Leverage presence of Internet path • Run circuit-switched network in call blocking mode • If call is blocked, fall back to Internet path • Engineer network for high utilization at the cost of blocking • File transfers: Use Internet path for reverse direction error control and flow control messages and some retransmissions • Same approach for other applications

5. What extra “stuff” is needed to realize this CHEETAH concept? • At circuit switches: • Ability for dynamic fast provisioning of circuits • At end hosts: • Applications upgrade to use circuits • A transport protocol • Signaling protocol client end • Routing decision module – choose between two network choices available to privileged CHEETAH end hosts

6. Provisioning research community • Example: Canarie’s UCLP • Created network to map Ethernet – EoS – Ethernet “lightpahts” with equipment like 15454s • 15454s do not implement routing/signaling/LMP protocols • For provisioning lightpaths • Created external databases to manage routing information: to reach a destination host, what is the set of switches to traverse? • Create external databases to manage inventory information - which 454 has what cards? • Inter-domain issues (multiple carriers) – policy manager • Then set up circuit with TL1 commands • Another example: Elematics – software company that created NMSs for routing, signaling functionality • key: not integrated with network elements

7. Fine solution • if we just want to build small-scale networks • Our goal: • large-scale connection-oriented network • why? reduce costs • the very concept of eScience is possible only because of the cheap wide-scale Internet • the very scientists we strive to serve today will only be served well if we create large-scale connection-oriented networks capable of providing rate-guaranteed connectivity! • Value of the network grows exponentially with number of scientists connected • Example: the huge-scale 64kbps telephone network would not have been possible without signaling integrated into switches

8. Problems with provisioning with NMS’s • Scalability • My pet peeve: • Call setup delays of in the order of seconds too high • Using network management systems outside the network elements to manage • routing data • inventory data takes significant effort – Operations costs • Other problems • Interoperability of different vendors’ equipment • Inter-carrier issues

9. Scalability “charges” • Levied against Optical/TDM networks: • “Widely recognized that the current GLIF optical/TDM networking model does not scale beyond a limited number of sites” Internet 2 talk dated 10/15/2003 • “While such circuit-switched networks may not necessarily be suitable for deployment of the scale of the Internet, they are still viable candidates for specialized deployments for connecting a small number of DOE large-scale science nodes” Report of DOE Workshop on Ultra High-Speed Transport Protocols and Dynamic Provisioning for Large-Scale Science Applications, April 10-11, 2003, Argonne, IL, dated Oct. 27, 2003. GLIF: Global Lambda Integration Facility

10. Inventory problem • TL1 command to set up a crossconnect through a 15454 • Command: • ENT-CRS <STS_PATH>:[<TID>]:<FROM>,<TO>:<CTAG>::[<CCT>][::]; • Example: • ENT-CRS-STS1:BODEGA:STS-5-1,STS-12-5:116::2WAY; • TID: unique name for the system • From and To: Access Identifiers – to identify timeslots on interfaces • STS-1 on the card in Slot 5 • STS-5 on the card in Slot 12 • CTAG: unique identifier used to match response with request • CCT: Crossconnection type: e.g., 1WAY or 2WAY

11. My take on networks • Network switches should be equipped with all the functionality needed for plug-and-play operation • just as seamless as booting up a PC these days with a network card - because of DHCP • DHCP not only gets IP address • It discovers a gateway address too • This implies everything - both back-stage and front-stage • back-stage: • address acquisitions • automatic neighbor discovery (manages its own inventory) • automatic topology discovery through routing protocol • routing table computation • front-stage: • packets show up – forward them as per routing table • Amazingly: most of the above achieved with connectionless packet switches • Ethernet switches • Even IP routers?

12. To create a connection-oriented (CO) network in the same plug-and-play vein • Additional back-stage functions: • handle call setup requests/releases: signaling protocol • perform authorization and authentication before call setup • encryption of data on user-plane irrelevant? • gather call-level data for billing • CO networks offer us the opportunity to build capitalistic networks in addition to today’s CL socialistic Internet • Front-stage • circuit-switched networks • data shows on interfaces – switch based on “position” – space, time, wavelength • CO PS networks • switch packets based on header data but switch according to resource allocation for CO

13. Network management • NM functions • Fault management • Performance monitoring • Accounting management • Security management • Configuration management • overall planning of topology • All these are essential to the running of a network but these are back-office operations! • Hence fine to offload these to NMSs. • But leave front-office operations at the NEs.

14. Signaling approach to connection setup: distributed • Call setup request carries destination IP address D + bandwidth B + incoming timeslot/l • Lookup routing data table (same function as in an IP router) • find outgoing interface O to reach destination D • Resource allocation • Allocate bandwidth B on interface O • Select outgoing timeslot/l • Program switch fabric • Map incoming timeslot/l to outgoing timeslot/l • Send call setup request to neighbor connected by interface O

15. Industry answer tosupport distributed signaling approach • IETF GMPLS • Routing – OSPF-TE • Routing built into network switches • Signaling – RSVP-TE • Link Management Protocol (LMP) • Inventory data stored in network switches • Auto-discovery of neighbors • OIF’s UNI-C, UNI-N, NNI • Addresses carriers’ inter-domain issues

16. Actually implemented! • Not just idle specifications • Implemented by many switch vendors • And interoperability-tested by an OIF-sponsored effort led by Univ. of New Hampshire • Demo’ed at OFC2003 • Demo’ed at Isocore in March 2004 • Another demo planned for Supercomm – June 2004

17. From keynote at Opticomm, Dallas, Oct. 03 • Rajiv Ramaswami, CTO , Optical Systems Group, Cisco, Keynote • UCP (Unified Control Plane) Benefits • Superfast Provisioning • Enables E2E circuit setup without SP intervention while reducing provisioning times • Enables future bandwidth on demand applications as policy & billing standards mature • Enhanced Scalability • Network level: Support for thousands of nodes, links and circuits per inter-connected network • Lightweight EMS: Move from EMS based (centralized) provisioning to node level (distributed) provisioning using signaling

18. UCP Benefits contd. • Interoperable vendor implementations • Reduces EMS/NMS integration / interoperability issues • UCP/GMPLS – A Driver for Evolution • Build Network as a Database • Simplify provisioning by driving intelligence (topology, circuit inventory and link characteristics) into the NEs with updates to EMS (CTC/CTM) • Enable migration from an NMS based network database to NEs based network database, retrievable on demand by NMS • Deliver Advanced Benefits • New services & features (Ethernet,OVPN & Storage) not possible today… • Reduce costs, increase revenues, address scale of growing networks • Enable multi – network/vendor/SP interoperability

19. Our research community • It was hard to wait for network equipment manufacturers to integrate these routing/signaling/LMP into their NEs • It was much easier for us to go off on our own and build NMS software external to switches • But now, we are there. The equipment vendors do have such NEs. Let’s use them.

20. Back to CHEETAH:Signaling protocol • Since in CHEETAH we propose holding circuits only for duration of file transfer • A 10MB file takes 800ms over a 100Mbps circuit • Reduce end-to-end call setup delay to r.t. propagation delay • We need fast signaling engines + high call handling capacity at MSPPs/switches • Solution: Hardware implementation • NSF funded project: We have completed implementing an RSVP-TE subset on an FPGA • Can handle 400,000 calls/sec • Each call setup takes 4s • Research work – hard for a university to actually build an operational switch. We are building a prototype SONET switch with a 40Gbps fabric and our signaling FPGA

21. Other aspects of CHEETAH • Transport protocol for file transfers on combination circuit + TCP/IP path • Fixed-Rate Transport Protocol (FRTP) on circuit + TCP on IP path • Routing decision module • Expected delays • Utilization consideration

22. When rc = 100Mbps and Tprop= 0.1ms rc = 1Gbps, Tprop= 0.1ms Crossover file sizes from delay perspective When Tprop = 50ms, always attempt a circuit

23. Plot of utilization u withrc= 100Mbps, k=20 For 50ms paths, set a crossover file size When load is low, operate at a high blocking rate Pb=0.3 Pb=0.01

24. CHEETAH implementation: NSF EIN project • Buy circuit switches that come close to “plug-and-play” • distributed signaling solution • distributed routing solution • auto-discovery of neighbors • Currently, only SONET switches integrated with GMPLS capability • RSVP-TE • OSPF-TE • LMP

25. Routing decision PC Routing decision Eth. Sw. PC SFTP SFTP Signaling client Signaling client NIC I NIC I TCP TCP Sig FRTP NIC II NIC II FRTP Ethernet Ctrl XC OC3 A lab demo with one circuit switch emulating a SONET network • Implement FRTP, signaling client, routing decision module at end hosts (Linux)

26. Wide-area deployment • Deploy SONET switches in Raleigh and Atlanta • Our scientist co-PIs are in NCSU and ORNL • Networking co-PIs in ORNL already have OC192 link from ORNL to Atlanta • Purchase lambda from NLR (?) for research tests – terminate on SONET OC48 cards • Test end host CHEETAH software for file transfers from ORNL Cray to NCSU (TB files created by TSI scientists) • Find other scientists who connect to ORNL in NC/DC area to test sharing of OC48 Raleigh-Atlanta link – to test signaling

27. Revision to CHEETAH thinking • Inspite of wide-spread deployed SONET network, upgrade of these switches with GMPLS software is eons away! • Now, we have the possibility of MPLS LSPs through Abilene • And dedicated circuits using VLANs through Ethernet switches • Hence, we should design a ....

28. Connection-oriented internet • To complement today’s Connectionless Internet • What’s a CO internet? • an internetwork of various CO networks • what’s CO: a network that can be asked for bandwidth before data flows • VLANs, MPLS, SONET, lambdas • User-plane interworking – with Ethernet • Control-plane interworking – each CO network could have its own signaling/routing/LMP solution

29. Conclusions • Let’s build such a connection-oriented internet!