Request for feedback on proposed direction for CHEETAH

1. Request for feedback on proposed direction for CHEETAH Malathi Veeraraghavan University of Virginia Sept. 22, 2006 • Outline • A quick status report on CHEETAH • Proposed direction discussion: • What's our goal for the CHEETAH network: • eScience network or a scalable GP network? • Bandwidth sharing modes: • Book-Ahead (BA) or Immediate-Request (IR)? • If IR, what applications? • Core or edge network?

2. CHEETAH project status • Network deployment • data plane • control plane • Networking software • Network service • Applications • eScience • general-purpose • Basic research on circuit/VC networking

3. UVa Via Vortex/HOPI CUNY Via Nysernet/HOPI GaTech CHEETAH network - data plane linksGbEthernet and SONET TN PoP SN16000 GbE GbE NCSU via NCREN GbEs GbE/ 10GbE card OC192 card Control card GbE End hosts OC-192 (via NLR/SLR/NCREN) NC PoP GA PoP SN16000 SN16000 GbE GbE/ 10GbE card Control card OC192 cards GbE/ 10GbE card GbE OC192 card Control card End hosts End hosts OC-192 GbE GbE ORNL

4. CHEETAH network - control plane links Design goal: scalable GMPLS network UVa SN16000 Openswan IPsec software on Linux end hosts CUNY GbE/ 10GbE card OC192 card Control card TN NCSU End hosts ns5 IPsec device Internet2 Call setup messages ns5 ns5 NC GA GbE/ 10GbE card Control card OC192 card GbE/ 10GbE card Control card End hosts OC192 card End hosts SN16000 SN16000 GaTech ORNL

5. Networking software • Sycamore switch comes with built-in GMPLS control-plane protocols: • RSVP-TE and OSPF-TE • We developed CHEETAH software for Linux end hosts: • circuit-requestor • allows users and applications to issue RSVP-TE call setup and release messages asking for dedicated circuits to remote end hosts • CircuitTCP (CTCP) code • Described in two ICC06 papers and a JSAC submission

6. Network service • On-demand circuit-switched service for 1Gb/s dedicated host-to-host circuits • Call setup delay: 1.5sec • Sycamore gave us a proprietary build for hybrid GbE-SONET-GbE circuits • No standard yet for such hybrid circuits • Sets up 7 OC3s and VCATs them to carry a GbE signal • In contrast, their GMPLS standards implementation for pure-SONET circuits incurs a call setup delay of 166ms (2-hop)

7. Applications • eScience: Terascale Supernova Initiative • File transfers: demo'ed but • held up by Cray X1 network I/O problems • Ensight remote visualization: demo'ed but • connectivity between NCSU CHEETAH-drop site and physicist's office is low speed • Conclusion: other factors, not the network! • general-purpose: • web caching • video apps

8. Basic research on bandwidth-sharing mechanisms in circuit/VC networks • Immediate-Request (IR) mode • Purpose: to determine what type of applications are well served by this mode • Key finding: if m, the link capacity expressed in channels, is 10, IR (Immediate Request) mode of bandwidth sharing leads to poor util. or high blocking • Published in an IEEE Globecom 2006 paper • Book-ahead bandwidth-sharing mechanisms • Developed a novel discrete-time Markov chain model for book-ahead bandwidth-sharing mechanisms • Results are submitted to IEEE/ACM Trans. on Networking)

9. Outline check • A quick status report on current CHEETAH • Proposed direction discussion: • What's our goal for the CHEETAH network: • eScience network or a scalable GP network? • Bandwidth sharing mode: • Book-Ahead (BA) or Immediate-Request (IR)? • If IR, what applications? • Core or edge network?

10. Observation • "Many e-science experiments ... are optimized to provide maximum throughput to a few facilities, as opposed to moderate throughput to millions of users, which is the raison d'etre for commercial networks."

11. eScience networks • eScience network requirements • Number of users small • Hard to achieve high utilization; also not impt. • Overprovision network to keep call blocking rate low • Focus on creating software to allow scientists to automatically provision high-speed application-specific topologies: AST, UCLP, OSCARS, USN scheduler, BRUW • Bandwidth-sharing algorithms of less concern

12. General-purpose commercial networks • Have to be scalable: large number of users • Metcalfe's statement: Value of a network increases exponentially with the number of users • High utilization is an important goal • Low call blocking probability or low waiting time for resources • Focus on efficient bandwidth-sharing algorithms

13. Circuit/VC service on GP commercial networks • Just for ISPs/enterprise admins: • needs similar to eScience • router-to-router circuits • limited number of users • high-bandwidth, long-held circuits • low price not a high priority • need BA mode of bandwidth sharing • For end users • large number of users • can only offer moderate BW and limited call holding times • IR mode of sharing becomes feasible

14. Outline check • A quick status report on current CHEETAH • Proposed direction discussion: • What's our goal for the CHEETAH network: • eScience network or a scalable GP network? • Bandwidth sharing modes: • Book-Ahead (BA) or Immediate-Request (IR)? • If IR, what applications? • Core or edge network?

15. BW sharing modes in circuit/VC networks m is the link capacity expressed in channels e.g., if 1Gbps circuits are assigned on a 10Gbps link, m = 10 • Mean waiting time is proportional to mean call holding time • Can afford to have a queueing based solution when m is small if calls are short Large m Moderate throughput Small m High throughput immediate-request with call blocking + retries ("call queueing") (video, gaming) Short calls Bank teller Long calls Doctor's office immediate-request with delayed-start times ("call queueing") (file transfers) book-ahead

16. Impact of increasing m at different values of link utilization Ud m=10 Pq=41% Prob. of arriving job finding all m circuits busy Offered load: call arrival rate/call departure rate Link capacity expressed in channels High-rate per-call circuits Low-rate per-call circuits

17. Impact of mean call holding time Number of ports aggregating traffic on to the link Mean waiting time for delayed calls Ud: 90% : per host call-generation rate

18. Main findings of analysis • Two key parameters: • If m is small (per-circuit BW is high) • and mean call holding time is large • then need BA to avoid long waiting times • and mean call holding is small (file transfers) • then use "call queueing" • If m is large, switch hardware costs increase • N, number of aggregation ports, high • level of demultiplexing high

19. Relate BW sharing modes to network types

20. Support for the BA mechanism of bandwidth sharing • Since RSVP-TE does not have parameters for BA calls (call duration, start time), this mode is not implemented in switch controllers • Cannot utilize the BW management software implemented in switch controllers as part of GMPLS control-plane software • Need an external scheduler to manage bandwidth into the future • Easiest to make it centralized - one per domain • The BA mode is necessary for high-BW, long-held calls

21. BA implementation approach • Develop and "standardize" protocols for scheduler-to-scheduler signaling for interdomain circuits (one centralized scheduler per domain) • e.g., Chin Guok (ESNET)'s WSDL spec. for ESNET-Abilene testing (OSCARS-BRUW) • Implement scheduler and test with other networks • Create software tools to enable scientists and ISP/enterprise admins to visualize network topologies and request appropriate circuits/VCs • High-BW, long-held: Therefore AAA is a must • Path being pursued by DRAGON, USN, OSCARS, UCLP

22. Argument: IR is just a "now" in BA • Difficult to have BA and IR coexist without some form of bandwidth partitioning • BA allows for high-BW, long-duration calls • IR calls will suffer a high call blocking rate if supported through BA scheduler (the "add-now-as-an-option-in-scheduler" solution) • Should you admit an IR call if it arrives a few seconds before start time of a BA call and hope it completes before the BA call start time, or reject the call and waste bandwidth? • If BW is partitioned, then implement scalable solution for IR - distributed bandwidth-management

23. Support for the IR mechanism of bandwidth sharing • Switches have built-in (G)MPLS control-plane software (RSVP-TE/OSPF-TE) • Bandwidth management is part of RSVP-TE switch controller software • Hence it is distributed bandwidth management • Need to limit call holding time - reminders for renewals and automatic release • Moderate-to-high per-call bandwidth

24. Is an opportunity being missed if distributed IR bandwidth sharing mode is not explored? • What opportunity? Four reasons: • Enable the creation of large-scale circuit/VC networks with moderate-rate circuits that can support a brand new class of applications • economic value for the networking industry • A "reservations-oriented" mode of networking to complement today's connectionless Internet • ala airlines that complement roadways • Could prove useful to FIND, GENI, net-neutrality • Alternative pricing models for bandwidth

25. Outline check • A quick status report on current CHEETAH • Proposed direction discussion: • What's our goal for the CHEETAH network: • eScience network or a scalable GP network? • Bandwidth sharing mode: • Book-Ahead (BA) or Immediate-Request (IR)? • If IR, what applications? • Core or edge network?

26. What "brand new class of applications?" • Large-m (moderate BW) • Video, video, video • Gaming • Remote software access • Small-m (high BW) short-held calls • Async storage • Web and P2P file transfers

27. Video applications • Improve quality of conferencing, telephony, surveillance, entertainment and distance-learning by a significant degree • Expend bandwidth for a higher-quality, lower latency, multi-camera, auto-movement, auto-mixing experience • Make the "flat world" flatter • Energy savings/environmental benefits • Moderate bandwidth - IR with call blocking/retries

28. Gaming applications • Current gamers buy personal graphics cards • Players talk of "lag" caused by differences in graphics processing speeds • Moderate-speed circuits can enable a new class of games in which rapidly-changing scenes are possible • compare movies in which multiple story lines keep scenes changing vs. gaming scenes • Players connect to graphics servers • Data transferred is not GL commands, but rather rendered data (doable?) • Moderate bandwidth - IR with call blocking/retries

29. Remote software access • Remote software access • Reduce computer administration cost • Personal computers vs. machine rooms • I loaded 22 new applications on my new laptop • Instead: connect and run! • Virtual Computing Laboratory: Mladen Vouk, NCSU • Moderate bandwidth - IR with call blocking/retries

30. Asynchronous storage • Asynchronous storage depots will lower costs for • backups • disaster recovery • Need for increased storage grows with multimedia files • High bandwidth, short calls • IR with delayed start

31. Larger files in web sites and P2P apps • Multimedia files in web sites • Increase the use of video/audio files in all sorts of web sites instead of ASCII • My own course PPT files: I use audio sparingly because of bandwidth • Think assembly instructions for electric fans, furniture • Kinesthetic learning - show me a video • Think hotel web pages • Show me exactly where the beach is relative to my room; do I have a balcony - saying it in text format is one thing; seeing it in a video format quite another! • Content distribution network, mirroring & web caching • High bandwidth, short calls • IR with delayed start

32. Do these apps really require circuit/VC networks with IR mode or can they simply use higher-BW IP based networks? • See analogous transportation networks (reason 2) • reservations-oriented airlines network • reservationless roadways network • Hypothesis: The socialistic mode of bandwidth sharing on the Internet discourages individual investment in network bandwidth (reason 4) • should we pay for bandwidth with tax dollars - "free" for the whole community? • "Tragedy of the commons" (Tanenbaum) • should we create a network where individuals can pay for bandwidth on congested links more directly? - think higher-toll HOV lanes

33. Invest in a complementary reservations-oriented network • Build a scalable circuit/VC network in which bandwidth is shared in IR mode • Scalability will create "Metcalfe's value" • Provides an opportunity to finally recoup our investment in (G)MPLS technologies • standards creation effort • implementation: Cisco, Juniper, Sycamore, Movaz • Assign at least a few of the optical testbeds that we are investing in now to study whether this IR mode of bandwidth sharing can help with our understanding of net-neutrality, economic growth, FIND questions • IR more natural in data world unlike in airlines (BA)

34. Outline check • A quick status report on current CHEETAH • Proposed direction discussion: • What's our goal for the CHEETAH network: • eScience network or a scalable GP network? • Bandwidth sharing mode: • Book-Ahead (BA) or Immediate-Request (IR)? • If IR, what applications? • Core or edge network?

35. Core vs. edge networks • Should the GMPLS IR scalable network be developed first as a core network or as a set of edge networks that feed into an IP-based core network? • Edge-network oriented apps: HOV bypass • Video • Gaming • Remote software access • Core-network oriented apps: "3TCP connections" with wide-area TCP connection through GMPLS network • Asynchronous storage • Large web sites: CDN/web caching

36. HOV bypass • The "end-to-end" in the CHEETAH name • We chose Ethernet in LANs and SONET in WANs for the data-plane, given the dominance of these two technologies, with the thinking that this would make "end-to-end" circuits affordable • BUT, even adding control-plane software modules to each Ethernet switch in a departmental closet, enterprise closet, etc. is quite expensive

37. HOV bypass • Further observation • if a link is lightly loaded, we don't really need to reserve bandwidth for a particular flow • Combining these two: • Concept of "partial-path circuits" • Determine if there is a bottleneck link on the end-to-end IP path • If so, send signaling request to a router on the bottleneck link asking for a BW reservation on bottleneck link (just as with HOV lanes)

38. Key idea • Use control-plane signaling request from end user or application to ask IP router for a "bypass" reservation and to get ready to isolate packets of its flow (Policy Based Routing) • Unlike ATM cut-through VCs solutions that required IP routers to somehow determine which flows to handle via ATM VCs • Ipsilon: long-lived flow detection • Comparable to the user who calls ahead and reserves a seat for a flight, before showing up at the airport

39. Enterprise network WAN-access router Use IR mode for HOV bypass WAN-access router WAN-access router bypass controller Enterprise network Enterprise network Edge network (e.g., NYSERnet) dynamically setup MPLS tunnel for period of flow Backbone network (e.g., Abilene) NYC PoP NC PoP likely lightly loaded links Edge network (e.g., NCREN) Enterprise network access: likely congested links WAN-access router

40. Using CHEETAH ideas in enterprise access for the HOV bypass • MSPPs were being deployed in enterprises to aggregate voice and IP traffic on to SONET WAN access links WAN-access router Enterprise network bypass controller Edge network (e.g., NYSERnet) add additional OCxx circuit (router needs extra or higher-BW interface already plugged in) • But increasingly Metro Ethernet is replacing SONET • MPLS seems more suitable

41. Core network solution:3TCP connections Airlines Roadways Roadways Airport Airport Appears that a comparable reason exists here for taking a flight: long-distance travel CAG: CHEETAH Application Gateway A web cache (using squid) with integrated cheetah software

42. Splitting high-BDP TCP connection • Improves end-to-end delay even if all three TCP connections used IP paths • Micah Beck, Logistical networking • Martin Swany, Phoebus • Sigcomm 05 Purdue poster • Many others

43. Cheetah improvement • Have the wide-area path go through a circuit/VC network • if RTT is 50ms, even if circuit rate is 10Mbps when Internet path bottleneck link rate is 100Mbps, there is a crossover file size beyond which the circuit path is faster - even with BIC • Trigger for call setup through CHEETAH arrives from web proxy connections initiated by clients • Thus, users not physically connected to CHEETAH still use CHEETAH service • CDN and mirror apps will be triggered by web server-to-local mirror writes

44. Web cache Web cache Web cache CHEETAH-HOPI interconnection Web cache VLAN Web cache Web cache McLean, VA NC Web cache MPLS 10GbE Web cache CHEETAH SONET HOPI network: courtesy of Rick Summerhill TN GA

45. CHEETAH as a core network • Job of a core network is to connect edge networks • Application just described only connects web servers, web caches, storage devices to CHEETAH SONET switches at PoPs • Could also provide router-to-router IR-triggered circuits between PoPs • Allow admin-triggered setups • Network management software for load monitoring and automatic triggers

46. Add routers and servers/storage to CHEETAH PoP NC PoP TN PoP Routers ...... Routers ...... Web servers/ mirrors OC192 Storage devices Web caches Storage devices Web caches Web servers/ mirrors GA PoP Routers OC192 ...... Storage devices Web caches Web servers/ mirrors

47. Conclusions • Proposed direction: • What's our goal for the CHEETAH network: • eScience network or a scalable GP network? • Bandwidth sharing modes: • Book-Ahead (BA) or Immediate-Request (IR)? • If IR, what applications? • Long-distance file transfers - web caching/CDN/mirrors and storage + router-to-router triggers • Core or edge network? • Interesting research: delayed-start BW-sharing