On Using Circuit-switched Networks for File Transfers

1. On Using Circuit-switched Networks for File Transfers Ph.D. Dissertation presented by Xiuduan Fang Department of Computer Science University of Virginia September 19, 2008

2. Outline • Overview • Hypothesis • Contributions & Publications • Motivation • Theoretical component: • Design and evaluate algorithms to support file transfers on circuit-switched networks • Experimental component: • Implement and demonstrate architecture for internetworking circuit-switched networks with the Internet • Conclusions & Future work

3. Hypothesis Circuit-switched networks, with dynamic call-by-call bandwidth sharing and support for heterogeneous-rate circuits, can be used efficiently to support file transfers, and can be evolved gradually into the existing Internet. Dissertation organization end-to-end circuits? Yes No Theoretical component Experimental component Internetworking architecture • Call-admission control (CAC): • rate allocation • minimum file size

4. Key Contributions • File transfers on a hybrid architecture • Constructed analytical models • Provided insights on how to design admission control • Proposed a novel heterogeneous rate-allocation scheme to lower file-transfer delay • Internetworking architecture • Designed and implemented a gateway to interconnect circuit networks with the Internet • Characterized the gateway performance

5. Publications • Ph.D. dissertation: • X. Fang and M. Veeraraghavan, On using circuit-switched networks for file transfers,” accepted to IEEE Globecom, New Orleans, LA, Nov. 2008. • X. Fang, M. Veeraraghavan, M. E. McGinley, and R. W. Gisiger, “An overlay approach for enabling access to dynamically shared backbone GMPLS networks,” in Proc. of IEEE ICCCN2007, Honolulu, Hawaii, Aug. 2007. • X. Fang and M. Veeraraghavan, “On using a hybrid architecture for file transfers,” Submitted to IEEE Transactions on Parallel and Distributed Systems, 2008. • MS thesis: • M. Veeraraghavan, X. Fang, and X. Zheng, “On the suitability of applications for GMPLS networks,” in Proc. of IEEE Globecom, San Francisco, CA, Nov. 2006. • X. Fang, X. Zheng, and M. Veeraraghavan, “Improving web performance through new networking technologies,”IEEE ICIW'06, Guadeloupe, French Caribbean, February 23-25, 2006.

6. Outline • Overview • Hypothesis • Contributions & Publications • Motivation • Theoretical component: • Design and evaluate algorithms to support file transfers on circuit-switched networks • Experimental component: • Implement and demonstrate architecture for internetworking circuit-switched networks with the Internet • Conclusions & Future work

7. Motivation • Why File Transfers on Circuit Networks? • Packet switching is considered better than circuit switching for file transfers • Pros: high throughput under light loads • Cons: • Unpredictable delays • Proportional fairness but no temporal fairness • eScience community is using high-speed circuit-switched networks for very large file transfers • Predictable service time (admission control) • Temporal fairness: give deference to job seniority

8. Dissertation Organization end-to-end circuits? Yes No Theoretical component: File transfers on a hybrid architecture Experimental component: Interconnect circuit networks with the Internet Call blocking for circuit network? • Designed a gateway • Implemented software • Characterized performance No Yes Call blocking circuit network Call queueing circuit network Published in ICCCN2007 rate allocation homogeneous rate allocation • Analytical model Homogeneous Heterogeneous Submitted to TPDS • Analytical model • Simulation model • Fairness issue • Analytical model • Simulation model Blocked calls rerouted to the Internet path Accepted by Globecom2008 For large files, waiting for high-speed circuit s is a better option than being immediately rerouted to Internet path

9. Hybrid Architecture - Example Internet2's new Dynamic Circuit (DC) network Yellow nodes: Ciena CD-CI SONET switches Blue nodes: Juniper T640 IP routers Courtesy: Rick Summerhill (2006)

10. Dissertation Organization end-to-end circuits? Yes No Theoretical component: File transfers on a hybrid architecture Experimental component: Interconnect circuit networks with the Internet Call blocking for circuit network? • Designed a gateway • Implemented software • Characterized performance No Yes Call blocking circuit network Call queueing circuit network Published in ICCCN2007 rate allocation homogeneous rate allocation • Analytical model Homogeneous Heterogeneous Submitted to TPDS • Analytical model • Simulation model • Fairness issue • Analytical model • Simulation model Blocked calls rerouted to the Internet path Accepted by Globecom2008 For large files, waiting for high-speed circuits is a better option than being immediately rerouted to Internet path

11. Call-blocking Circuit Network • Goal: design efficient connection-admission control (CAC) algorithms • Metrics: file-transfer delay and utilization • Block call if circuit is unavailable; reroute to Internet • Our focus: • What is an appropriate minimum file size? • Serve files sized x > minimum file size, Â,via the circuit network • What is an appropriate circuit rate, r, for a file transfer?

12. Analytical Model Assumptions: • Single class • homogeneous rate allocation • m circuits; per-circuit rate, r=C/m • Call arrival process: Poisson with rate, ¸0[Paxson95] • Call holding times: Pareto distribution [Crovella97] Internet N ¸0 x > Â Y ¸0 1 Link L capacity C routing decision … n Circuit network [Paxson95] V. Paxson and S. Floyd, "Wide area traffic: the failure of Poisson modeling," Networking, IEEE/ACM Transactions on , vol.3, no.3, pp.226-244, Jun 1995 [Crovella97]  M. E. Crovella and A. Bestavros, Self-Similarity in World Wide Web Traffic: Evidence and Possible Causes, IEEE/ACM Transactions on Networking, 5(6):835--846.

13. Key Insights • Combine M/G/m/m loss model & TCP delay model • Erlang-B formula: input the number of channels, m,& traffic load; output: call blocking probability and utilization • TCP model: bottleneck link rate, round-trip time, packet loss rate [Padhye98] • Two criteria to select  • Delay-based (user-perspective): compare delay estimates across two paths • Utilization (service provider-perspective): make circuit-setup overhead a small fraction (e.g., 10%) of circuit file-transfer delay • Define a metric to quantify mean delay reduction R = s-1(E[Ttcp(x)]-E[Tcircuit(x)])¢fX(x)dx • Compute mopt (ropt = C/ mopt) & Âopt that maximize R 1 [Padhye98]J. Padhye, V. Firoiu, D. Towsley, and J. Kurose, “Modeling TCP throughput: A simple modeland its empirical validation,” in Proceedings of the ACM SIGCOMM, Aug. 1998, pp. 303–314.

14. Key Results • To maximize R, roptshould be much higher than effective throughput on the Internet path • e.g., Internet path: bottleneck link rate = 100 Mb/s, RTT = 50 ms, packet loss rate = 1% ) effective throughput = 1.9 Mb/s Circuit path: link capacity = 10 Gb/s, call-setup delay = 1 sec ) ropt = 63 Mb/s & Âopt = 75 MB • If r = 2 Mb/s ) Â = 4.5 MB ) Files of size (4.5 MB, 75MB) will get lower delay on circuits But, mean delay will increase; hence directed to Internet • Load sensitive: under low loads, • Larger per-call circuit rate, ropt • Larger ropt ) Larger minimum file size, Âopt • Relax utilization criterion to decrease Âopt • RTT sensitive: Larger ropt& Âoptfor short-RTT path

15. Dissertation Organization end-to-end circuits? Yes No Theoretical component: File transfers on a hybrid architecture Experimental component: Interconnect circuit networks with the Internet Call blocking for circuit networks? • Designed a gateway • Implemented software • Characterized performance No Yes Call blocking circuit network Call queueing circuit network Published in ICCCN2007 homogeneous rate allocation rate allocation • Analytical model Homogeneous Heterogeneous Submitted to TPDS • Analytical model • Simulation model • Fairness issue • Analytical model • Simulation model Blocked calls sent to the Internet path Accepted by Globecom2008 For large files, waiting for high-speed circuits is a better option than being immediately rerouted to Internet path

16. Homogeneous Rate Allocation • Key question: how much bandwidth should be allocated for each file transfer so that the system performance is optimized in terms of mean response time at a given effective utilization? • Metrics: mean response time • File size: bounded-Pareto distribution • Call arrival: Poisson

17. M/G/m queueing model • Goal: compute per-call circuit rate, ropt (i.e., C/mopt) • Input: • A set of m = {1, 10, 100, 1000} • Link capacity C= 10 Gb/s ) r = {10Gb/s, 1Gb/s, 100Mb/s, 10Mb/s} • Call setup delay = 1 sec • Bounded-Pareto parameters ) the first two moments of service time • Traffic load 2 (0, 1) • Output: • Effective utilization: call-setup delay overhead • Mean waiting time

18. Numerical Results Bandwidth allocation should be load sensitive

19. Heterogeneous Rate Allocation • Heterogeneous scheme: divide calls into classes based on file size & allocate each class a different-rate circuit A complete-partitioning system

20. Analytical model • Multiple separate M/G/m subsystems • Basis for classifying calls: cutoff points, Â1,…, Ân-1? • Bandwidth allocation per subsystem, C1, …, Cn? • Ideal per-call circuit rate for each class, r1, …, rn? • To compute optimal operating point that minimizes mean response time: • Mathematica optimization package • e.g., for a 2-class system • Start with an initial value for Â1 • Determine C1, C2& r1, r2 • Vary Â1 to study its impact • Fairness: • Fairness ratio: ratio of mean slowdown of 2 classes • Slowdown: ratio of waiting time to service requirement

21. File-size distribution parameters: • Smallest file size: l = 1 MB • Largest file size: u = 1 TB • Cutoff point: Â = 1000 MB • Homogeneous system is virtually divided into 2 subsystems by Â

22. Fairness Ratio (small-file to large-file) • A complete-partitioning heterogeneous scheme treats small files more fairly when compared with a complete-sharing homogeneous scheme Homogeneous system (at all utilization levels) Heterogeneous system

23. Simulation Study • Single-link: simulation results are consistent with analytical results • Multi-link: fairness study • Short-path vs. long-path calls • Work-conserving scheme: unfair to long-path calls • Proposed conditional-priority scheme: give priority to long-path calls based on queue occupancy • Small-file vs. large-file calls • Complete-partitioning heterogeneous scheme

24. Key Results • Complete-partitioning heterogeneous rate allocation • Large files allocated high-rate circuits • Lowers mean response time • Treats small files more fairly when compared with complete-sharing • Requires a network management system to monitor traffic load & dynamically update partitions • Conditional priority scheme improves the fair treatment between long-path and short-path calls

25. Dissertation Organization end-to-end circuits? No Yes Theoretical component: File transfers on a hybrid architecture Experimental component: Interconnect circuit networks with the Internet Call blocking for circuit networks? • Designed a gateway • Implemented software • Characterized performance No Yes Call blocking circuit network Call queueing circuit network Published in ICCCN2007 rate allocation homogeneous rate allocation • Analytical model Homogeneous Heterogeneous Submitted to TPDS • Analytical model • Simulation model • Fairness issue • Analytical model • Simulation model Blocked calls rerouted to the Internet path Accepted by Globecom2008 For large files, waiting for high-speed circuit s is a better option than being immediately rerouted to Internet path

26. Experimental Component • Motivation: • It is expensive to deploy a new networking technology on an end-to-end basis • As link speeds increase, high-capacity circuit switches are cheaper than packet switches • Circuit-switched (CS) networks operated in shared mode ) admission control (AC) phase • Connectionless (CL) networks have no admission control phase • So internetworking CL + shared CS is a challenge • Our solution: gateway that implements all sub-layers of the network layer with data-plane and control-plane (AC) • Metrics: reliable file transfer, circuit utilization, forwarding rate

27. Related Work • State-of-the-art: IP routers • Original purpose: interconnect connectionless networks [Cerf74, RFC791, Clark88] • Connection-oriented networks when used in the Internet are used only in leased-line mode • Proposed but not deployed: • IP-over-ATM internetworking: Ipsilon's IP switching • Routers have to "guess" which flows are long-lived • TCP switching: IP switching with protocol classifier [Cerf74] V. G. Cerf and R. E. Kahn, “A protocol for packet network intercommunication,” IEEE Transactions on Communications, vol. 22, no. 5, pp. 637–648, May 1974. [Clark88] D. D. Clark, “The design philosophy of the DARPA Internet protocols,” in SIGCOMM. Stanford, CA: ACM, Aug. 1988, pp. 106–114.

28. Internetworking Architecture Connectionless Connectionless

29. Internetworking Architecture

30. Gateway Design • Start with an open-source Web proxy software package called Squid • Data-plane: • Base functionality provided by Squid • Integrated Circut-TCP (removes Slow Start, receive-side autotuning) • Control-plane: Integrated RSVP-TE signaling client module into Squid to initiate circuit setup/release

31. Gateway Design contd. • Unpredictable rate across connectionless (CL) segments • But fixed-rate across circuit-switched (CS) segments • What if these are mismatched? • Need buffering within gateways • Buffers are finite: so possibility of losses? • Squid implementation: back-pressure mechanism; • Data not read from incoming TCP buffer if Squid buffer (controlled by read_ahead_gap) is full • Latter is full if outgoing TCP buffer is full • Leads to circuit utilization problems • Answer: main memory or disk buffering in gateways + multiplexing on circuits

32. Experimental Hypothesis A modified version of Squid software can be used as a gateway to interconnect circuit-switched networks and connectionless packet-switched networks for reliable file transfers, and can support an effective throughput of 460 Mb/s when executed on a Linux 2.6.20 host with a 2.8GHz Xeon processor and 1 GB memory.

33. Experimental setup to test if there is buffer overflow • NIC speeds: CHEETAH NIC (NIC2) = GbE, Internet NIC (NIC1) ¸ 100 Mb/s • Circuit (zelda1 $ zelda4) rate=155Mb/s, link (zelda4 $ zelda5) rate=1Gb/s • Control link rate on zelda1 ! zelda2 path to mismatch sending and receiving rates • The parameter read_ahead_gap controls CAG’s application buffer for each flow, read_ahead_gap = 16 KB (default value)

34. CAG zelda1’s forwarding rate CAG zelda1’s CPU and memory usage • Key results: • No packet loss in buffers within CAGs due to a back-pressure mechanism • Drawback: low circuit utilization • e.g., only 1/155 < 1% for 1 Mb/s bottleneck link rate CAG zelda1’s receive window size for zelda1 $ zelda4 CTCP connection

35. Improving Circuit Utilization • Configured read_ahead_gap: • e.g., when read_ahead_gap (for CAG zelda1) = 1 GB, circuit utilization = 90% for a 1-GB file transfer • Problem: unscalable because Squid only uses main memory for buffering in-transit data • Disk buffering: used two instances of Squid on a CAG

36. Other Experiments & Analysis • Measured maximum forwarding rate • Stress test by using long flows: 460 ± 4.75 Mb/s • Measured user-perceived throughput • Throughput improvement when circuits replace congested Internet paths. • Related the internetworking architecture with the TCP/IP & OSI reference models • Fits into the OSI model

37. Conclusions • File transfers on circuit-switched (CS) networks • Advantage relative to packet switching: predictable service time • Packet switching (PS) better for small file transfers • Call setup delay >> Transfer time (link rates ↑, transfer time ↓) • Predictability not a concern when absolute delays are low • Hence hybrid architecture: PS for small; CS for large • Call admission control algorithms designed to be fair across small-file, large-file & across short-path, long-path Internet path metrics Circuit network operation

38. Conclusions contd. • Designed a gateway called CAG to interconnect connectionless networks with circuit networks • CAG implements all sub-layers of the network layer with data-plane and control-plane (admission control) • CAG supports reliable file transfers • File transfers need high-speed links on whole path • Gradually evolving circuit-switched networks for access (current bottleneck) will lead to improved performance

39. Future Work • More sophisticated bandwidth-sharing schemes • Currently studied a complete partitioning scheme • To avoid sensitivity to network management system performance as is the case with partitioning • Hardware-based implementation of CAG with the support of disk buffering for in-transit data • Current software implementation could slow down effective transmission rates

40. Questions from Form G111

41. Thank you! Questions?

42. Questions from Form G111 - Defining the problem • Has the student stated the problem clearly, provided its motivation, and the requirements for a solution? • In the context of new optical circuit-switched technologies and new application requirements, how do we support file transfers efficiently on a dynamically shared circuit-switched network and how can we interconnect a circuit network with a connectionless network?

43. Questions from Form G111 -Analysis of previous and related work • Theoretical component: file transfers on circuit networks • Packet switching is considered better • But circuit switching provides rate guarantees • Very large file transfers on optical connection-oriented testbeds • e.g.: ESnet4, NSF DRAGON, CA*net4, UKLight, JGN2, etc. • Focus: implementation & inter-domain usage • Our work: how much bandwidth to allocate per file transfer • File transfers have not been considered on other circuit/virtual-circuit networks • e.g.: telephone networks, ATM • Experimental component: interconnect circuit networks with connectionless networks • State-of-the-art: IP routers • Original purpose: interconnect connectionless networks • Used leased line modes to include circuit networks • Proposed but not deployed: IP switching & TCP switching • Our work: gateway that handles service-type mismatch between connectionless and circuit networks

44. Questions from Form G111 -Success criteria • Has the student adequately defined the measure(s) of success to be used to evaluate the work? Is there a well defined metric with a goal? Does the metric adequately represent the desired success criteria? • Success criteria • Theoretical work: use a hybrid architecture for file transfers • Call blocking circuit network: optimal design parameters to maximize mean delay reduction • Call queueing circuit network: optimal design parameters to minimize mean response time at a given effective utilization • Experimental work: designed an internetworking gateway called CAG • CAG supports reliable file transfers • Improved circuit utilization • Measured maximum forwarding rate of CAG • Metrics • Theoretical work: file-transfer delay, utilization, mean delay reduction, fairness ratio • Experimental work: reliable file transfer, circuit utilization, forwarding rate, user-perceived throughput

45. Questions from Form G111 - Solution • Is the approach taken well executed? Does it appear to be correct? Is the work technically challenging? Does the student utilize appropriate professional standards? • A combination of analytical, simulation, and experimental methods • Call blocking circuit network for file transfers • Analytical model • Call queueing circuit network for file transfers • Analytical model • Simulation model • An internetworking gateway • Software implementation • Experimentation and measurements • Architecture positioning

46. Questions from Form G111 - Innovation and risk • To what extent is the work innovative? Has the student taken a risk in applying the chosen approach? • Bandwidth sharing problem on using circuit networks for file transfers has not been studied before • The problem of internetworking connectionless networks and dynamically shared circuit networks has not been addressed widely (only one previous solution – from the 90s which proved unviable)

47. Questions from Form G111 -Broader implications • Has the student considered the broader implications of the work? Broader implication may include social, economic, political, technical, ethical, business, etc. • Enable the deployment of high-speed circuit networks at low costs (sharing) to provide predictable-delay services • New applications can be created with this type of service • Integrated with Internet • Avoids need for desert-start deployment

48. Background – High-Speed Circuit-Switching • Data-plane technologies • Switching: Time Division Multiplexing (TDM) & Wavelength Division Multiplexing (WDM) • Mapping: to carry Ethernet frames via SONET signals or WDM lightpaths • Control plane: Generalized MultiProtocol Label Switched (GMPLS) • Three components: signaling, routing, & management • Bandwidth sharing mode: immediate-request (IR) • Equipment examples: • SONET switches: Sycamore SN16000 • WDM switches: Adva/Movaz RayExpress OADM

49. Layers in OSI reference model AL: Application Layer TL: Transport Layer DLL or L: Data-Link Layer PHY or P: Physical Layer • Sublayers of network layer (NL) • SNICF: Subnetwork Independent Convergence Function • SNDCF: Subnetwork Dependent Convergence Function • SNACF: Subnetwork Access Function • SNSF: Subnetwork Switching Function [ITU X.200] http://www.itu.int/rec/T-REC-X.200-199407-I/en [Callon83]R. E. Callon, "Internetwork protocol,“ Proc of the IEEE, vol. 71, no. 12, pp. 1388-1393, Dec. 1983

50. Layers in the Internetworking Architecture This internetworking architecture fits into OSI reference model