Bulk-Data Metanet: Virtualization by Example

1. Bulk-Data Metanet: Virtualization by Example • Sergey Gorinsky • Applied Research Laboratory • Department of Computer Science and Engineering • Washington University in St. Louis • St. Louis, MO 63130-4899, USA • 2006/11/8, NSF FIND kickoff meeting

2. Overview • Bulk-Data Transfers • Bulk-Data Metanet as Part of the Overall Architecture • Inefficacies of Internet Services for Bulk Data • Potential of Transfer Scheduling • Fruits of Virtualization • Research Agenda Sergey Gorinsky (Washington University in St. Louis), “Bulk-Data Metanet: Virtualization by Example ”, 2006/11/8 2

3. Bulk-Data Transfers • Definition • transfer time is much longer than round-trip time (RTT) • deliberately an imprecise definition • Performance metric • transfer time • and not, e.g., throughput during any smaller time interval • Sample applications • large-scale science • e.g., astronomical data sets • software download • e.g., operating systems Sergey Gorinsky (Washington University in St. Louis), “Bulk-Data Metanet: Virtualization by Example ”, 2006/11/8 3

4. User User User User User User Another end-to-end metanet Substrate Optical flow switching network GMPLS network Another physical network Bulk-Data Metanet as Part of the Overall Architecture Bulk-Data Metanet Sergey Gorinsky (Washington University in St. Louis), “Bulk-Data Metanet: Virtualization by Example ”, 2006/11/8 4

5. Inefficacies of Internet Services for Bulk Data • In realizing the intended ideal of fair efficient allocation • TCP discrimination against flows with long RTTs • slow convergence of TCP throughput to fair efficient rates • solutions proposed for the above • numerous congestion control proposals of various efficiency and degree of network support • In pursuing a wrong ideal • instantaneously fair allocations do not minimize transfer times • prioritized service can improve the average transfer time • e.g., Shortest Job First scheduling of CPU • danger: starvation of some (long) transfers Sergey Gorinsky (Washington University in St. Louis), “Bulk-Data Metanet: Virtualization by Example ”, 2006/11/8 5

6. Search for an Ideal Allocation for Bulk-Data Transfers • Farsighted Internet congestion control (Infocom 2005 paper by Key, Massoulie, and Vojnovic) • transmits nothing when congestion is heavy • transmits more than TCP when congestion is light • can starve large transfers under persistent congestion • provides no benefits within the class of bulk-data transfers • Isolated specialized networks (UltraScience Net, CHEETAH, OSCARS, DRAGON) • schedule transfers based on message sizes and topology • rely on GMPLS to establish dedicated end-to-end channels • are technologically limited to transfers times of minutes or more • have limited reach and high cost Sergey Gorinsky (Washington University in St. Louis), “Bulk-Data Metanet: Virtualization by Example ”, 2006/11/8 6

7. Potential of Transfer Scheduling: Simulation View • Constraints • no transfer finishes later than with maxmin-fair rates • network has a single bottleneck • Virtual Finish Time First (ViFi) algorithm • transfer messages one at a time with preemption in the order of their maxmin-fair finish times • Simulation settings • 3000 messages on 10 Tbps path • Poisson arrivals with the average rate of 1 message per second • uniformly distributed message weights from set {1, 2, 3, 4} • Pareto-distributed message sizes with Pareto index 1.5 and minimum size 500 GB Sergey Gorinsky (Washington University in St. Louis), “Bulk-Data Metanet: Virtualization by Example ”, 2006/11/8 7

8. Number of Pending Messages (one experiment) Sergey Gorinsky (Washington University in St. Louis), “Bulk-Data Metanet: Virtualization by Example ”, 2006/11/8 8

9. Distributions of Transfer Times (10000 experiments) Sergey Gorinsky (Washington University in St. Louis), “Bulk-Data Metanet: Virtualization by Example ”, 2006/11/8 9

10. Fruits of Virtualization • Better services and lower costs for users • specialized service improving transfer times • lower costs than in isolated specialized networks • New sources of revenue for substrate providers • ability to sell more advanced technology (e.g., optical flow switching) to metanet providers at a higher price despite limited deployment • Lower costs and new revenues for metanet providers • dynamic lease and release of physical infrastructures • access to various types and sets of end-to-end resources (e.g., both GMPLS and optical flow switching) • ability to attract new types of users (e.g., software downloaders) Sergey Gorinsky (Washington University in St. Louis), “Bulk-Data Metanet: Virtualization by Example ”, 2006/11/8 10

11. Research Agenda • Algorithms for optimal transfer scheduling • hard problem in general topologies • efficient implementations for online operation • Security and robustness • human factor • enforcing the resource allocation schedule • Efficiency coordination and timing • distributed activation and renegotiation of the allocation schedule • new algorithms for transmission control at hosts and routers • Interface between the metanet and physical substrate • link capacities, processing power, reconfiguration time, buffer space, geographical location? Sergey Gorinsky (Washington University in St. Louis), “Bulk-Data Metanet: Virtualization by Example ”, 2006/11/8 11

12. Additional Slides Sergey Gorinsky (Washington University in St. Louis), “Bulk-Data Metanet: Virtualization by Example ”, 2006/11/8 12

13. Scheduling versus Maxmin-Fair Sharing: An Example Arrival time 11, size 18, weight 1 Arrival time 0, size 72, weight 2 Arrival time 1, size 72, weight 3 Arrival time 26, size 18, weight 4 Maxmin- fair Capacity = 6 Time 0 1 11 13 16 23 26 27 29 30 ViFi Capacity = 6 Sergey Gorinsky (Washington University in St. Louis), “Bulk-Data Metanet: Virtualization by Example ”, 2006/11/8 13