Grids and Advanced Networking Tokyo, 14 May 2003

1. Grids and Advanced Networking Tokyo, 14 May 2003 An Exposed Approach to Reliable Multicast in Heterogeneous Logistical NetworksMicah Beck, Assoc. Prof. & DirectorLogistical Computing & Internetworking (LoCI) Lab

2. Authors Micah Beck Ying Ding Erika Fuentes Sharmila Kancherla LoCI Lab James S. Plank Terry Moore Alex Bassi Yong Zheng Hunter Hagewood PlanetLab Credits

3. Funding Dept. of Energy SciDAC National Science Foundation ANIR UT Center for Info Technology Research University of Tennessee Micah Beck James S. Plank Jack Dongarra University of California, Santa Barbara Rich Wolski Logistical Networking Research at UTK

4. What is Logistical Networking? • A scalable mechanism for deploying shared storage resources throughout the network • A general store-and-forward overlay networking infrastructure • A way to break transfers into segments and employ heterogeneous network technologies on the pieces

5. Why “Logistical Networking” • Analogy to logistics in distribution of industrial and military personnel & materiel • Fast highways alone are not enough • Goods are also stored in warehouses for transfer or local distribution • Fast networks alone are not enough • Data must be stored in buffers/files for transfer or local distribution

6. The Network Storage Stack Applications • Our adaptation of the network stack architecture for storage • Like the IP Stack • Each level encapsulates details from the lower levels, while still exposing details to higher levels Logistical File System Logistical Tools L-Bone exNode IBP Local Access Physical

7. IBP: The Internet Backplane Protocol • Storage provisioned on community “depots” • Very primitive service (similar to block service, but more sharable) • Goal is to be a common platform (exposed) • Also part of end-to-end design • Best effort service – no heroic measures • Availability, reliability, security, performance • Allocations are time-limited! • Leases are respected, can be renewed • Permanent storage is to strong to share!

8. Data Movers • Module implementing standard point-to-multipoint transfer between IBP allocations • Uniform API allows independence from the underlying data transfer protocol • Not every DM can apply to every transfer • Caller responsible for determining validity • Current options: Multi-TCP, Multi-UDP (reliable), UDP Multicast (unreliable)

9. mcopy operation • Encapsulates shared buffering, management of multiple low level transfers Memory Receiving Depots Sending Depot 1. Buffering 2. Parallel Transfers File System

10. Heterogeneity in mcopy • TCP connections • Unreliable UDP multicast • Reliable UDP with flow control, retransmit • Reliable UDP with TCP control channel • SABUL (R. Grossman, University of Chicago) • Reliability must be end-to-end!

11. Comparison of Sending Rates in the LAN

12. Heterogeneous Multicast

13. End-to-End Reliability through Retransmission IBP depots 2. IBP mcasts 1. IBP upload 3. IBP download 5. TCP retransmission source 4. TCP control |channel destination

14. Other Approaches to Reliable Multicast • Retransmission in orginal group • Multiple groups for retransmission assigned dynamically to sets of missed blocks • Retransmission from intermediate nodes • Application-dependent approaches • Video doesn’t need perfect reliability • Time deadlines alter retransmission priorities

15. Exposed Approach to Multicast • Many important elements are under the control of an endpoint (the source) • Topology of multicast tree • Choice of mcast operation types • Handling of intermediate errors • Performance optimization • Global & app-specific strategies possible

16. Limitations of Exposed Approach • Scalability problems • Control from one end-point is limiting • Not sufficient for public media distribution • A distributed control infrastructure is required • Active routers provide a natural platform • Tamanoir project of ENS-Lyon may provide a testbed for this architecture • Laurent Lefevre, Jean-Patrick Gelas

17. Topology and Performance • Choosing tree nodes (can we detect underlying Layer 2 topology?) • Where is UDP multicast enabled? • Where is are UDP flooding protocols legal? • Evaluating reliability, performance of component mcasts • Trading off scalability for reliability and performance

18. Experiment:Three Approaches • 10 recievers • Direct Unicast TCP to all nodes • Pure TCP overlay multicast • TCP Data Mover used at every tree node • Mixed TCP/UDP multicast • TCP Data Mover used in backbone • UDP multicast in edge networks • Caveat: Measurements are not end-to-end!

19. Direct Unicast TCP A S D 1 6 2 5 B C 4 3

20. Pure TCP Overlay Multicast A S D 1 6 2 5 B C 4 3

21. Mixed TCP in Backbone/UDP Mcast at Edge

22. Experimental Results Direct TCP vs Overlay • 10 simultaneous TCP streams/connection • 50 MB transfers • Sending rate (not scaled by recievers) • Direct TCP Unicast 3.4 Mb/s • Pure TCP Overlay Multicast 5.1Mb/s • Speedup obtained: 50%

23. Experimental Results Overlay TCP vs Mixed • 10 recievers • No rate control on UDP Multicast, can’t run multiple streams • Comparing Overlay TCP with single TCP stream/connection to Mixed, there is a 15% speedup • UDP at edge offers some speedup over TCP

24. Conclusions • Logistical Networking implements a scalable overlay networking infrastructure • Data Movers provides support heterogeneity even within a single transfer • Exposed & heterogeneous multicast can achieve speedups in the WAN • Defining the tree and managing it for reliability and performance is a challenge

25. L-Bone: January 2003 Current Storage Capacity: 13 TB

26. http://loci.cs.utk.edu Micah Beckmbeck@cs.utk.edu