The Kangaroo approach to Data movement on the Grid

1. The Kangaroo approach to Data movement on the Grid Rajesh Rajamani (raj@cs.wisc.edu) June 03, 2002

2. Condor and the Grid • What is the Grid? • Grid applications tend to have demanding I/O needs. Process/produce large amounts of data • Need for an efficient mechanism to transfer data

3. Motivation • Suppose you want to transfer 10 large (10s of GB) and 100 small files (in KB) from UW to Vietnam - • Would you use FTP, if you know that - • Network connection to Vietnam is minimal at the best of times ? • Would you be satisfied with the reliability and performance of FTP?

4. File System Motivation • Shortcomings of FTP - 1. Performance suffers in case of transient errors Fast, but unreliable link slow link D App R R R Router R Destination D Data

5. Motivation • TCP checksum can fail to detect errors for large xfers • TCP uses 16-bit checksum, which is one’s complement of the one’s complement sum of the TCP segment (RFC1071) 0110011001100110 0101010101010101 0000111100001111 ------------------------ 1100101011001010 0011010100110101

6. TCP checksum • All burst errors of 16 or fewer bytes will be detected except when 0x0000 is replaced by 0xFFFF and vice versa • Other errors are detected only probabilistically 1110011001100110 0101010101010101 Original checksum = 1000111100001111 0011010100110101 ------------------------ 1100101011001011 (one’s complement sum) 0011010100110100 (checksum)

7. Checksum errors 0110011001100110 0011010101010101 Original checksum = 0010111100001111 0011010100110101 ------------------------ 1100101011001010(one’s complement sum) 0011010100110101 (checksum) • Error goes undetected!!!

8. Motivation (contd.) 3. Doesn’t support persistent queues 4. Opens a connection for every transfer (which in most cases is a single file)

9. 0 0 1 1 . . . . Calculate (n-k) FEC blocks . . . . . . . . N -1 K -1 K N K File to be transferred N >> K FEC Block Motivation (contd.) • No support to efficiently broadcast/multicast files

10. App File System File System File System File System The Vision K =Kangaroo server K K K Data Movement System K K K K

11. Outline • Motivation • Architecture • Replication • Conclusions

12. Architecture • Message based architecture • Layers • Application • Adaptation • Consistency • Transport

13. Blocking procedure call Non-blocking message File System Architecture Application open, read, write, close, fsync Adaptation get, put, push, commit open, read, write, close, fsync Consistency Consistency get, put, commit, route get, put, push,commit ack ack Kangaroo Transport Kangaroo Transport Kangaroo Transport TCP/IP TCP/IP TCP/IP

14. Blocking procedure call Non-blocking message Adaptation layer Application open, read, write, close, fsync Adaptation get, put, push, commit Consistency

15. Interposition agents Application Application Std Library Agent Kernel Std Library Kernel

16. Bypass and PFS • Bypass - General-purpose tool for trapping and redirecting standard library procedures. • PFS – Pluggable File System • PFS traps all I/O operations. It converts file-related operations to Kangaroo operations. • Can be applied at run-time to any dynamically-linked program: • setenv LD_PRELOAD ./pfs_agent.so • vi /kangaroo/antipholus.cs.wisc.edu/u/r/a/raj/.cshrc • gcc /http/www.cs.wisc.edu/~raj/example.c -o /kangaroo/antipholus.cs.wisc.edu/u/r/a/raj/temp/example

17. Bypass Mechanism • A static call is bound to a function definition when the library is linked. Dynamic calls are bound at run time by the loader. • The preload/bypass mechanism is a hook only into the dynamic loader

18. Multiple Bypass Application Global DS Entry points open _open __open read write layer Switch opendir readdir socket accept connect Interposition agent printf scanf open _open __open read write Standard Library

19. printf() calls write() Application Global DS Entry points write open _open __open read layer Switch opendir readdir socket accept connect Interposition agent printf scanf open _open __open read write Standard Library

20. Blocking procedure call Non-blocking message File System Architecture Application open, read, write, close, fsync Adaptation get, put, push, commit open, read, write, close, fsync Consistency Consistency get, put, commit, route get, put, push,commit ack ack Kangaroo Transport Kangaroo Transport Kangaroo Transport TCP/IP TCP/IP TCP/IP

21. Blocking procedure call Non-blocking message File System Consistency layer Adaptation get, put, push, commit open, read, write, close, fsync Consistency Consistency get, put, commit, route get, put, push,commit ack ack Kangaroo Transport Kangaroo Transport

22. Kangaroo message format Source IP Destination IP File name Operation – put, get, push Data/Offset

23. Consistency layer • Interface • Get block, put block, commit blocks, push blocks • Semantics • Ordering - Order preserving • Reliability - Detects success • Duplication - Delivers at most once

24. Ordering and Duplication • For most applications, order of writes doesn’t matter. E.g. - writes to different offsets • Only renames and writes that overlap are non-idempotent. Majority of the operations are idempotent • Consistency layer deals with non-idempotent operations by using kangaroo_push

25. Blocking procedure call Non-blocking message File System Architecture Application open, read, write, close, fsync Adaptation get, put, push, commit open, read, write, close, fsync Consistency Consistency get, put, commit, route get, put, push,commit ack ack Kangaroo Transport Kangaroo Transport Kangaroo Transport TCP/IP TCP/IP TCP/IP

26. Transport layer Consistency Consistency get, put, commit, route get, put, push,commit ack ack Kangaroo Transport Kangaroo Transport Kangaroo Transport TCP/IP TCP/IP TCP/IP

27. Transport layer • Interface • Send message, query route, query status • Semantics • Ordering - None (or worse!) • Reliability - Likely, but not guaranteed. • Duplication - Unlikely, but possible. • Performance • Uses all available resources (net, memory, disk) to maximize throughput • Replication necessary to provide reliability • Must cache blocks until delivery is acknowledged • Replication may introduce inconsistencies

28. Cedar – Bandwidth regulator • Standard socket abstraction. • Enforces limits on how much bandwidth can be consumed across multiple times scales. • Also measures congestion and reports to locally-determined manager. • Example: • If conditions are good, do not exceed 10Mb/s. • If there is competition for the link, fall back to no more than 1Mb/s.

29. Replication • Endpoints keep copies of data blocks until they are acknowledged. • Receiver sends cumulative acks and commits unbroken sequences. • Replication can be costly • Choices - • Replicate @ sender • Pass the buck - Change end-point

30. D D D sender S D Data Pass the Buck ACK • Delete the local copy after a one-hop ack. Requires atomic accept and sync. (Similar to email) K K K K S D

31. Conclusions • Large scale file transfer is not a solved problem. Many interesting research issues • Kangaroo provides an efficient mechanism to transfer large datasets over the WAN. It does so by using all available resources (net, memory, disk) to maximize throughput • Kangaroo is a data movement system, an overlay network, a distributed buffer system

32. Kangaroo • Uses three basic techniques to address some issues • Uses a virtual circuit to transfer data • Buffering on intermediate nodes improves performance • Retries till it succeeds (when it encounters an error).

33. As side effects • Kangaroo • Allows applications to hide latency by overlapping I/O and CPU • Maximizes use of resources However, • No support for inter-process consistency • Does NOT provide low-latency single file transfers • Does not use checksums, but will do so in the near future • No support for multicasting to multiple clients – AS YET

34. App File System File System File System File System Future - Multiroute and Multicast K K K K K K K

35. References • D. Thain, J. Basney, S.-C. Son, and M. Livny, "The Kangaroo Approach to Data Movement on the Grid", Tenth IEEE Symposium on High Performance Distributed Computing (HPDC10), San Francisco, California, August 7-9, 2001 • J.Stone and Craig Partridge, “When The CRC and TCP Checksum Disagree, SIGCOMM 2000 • Gemmell, Jim, Schooler, Eve, and Gray, Jim, Fcast Multicast File Distribution, IEEE Network, Vol. 14, No. 1, Jan 2001