High Performance Cluster Computing Architectures and Systems

1. High Performance Cluster ComputingArchitectures and Systems Hai Jin Internet and Cluster Computing Center

2. Multiple Path Communication • Introduction • Heterogeneity in Networks and Applications • Multiple Path Communication • Case Study • Summary and Conclusion

3. Introduction (1) • Communication is major factor of performance in network-based computing • Utilize all available network resources within a system • Substantial benefits in simultaneously exploiting multiple independent communication paths between processors • Heterogeneous network environment • Single application execution • Different network - different performance characteristic

4. Introduction (2) • Performance-based path determination (PBPD) • Exploiting multiple physical communication paths & multiple communication protocols within a single parallel application program to reduce communication overhead • Different physical networks & protocols have different performance characteristics • Different types of communication within an application might be best suited to one of several different communication paths • bulk data transfer (image data): – high-bandwidth communication • short control messages (synch, acknowledgement): - low-latency communication

5. Introduction (3) • Performance-based path determination (PBPD) • Performance-based path selection (PBPS) • Dynamically select the best communication path among several in a given system • Performance-based path aggregation (PBPA) • Aggregate multiple communication paths into a single virtual path for sending an individual message in an application • Each independent path simultaneously carries a fraction of a single message • Useful in bandwidth-limited situations • Control is based on message size, message type, or network traffic load

6. Heterogeneity in Network and Applications • Heterogeneity inherent interprocessor communication • Different types of messages • latency-limited • bandwidth-limited

7. Varieties of Communication Networks • WAN (1 Mbps) • MAN (100 Mbps) • SAN (more than 1 Gbps) • Ethernet (10 Mbps – 1000 Mbps) • ATM (155 Mbps) • FDDI (100 Mbps) • HiPPI (800 Mbps – 1.6 Gbps) • Infinitband (10Gbps)

8. Exploiting Multiple Communication Paths • Messages • size • communication groups • priority level • Different type of network services to support different types of message communication • Network services • connection oriented • connectionless • Both are logically sufficient for the implementation of any communication pattern • Each offers performance & programming benefits for some classes of applications

9. Benefits to Support Multiple Communication Paths • Efficient network utilization • Most appropriate communication paths to be used • Alternative network protocols • Connection oriented / connectionless • Robust communication • Multiple networks provides extra reliability • Network load balancing • Quality of service • Multiple paths can support diverse QoS requirements

10. Multiple Path Communication • A single application is likely to require several different types of messages for communication • Different types of messages may be better suited to a different type of communication mechanism

11. Performance-Based Path Selection (PBPS) • Useful when one provides better performance in one situation while the other is better in another situation • Possible to dynamically select the appropriate communication path for a given communication event f1(m1) = t1 f2(m2) = t2 fPBPS(m) = Best[fi(m)], where (i= 1.. N)

12. When sending a message of size mi, performance-based path selection (PBPS) uses the lower latency curve among f1(mi) and f2(mi)

13. Performance-Based Path Aggregation (PBPA) • Can be applied when different paths show similar characteristics • 2 nearly identical networks • aggregate 2 networks into a single virtual network • bandwidth will be nearly twice • Divide – transmit - aggregate • Important consideration • determine the size of the submessages fPBPA(m) = fi(mi), where (i= 1.. N), (m =  mi)

14. When using performance-based path aggregation (PBPA) with two networks, a message of size m1+ m2 is split into two submessages such that messages of size m1 and m2 are sent over networks f1(mi) and f2(mi) simultaneously

15. PBPD Library • Custom library whose main feature is the support of multiple communication paths in a single application program • Based on common TCP layer • Add integer field ‘length’ tells message size • Used by PBPA if a message is too small to segment • Handle the multiplexing of different TCP connections • Use UNIX select system call with appropriate table lookups

16. Implementation and Protocol Hierarchy of the PBPD Communication Routines

17. Case Study: Multiple Path Characteristic • Communication type in parallel application program • point-to-point, collective • 4 Silicon Graphics Challenge L shared-memory multiprocessors. • 4 or 8 R10000 processors at 196MHz per node • 10Mbps Ethernet and 266Mbps Fiber Channel

18. Multiple Heterogeneous Network Configuration used in the Experiments

19. Point-to-point and Broadcast Characteristics of Ethernet using the TCP and UDP Communication Protocols

20. Example of a Broadcast Operation of a Separate Addressing Method using the PBPA Technique

21. Point-to-point Characteristics of Ethernet and Fibre Channel using the TCP Protocol

22. Broadcast Characteristics of Ethernet and Fibre Channel using the TCP Protocol

23. Case Study: Communication Patterns of Parallel Applications • Performance of PBPD at application-level depends on the communication patterns of the specific application being executed

24. Parallel Benchmark Programs Tested

25. Case Study: Computation Model • Data parallelization • Medium to coarse-grained parallelism • Similar communication pattern for every node • except when starting application • Each processor tends to alternate computation and communication at the same time • communication congestion is inevitable

26. Relative Times of Communication Events for the IS Benchmark

27. Relative Times of Communication Events for the MG Benchmark

28. Case Study: Message Size and Destination Distributions • Distribution of message destinations • Uniformly distributed among all nodes • Biased to some destinations for each node • Distribution of message sizes

29. Overall Message Destination Distribution for All of the Test Programs

30. The Message Destination Distribution for Each Processor in the CG Benchmark

31. The Cumulative Distribution of Message Sizes in the Test Programs

32. Experiments Results • Parameterize communication pattern • Point-to-point communication • Small message mean size l1, probability to appear is b • Large message mean size l2, probability to appear is 1 - (b+c) • Broadcast communication • Message mean size l3, probability to appear is c • Assumption • 1 master, p-1 slave processors • Message size for each communication follow Poison distribution with 3 different mean value l1, l2, l3

33. Application type pt2pt (small) mean l1 pt2pt (large) mean l2 broadcast mean l3 b 1-(b+c) c A 45% 45% 10% B 25% 25% 50% C 5% 5% 90% Parameter Values Used in the Synthetic Benchmark

34. Speedup using PBPS with the TCP and UDP Protocols over Ethernet. The speedups are normalized to the case when using the TCP protocol alone

35. Speedup using PBPA Technique with Ethernet and Fiber Channel using the TCP Protocols. The speedups are normalized to the case when using the Ethernet alone

36. Summary and Conclusion • Communication overhead can be reduced by exploiting heterogeneity in both communication path and application • Reduce communication overhead • PBPD technique can achieve performance improvement based on message type