PVM and MPI What is more preferable?

1. PVM and MPIWhat is more preferable? Comparative analysis of PVM and MPI for the development of physical applications on parallel clusters Ekaterina Elts Scientific adviser: Assoc. Prof. A.V. Komolkin

2. Introduction • Computational Grand Challenge problems • Parallel processing – the method of having many small tasks to solve one large problem • Two major trends : • MPPs - (massively parallel processors) – but cost $$$ ! • distributed computing

3. Introduction • The hottest trend today is PC clusters running Linux • Many Universities and companies can afford 16 to 100 nodes. • PVM and MPI are the most used tools for parallel programming

4. Contents • Parallel Programming • A Parallel Machine Model Cluster • A Parallel Programming Model Message Passing Programming Paradigm • PVM and MPI • Background • Definition • A Comparison of Features • Conclusion

5. A Sequential Machine Model A central processing unit (CPU) executes a program that performs a sequence of read and write operations on an attached memory The von Neumann computer SISD – Single InstructionStream – Single DataStream

6. A Parallel Machine Model Interconnect … The cluster A node can communicate with other nodes by sending and receiving messages over an interconnection network The von Neumann computer MIMD – Multiple InstructionStream – Multiple DataStream

7. A Parallel Programming Model input input output output Sequential (serial) algorithm Parallel algorithm

8. Example: scalar product of vectors input input do i=1,N S=s+aibi enddo do i=1,N/2 s1=s1+aibi enddo do i=N/2+1,N s2=s2+aibi enddo output S=s1+s2 output print S print S Sequential (serial) algorithm Parallel algorithm

9. A Parallel Programming Model • Message Passing 4 5 1 3 2 0 detailed picture of a single task Many small tasks solve one large problem Instantaneous state of computation

11. Message Passing Paradigm • Each processor in a message passing program runs a separate process (sub-program, task) • written in a conventional sequential language • all variables are private • communicate via special subroutine calls

12. Messages • Messages are packets of data moving between processes • The message passing system has to be told the following information: • Sending process • Source location • Data type • Data length • Receiving process(es) • Destination location • Destination size

13. SPMD Single Program Multiple Data Same program runs everywhere Each process only knows and operates on a small part of data MPMD Multiple Program Multiple Data Each process perform a different function (input, problem setup, solution, output, display) Message Passing

14. What is Master/Slave principle? • The master has the control over the running application, it controls all data and it calls the slaves to do there work PROGRAM IF (process = master) THEN master-code ELSE slave-code ENDIF END

15. Simple Example SPMD&Master/Slave ForifromrankstepsizetoNdo s=s+aibi enddo a1b1+a1+sizeb1+size+a1+2*sizeb1+2*size+… slave master S=s1+s2 slave

16. PVM The development of PVM started in summer 1989 at Oak Ridge National Laboratory (ORNL). PVM was effort of a single research group, allowing it great flexibility in design of this system MPI The development of MPI started in April 1992. MPI was designed by the MPI Forum (a diverse collection of implementors, library writers, and end users) quite independently of any specific implementation PVMandMPIBackground MPI-1 MPI-2 PVM-1 PVM-2 PVM-3 PVM-3.4 1989 90 94 96 97 99 2000

17. PVM A distributed operating system Portability Heterogeneity Handling communication failures MPI A library for writing application program, not a distributed operating system portability High Performance Heterogeneity Well-defined behavior PVMandMPIGoals Note:implementation ≠ specification! MPI implementations: LAM, MPICH,…

18. What is MPI ? MPI - Message Passing Interface • A fixed set of processes is created at program initialization, one process is created per processor mpirun –np 5 program • Each process knows its personal number (rank) • Each process knows number of all processes • Each process can communicate with other processes • Process can’t create new processes (in MPI-1)

19. What is PVM ? PVM - Parallel Virtual Machine • Is a software package that allows a heterogeneous collection of workstations (host pool) to function as a single high performance parallel machine (virtual) • PVM, through its virtual machine provides a simple yet useful distributed operating system • It has daemon running on all computers making up the virtual machine

20. PVM Daemon (pvmd) • UNIX process which oversees the operation of user processes within a PVM application and coordinates inter-machine PVM communications • The pvmd serves as a message router and controller • One pvmd runs on each host of a virtual machine • The first pvmd (started by hand) is designated the master, while the others (started by the master) are called slaves • Only the master can start new slaves and add them to configuration or delete slave hosts from the machine

21. Executing PVM system routines Executing user computation master

22. What is Not Different? • Portability– source code written for one architecture can be copied to a second architecture, compiled and executed without modification (to some extent) • Support MPMD programs as well as SPMD • Interoperability – the ability of differentimplementations of the same specification to exchange messages • Heterogeneity (to some extent) PVM & MPI are systems designed to provide users with libraries for writing portable, heterogeneous, MPMD programs

23. Heterogeneity • Architecture • Data format • Computational speed • Machine load • Network load static dynamic

24. Heterogeneity: MPI • Different datatypes can be encapsulated in a single derived type, thereby allowing communication of heterogeneous messages. In addition, data can be sent from one architecture to another with data conversion in heterogeneous networks (big-endian, little-endian).

25. Heterogeneity: PVM • The PVM system supports heterogeneity in terms of machines, networks, and applications. With regard to message passing, PVM permits messages containing more than one datatype to be exchanged between machines having different data representations.

26. Process control • Ability to start and stop tasks, to find out which tasks are running, and possibly where they are running. • PVM contains all of these capabilities – it can spawn/kill tasks dynamically • MPI -1 has no defined method to start new task. MPI -2 contain functions to start a group of tasks and to send a kill signal to a group of tasks

27. Resource Control • PVM is inherently dynamic in nature, and it has a rich set of resource control functions. Hosts can be added or deleted • load balancing • task migration • fault tolerance • efficiency • MPI is specifically designed to be static in nature to improve performance

28. Virtual topology - only forMPI • Convenient process naming • Naming scheme to fit the communication pattern • Simplifies writing of code • Can allow MPI to optimize communications

29. Virtual topology example A virtual topology of twelve processes - grid with a cyclic boundary condition in one direction e.g. processes 0 and 9 are ``connected''. The numbers represent the rank and the conceptual coordinates mapped to the ranks

30. Message Passing operations • MPI : Rich message support • PVM: Simple message passing

31. Point-to-Point communications An asynchronous communication completes as soon as the message is on its way A synchronous communication does not complete until the message has been received.

32. Non-blocking operations Non blocking communication allows useful work to be performed while waiting for the communication to complete

33. Collective communications Broadcast A broadcast sends a message to a number of recipients Reduction operations Reduction operations reduce data from a number of processors to a single item. Barrier A barrier operation synchronises a number of processors.

34. Fault Tolerance: MPI • MPI standard is based on a static model • If a member of a group failed for some reason, the specification mandated that rather than continuing which would lead to unknown results in a doomed application, the group is invalidated and the application halted in a clean manner. • In simple if something fails, everything does.

35. Fault Tolerance: MPI

36. Fault Tolerance: MPI Failed Node There is a failure and…

37. Fault Tolerance: MPI Failed Node … the application is shut down

38. Fault Tolerance: PVM • PVM supports a basic fault notification scheme: it doesn’t automatically recover an application after a crash, but it does provide notification primitives to allow fault-tolerant applications to be built • The Virtual Machine is dynamically reconfigurable • A pvmd can recover from the loss of any foreign pvmd except the master. The master must never crash

39. Fault Tolerance: PVM Virtual Machine

40. Fault Tolerance: PVM Virtual Machine Failed Node

41. Fault Tolerance: PVM Virtual Machine Fast host delete or recovery from fault

42. PVM Virtual machine concept Simple message passing Communication topology unspecified Interoperate across host architecture boundaries Portability over performance Resource and process control Robust fault tolerance MPI No such abstraction Rich message support Support logical communication topologies Some realizations do not interoperate across architectural boundaries Performance over flexibility Primarily concerned with messaging More susceptible to faults Conclusion Each API has it’s unique strengths

43. PVM is better for: Heterogeneous cluster, resource and process control The size of cluster and the time of program’s execution are great MPI is better for: Supercomputers (PVM is not supported) Application for MPP Max performance Application needs rich message support Conclusion Each API has it’s unique strengths

44. Acknowledgments • Scientific adviser Assoc. Prof. A.V.Komolkin

45. ??? Thank you for your attention! Questions?