Parallel Computing

1. Parallel Computing • A task is broken down into tasks, performed by separate workers or processes • Processes interact by exchanging information • What do we basically need? • The ability to start the tasks • A way for them to communicate

2. Communication • Cooperative • All parties agree to transfer data • Message passing is cooperative • Data must be explicitly sent and received • Any change in the receiver’s memory is made with the receiver’s participation • One-sided • One worker performs the transfer of data • Data can be accessed without waiting for another process

3. What is MPI? • A message passing library specification • Message-passing model • Not a compiler specification (i.e. not a language) • Not a specific product • Designed for parallel computers, clusters, and heterogeneous networks

4. The MPI Process • Development began in early 1992 • Open process/Broad participation • IBM,Intel, TMC, Meiko, Cray, Convex, Ncube • PVM, p4, Express, Linda, … • Laboratories, Universities, Government • Final version of draft in May 1994 • Public and vendor implementations are now widely available

5. Point to Point Communication • A message is sent from a sender to a receiver • There are several variations on how the sending of a message can interact with the program

6. Synchronous • A synchronous communication does not complete until the message has been received • A FAX or registered mail

7. Asynchronous • An asynchronous communication completes as soon as the message is on the way. • A post card or email

8. Blocking and Non-blocking • Blocking operations only return when the operation has been completed • Normal FAX machines • Non-blocking operations return right away and allow the program to do other work • Receiving a FAX

9. Collective Communications • Point-to-point communications involve pairs of processes. • Many message passing systems provide operations which allow larger numbers of processes to participate

10. Types of Collective Transfers • Barrier • Synchronizes processors • No data is exchanged but the barrier blocks until all processes have called the barrier routine • Broadcast (sometimes multicast) • A broadcast is a one-to-many communication • One processor sends one message to several destinations • Reduction • Often useful in a many-to-one communication

11. What’s in a Message? • An MPI message is an array of elements of a particular MPI datatype • All MPI messages are typed • The type of the contents must be specified in both the send and the receive

12. Basic C Datatypes in MPI

13. MPI Handles • MPI maintains internal data-structures which are referenced by the user through handles • Handles can be returned by and passed to MPI procedures • Handles can be copied by the usual assignment operation

14. MPI Errors • MPI routines return an int that can contain an error code • The default action on the detection of an error is to cause the parallel operation to abort • The default can be changed to return an error code

15. Initializing MPI • The first MPI routine called in any MPI program must be the initialization routine MPI_INIT • MPI_INIT is called once by every process, before any other MPI routines int mpi_Init( int *argc, char **argv );

16. Skeleton MPI Program #include <mpi.h> main( int argc, char** argv ) { MPI_Init( &argc, &argv ); /* main part of the program */ MPI_Finalize(); }

17. Communicators • A communicator handle defines which processes a particular command will apply to • All MPI communication calls take a communicator handle as a parameter, which is effectively the context in which the communication will take place • MPI_INIT defines a communicator called MPI_COMM_WORLD for each process that calls it

18. Communicators • Every communicator contains a group which is a list of processes • The processes are ordered and numbered consecutively from 0. • The number of each process is known as its rank • The rank identifies each process within the communicator • The group of MPI_COMM_WORLD is the set of all MPI processes

19. Point-to-point Communication • Always involves exactly two processes • The destination is identified by its rank within the communicator • There are four communication modes provided by MPI (these modes refer to sending not receiving) • Buffered • Synchronous • Standard • Ready

20. Standard Send • When using standard-mode send • It is up to MPI to decide whether outgoing messages will be buffered. • Completes once the message has been sent, which may or may not imply that the massage has arrived at its destination • Can be started whether or not a matching receive has been posted. It may complete before a matching receive is posted. • Has non-local completion semantics, since successful completion of the send operation may depend on the occurrence of a matching receive.

21. Standard Send MPI_Send( buf, count, datatype, dest, tag, comm ) Where • buf is the address of the data to be sent • count is the number of elements of the MPI datatype which buf contains • datatype is the MPI datatype • dest is the destination process for the message. This is specified by the rank of the destination within the group associated with the communicator comm • tag is a marker used by the sender to distinguish between different types of messages • comm is the communicator shared by the sender and the receiver

22. Synchronous Send • MPI_Ssend( buf, count, datatype, dest, tag, comm ) • can be started whether or not a matching receive was posted • will complete successfully only if a matching receive is posted, and the receive operation has started to receive the message sent by the synchronous send. • provides synchronous communication semantics: a communication does not complete at either end before both processes rendezvous at the communication. • has non-local completion semantics.

23. Buffered Send • A buffered-mode send • Can be started whether or not a matching receive has been posted. It may complete before a matching receive is posted. • Has local completion semantics: its completion does not depend on the occurrence of a matching receive. • In order to complete the operation, it may be necessary to buffer the outgoing message locally. For that purpose, buffer space is provided by the application.

24. Ready Mode Send • A ready-mode send • completes immediately • may be started only if the matching receive has already been posted. • has the same semantics as a standard-mode send. • saves on overhead by avoiding handshaking and buffering