1 / 29

Writing Message Passing Parallel-Programs with MPI

Writing Message Passing Parallel-Programs with MPI. Outline. Background of Parallel Computing Message Passing Interface (MPI) MPI Programs Point-to-point communication Collective communication Running MPI Programs & Homework #1. Parallel Computing (1). Sequential programming paradigm. M.

wells
Download Presentation

Writing Message Passing Parallel-Programs with MPI

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Writing Message Passing Parallel-Programs with MPI

  2. Outline • Background of Parallel Computing • Message Passing Interface (MPI) • MPI Programs • Point-to-point communication • Collective communication • Running MPI Programs & Homework #1

  3. Parallel Computing (1) • Sequential programming paradigm M Memory P Processor

  4. Parallel Computing (2) • Message passing parallel programming paradigm • several instances of the sequential paradigm are considered together • separate workers or processes • interact by exchanging information exchange information M M M Memory M … P P P Processor P workers communication network

  5. Communicating with other processes • Data must be exchanged with other workers • cooperative : all parties agree to transfer data • one sided : one worker performs transfer of data worker 0 worker 0 worker 1 worker 1 PUT(data) SEND(data) (memory) (memory) RECV(data) GET(data)

  6. What is MPI? • A message-passing library specification • extended message-passing model • for parallel computers, clusters and heterogeneous networks • not a language or compiler specification • not a specific implementation or product • support send/receive primitives communicating with other workers

  7. MPI Characteristics (1/2) • Reliable • in-order data transfer without data loss • Efficient • several point-to-point and collective communications • Extensible • MPI supports the development of parallel libraries • Platform-independent • MPI does not make any restrictive assumptions about the underlying hardware architecture

  8. MPI Characteristics (2/2) • MPI is very large (125 functions) • MPI’s extensive functionality requires many functions • MPI is very small and simple (6 functions) • many parallel programs can be written with just 6 basic functions MPI_Init() MPI_Finalize() MPI_Comm_rank() MPI_Comm_size() MPI_Send() MPI_Recv()

  9. MPI Program Format 0 : #include “mpi.h” /* In Fortran, include ‘mpif.h’ */ 1 : #include <stdio.h> 2 : 3 : int main(int argc, char** argv) { 4 : int rank, size; 5 : MPI_Init(&argc, &argv); 6 : MPI_Comm_rank(MPI_COMM_WORLD, &rank); 7 : MPI_Comm_size(MpI_COMM_WORLD, &size); 8 : printf(“Hello, world! I’m %d of %d\n”, rank, size); 9 : 10: MPI_Finalize(); 11: return 0; 12:}

  10. Initializing and Exiting MPI • Initializing MPI • every MPI program must call this routine once, before any other MPI routines • Clean-up of MPI • every MPI program must call this routine when all communications have completed MPI_Init(int *argv, char ***argv); MPI_Finalize();

  11. MPI_COMM_WORLD Communicator • Communicator • MPI processes can only communicate if they share a communicator • MPI_COMM_WORLD • predefined default communicator in MPI_Init()call MPI_COMM_WORLD 0 5 1 3 2 4 predefined communicator for six processes

  12. Rank & Size • How do you identify different processes? • an MPI process can query a communicator for information about the group • a communicator returns in rank of the calling process • How many processes are contained within a communicator? • a communicator returns the # of processes in the communicator MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size);

  13. Message • Message • an array of elements of a particular MPI datatype • basic MPI datatype • MPI_(UNSIGNED_)CHAR : signed(unsigned) char • MPI_(UNSIGNED_)SHORT : signed(unsigned) short int • MPI_(UNSIGNED_)INT : signed(unsigned) int • MPI_(UNSIGNED_) LONG : signed(unsigned) long int • MPI_FLOAT : float • MPI_DOUBLE : double MPI datatype

  14. Point-to-Point Communications communicator • communication between only 2 processes • source process sends message to destination process • destination process is identified by its rank in the communicator 0 dest 5 1 3 2 source 4

  15. Sending a Message MPI_Send(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_COMM_WORLD); /* (IN) buf : address of the data to be sent */ /* (IN) count : # of elements of the MPI Datatype */ /* (IN) dest : destination process for the message (rank of the destination process) */ /* (IN) tag : marker distinguishes used message type */

  16. Receiving a Message MPI_Recv(void *buf, int count, MPI_Datatype datatype, int source, /* MPI_ANY_SOURCE */ int tag, /* MPI_ANY_TAG */ MPI_COMM_WORLD, MPI_Status *status); /* (IN) buf : address where the data should be placed*/ /* (IN) count : # of elements of the MPI Datatype */ /* (IN) source : rank of the source of the message */ /* (IN) tag : receives a message having this tag*/ /* (OUT) status : some information to be used at later */

  17. integer array (process 0) .... .... .... .... .... (process 0) (process 1) (process 2) (process 3) local sum local sum local sum local sum total sum (process 0) Example

  18. Example (con’d) #include "mpi.h" #include <stdio.h> #include <math.h> #define NPROC 4 #define TOTAL_LEN 40 #define LOCAL_LEN (TOTAL_LEN / NPROC) void InitBuffer(int *buf, int size) { int i; for (i = 0; i < size; i++) buf[i] = i + 1; } int SumFromEachProc(int *buf, int size) { int i; int sum = 0; for (i = 0; i< size; i++) sum += buf[i]; return sum; }

  19. void main(int argc, char *argv[]) { int i; int rank; /* rank = process id */ int nproc; /* number of processes */ int Sum; /* global sum */ int Buf[TOTAL_LEN]; /* global buffer */ int localSum; /* local sum */ int recvBuf[LOCAL_LEN]; /* local buffer */ int recvSum; /* sum from other process */ MPI_Status status; MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &nproc); MPI_Comm_rank(MPI_COMM_WORLD, &rank);

  20. Example (con’d) if (rank == 0) { InitBuffer(Buf, TOTAL_LEN); for (i = 0; i < nproc; i++) MPI_Send(&Buf[i*LOCAL_LEN], LOCAL_LEN, MPI_INT, i, 0, MPI_COMM_WORLD); } MPI_Recv(recvBuf, LOCAL_LEN, MPI_INT, 0, 0, MPI_COMM_WORLD, &status); localSum = SumFromEachProc(recvBuf, LOCAL_LEN); MPI_Send(&localSum, 1, MPI_INT, 0, rank, MPI_COMM_WORLD); if (rank == 0) { Sum = 0; for (i = 0; i < nproc; i++) MPI_Recv(&recvSum, 1, MPI_INT, i, i, MPI_COMM_WORLD, &status); Sum += recvSum; } printf("total sum = %d\n", Sum); } MPI_Finalize(); }

  21. Collective Communications • What is collective communication ? • communications involving a group of processes • called by all processes in a communicator • Three classes of collective operations: • synchronization • barrier • data movement • broadcast, scatter, gather • collective computation • global sum, global maximum, etc.

  22. Barrier Synchronization • Barrier Synchronization • blocks the calling process until all other group members have called it MPI_Barrier(MPI_Comm communicator);

  23. Broadcast Operation • Broadcast data from one process to all processes in a communicator data A0 A0 rank broadcast A0 A0 A0 MPI_Bcast(void *buf, /* INOUT buffer */ int count, MPI_Datatype datatype, int root, MPI_COMM_WORLD)

  24. Scatter Operation • Scatter data from one process to all processes in a communicator data A0 A1 A2 A3 A0 rank scatter A1 A2 A3 MPI_Scatter(void *sendbuf, int sendcount, /* sent to each process */ MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_COMM_WORLD);

  25. Gather Operation • Gather data from all processes in a group to one process data A0 A0 A1 A2 A3 rank gather A1 A2 A3 MPI_Gather(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, /* for any single recv */ MPI_Datatype recvtype, int root, MPI_COMM_WORLD);

  26. Global Reduction Operations • Global reduction operation • sum, max, min, etc. • return a result to a process or all processes A B C D A B C D reduce(op) rank MPI_Reduce(…); MPI_Allredule(…); E F G H E F G H I J K L I J K L

  27. Example int main(int argc, char *argv[]) { /* local variable initialization */ MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &nproc); MPI_Comm_rank(MPI_COMM_WORLD, &rank); InitBuffer(Buf, TOTAL_LEN); MPI_Scatter(Buf, LOCAL_LEN, MPI_INT, recvBuf, LOCAL_LEN, MPI_INT, 0, MPI_COMM_WORLD); localSum = SumFromEachProc(recvBuf, LOCAL_LEN); MPI_Reduce(&localSum, &globalSum, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD); if (rank == 0) printf("global SUM = %d\n", globalSum); MPI_Finalize(); }

  28. Running MPI Program • Environments • cal1.kaist.ac.kr ~ cal8.kaist.ac.kr 8 alpha cluster • 100Mbps Fast Ethernet • linux2.2.14-6 • mpich-1.2.3 MPI implementation • Compiling • mpicc –o <exec_file_name> <src_file_name> • e.g. mpicc –o sum sum.c • Running • modify .rhosts in your home directory • cal1.kaist.ac.kr user-ID, etc. • mpirun –np <# of proc> <exec_file_name> • e.g. mpirun –np 4 sum

  29. References • Learning more MPI • http://www-unix.mcs.anl.gov/mpi/

More Related