Chapter 4

1. Chapter 4 Message-Passing Programming

2. The Message-Passing Model

3. What is MPI? • MPI (Message Passing Interface). • MPI is a library specification for message passing. • MPI was designed for high performance on both massively parallel machines and on workstation clusters.

4. MPI History • In 1992, Supercomputing conference agreed to develop and then implement a common standard for message passing. • The first MPI standard, called MPI-1 was completed in May 1994. •  The second MPI standard, MPI-2, was completed in 1998.

5. MPI History •  The most popular one being the Argonne's MPICH  (based on the P4 package and Chameleon  - hence the ``CH'' suffix). • There were still many supercomputer vendors that released their own implementations of MPI. (i.e. LAM MPI, Intel MPI e.t.c.)

6. What’s in MPI-2 • Dynamic process management • Adding processes to a running MPI computation • Parallel I/O • C++ and Fortran 90 bindings • Misc • Interaction with threads • Interoperability between language • Extensions/enhancements to MPI-1

7. Why MPI? • Portability: MPI has been implemented for almost every distributed memory architecture. • Speed: Each implementation is in principle optimized for the hardware upon which it runs. • Most MPI implementations are directly callable from Fortran, C and C++, and from any language capable of interfacing with such libraries (such as C#, JAVA or Python).

8. Message Passing • A paradigm of communication where messages are sent from a sender to one or more recipients. • Message Passing Systems may have satisfied the following conditions: 1. Transferredreliably.2. Guaranteed to be delivered in order.3. Synchronous or asynchronous.4. Passed one-to-one, one-to-many or many-to-one.

9. Programming Model (1/2) • MPI lends itself to virtually any distributed memory parallel programming model. • As shared memory systems became more popular, particularly SMP/NUMA architectures, MPI implementations for these platforms appeared. • MPI is now used on just about any common parallel architecture including massively parallel machines, SMP clusters, workstation cluster and heterogeneous networks.

10. Programming Model (2/2) • The number of tasks dedicated to run a parallel program is static. New tasks can not be dynamically spawned during run time. • All parallelism is explicit: • The programmer is responsible for correctly identifying parallelism and implementing parallel algorithms using MPI constructs.

11. Error Handling • By default, an error causes all processes to abort. • The user can cause routines to return (with an error code) instead. • In C++, exceptions are thrown (MPI-2) • A user can also write and install custom error handlers.

12. The Process • A process is a program in execution. A program is not a process; a program is a passive entity, whereas a process is an active entity.

13. SPMD Coding Model • Single Program Multiple Data

14. Communicators and Groups • Communicator objects connect groups of processes in the MPI session. • Each communicator gives each contained process an independent identifier and arranges its contained processes in an ordered topology. • MPI understands single group intracommunicator operations, and bilateral intercommunicator communication.

15. Rank • Within a communicator, every process has its own unique integer identifier (i.e., Rank). • A rank is also called a “task ID”. • Ranks are contiguous and begin at zero. • Used by the programmer to specify the source and destination of messages.

16. MPI Program Architecture MPI include file Declarations, prototypes, etc. Program Begins (Serial code) Initialize MPI environment (Parallel code begins) Do work and make message passing calls Terminate MPI Environment (Parallel code ends) (Other Serial code)

17. Booting the MPI in Cluster for Administer • In administer (root), the Intel MPI Library uses a Multi-Purpose Daemon (MPD) job startup mechanism. To run programs compiled with mpicc (or related) commands, you must first set up MPD daemons. • In administer (root), shutdown the MPD daemon: pn1: ~ # mpd & [1] 5380 pn1: ~ # mpdallexit

18. Booting the MPI in Cluster for Users • Before running a MPI program, you need to boot machines by “mpdboot” command • Use the MPDTRACE daemon to trace booting nodes (computers): user@pn1:~> mpdboot -n 4 user@pn1:~> mpdtrace pn1 pn4 pn3 pn2

19. Compiling MPI Programs • In general, starting an MPI program is dependent on the implementation of MPI you are using, and might require various scripts, program arguments, and/or environment variables. • In C language: • Use the gcc compiler • Use the intel compiler user@pn1:~> mpicc -o a.out sample.c user@pn1:~> mpiicc -o a.out sample.c

20. Running a Program in Cluster • To run a MPI program, the command is as follow: • -n is the number of processes that run on the program. user@pn1:~> mpiexec -n 8 ./a.out Process 5 of 8 is on pn4 Process 6 of 8 is on pn4 Process 0 of 8 is on pn1 Process 2 of 8 is on pn1 Process 1 of 8 is on pn1 Process 3 of 8 is on pn1 Process 7 of 8 is on pn4 Process 4 of 8 is on pn4 pi is approximately 3.1415926535896137, Error is 0.0000000000001794 wall clock time = 0.231002

21. A Minimal MPI Program in C #include “mpi.h” #include <stdio.h> int main(int argc, char *argv[]){ int id, t_process; MPI_Init(&argc,&argv); MPI_Comm_rank(MPI_COMM_WORLD, &id); MPI_Comm_size(MPI_COMM_WORLD, &t_process); printf(“Hello, world! This is process %d of %d\n”,id, t_process); MPI_Finalize(); return 0; }

22. Sample Result • The sample result is as follow: user@pn1:~> mpiicc -o a.out sample.c user@pn1:~> mpirexec -n 8 ./a.out Hello, world! This is process 0 of 8 Hello, world! This is process 6 of 8 Hello, world! This is process 4 of 8 Hello, world! This is process 5 of 8 Hello, world! This is process 7 of 8 Hello, world! This is process 2 of 8 Hello, world! This is process 1 of 8 Hello, world! This is process 3 of 8

23. Sample Result START Process0 Process1 Process2 MPI_Init MPI_Init MPI_Init MPI_Rank id = 0 MPI_Rank id = 1 MPI_Rank id = 2 MPI_Comm_size t_process = 3 MPI_Comm_size t_process = 3 MPI_Comm_size t_process = 3 • Print Hello world! This is process 0 of 3 • Print Hello world! This is process 1 of 3 • Print Hello world! This is process 2 of 3 MPI_Finalize MPI_Finalize MPI_Finalize STOP

24. MPI Initial Function • MPI_Init must be called • before any other MPI function, • in every MPI program, • only once in an MPI program. • Note: All MPI identifiers, including function identifiers, begin with the prefix MPI_ , followed by a capital letter and a series of lowercase letters and underscores.

25. MPI_Comm_rank and MPI_Comm_size • MPI_COMM_WORLD is the default communicator that you get “for free”. • MPI_Comm_rank is to determine its rank within a communicator. • MPI_Comm_size is to determine the total number of processes in a communicator.

26. MPI Finalize Function • Terminates the MPI execution environment. • Should be the last MPI routine called in every MPI program. • Allowing the system to free up resources (such as memory) that have been allocated to MPI.

27. Circuit Satisfiability

29. #include <mpi.h> #include <stdio.h> Int main (int argc, char *argv[]) { int i; int id; int p; void check_circuit (int, int); MPI_Init (&argc, &argv); MPI_Comm_rank (MPI_COMM_WORLD, &id); MPI_Comm_size (MPICOMM_WORLD, &p); for (i = id; i < 65536; i += p) check_circuit (id, i); printf {“process %d is done\n”, id); fflush (stdout); MPI_Finalize (); return 0; }

30. #define EXTRACT_BIT(n,i) ((n& (I << i))? 1:0) void check_circuit (int id, int z) { int v[16]; int i; for (i = 0; i < 16; i++) v[i] = EXTRACT_BIT (z,i); if ((v[0] || v[1]) && (!v[1] || !v[3]) && (v[2] || v[3]) && ……) { printf (“%d) %d%d ….%d“, id, v[0], v[1], …, v[15]); fflush (stdout); } }

31. Output (1/3)

32. Output (2/3)

33. Output (3/3)