OpenMP

1. OpenMP Presented by Kyle Eli

2. OpenMP • Open • Open, Collaborative Specification • Managed by the OpenMP Architecture Review Board (ARB) • MP • Multi Processing

3. OpenMP is… • A specification for using shared memory parallelism in Fortran and C/C++. • Compiler Directives • Library Routines • Environment Variables • Usable with • Fortran 77, Fortran 90, ANSI 89 C or ANSI C++ • Does not require Fortran 90 or C++

4. OpenMP requires… • Platform support • Many operating systems, including Windows, Solaris, Linux, AIX, HP-UX, IRIX, OSX • Many CPU architectures, including x86, x86-64, PowerPC, Itanium, PA-RISC, MIPS • Compiler support • Many commercial compilers from vendors such as Microsoft, Intel, Sun, and IBM • GCC via GOMP • Should be included in GCC 4.2 • May already be available with some distributions

5. OpenMP offers… • Consolidation of vendor-specific implementations • Single-source portability • Support for coarse grain parallelism • Allows for complex (coarse grain) code in parallel applications.

6. OpenMP offers… • Scalability • Simple constructs with low overhead • However, still dependent on the application and algorithm. • Nested parallelism • May be executed on a single thread • Loop-level parallelism • However, no support for task parallelism • They’re working on it

7. OpenMP compared to… • Message Passing Interface (MPI) • OpenMP is not a message passing specification • Less overhead • High Performance Fortran (HPF) • Not widely accepted • Focus on data parallelism

8. OpenMP compared to… • Pthreads • Not targeted for HPC/scientific computing • No support for data parallelism • Requires lower-level programming • FORALL loops • Simple loops • Subroutine calls can’t have side-effects • Various parallel programming languages • May be architecture specific • May be application specific

9. The OpenMP Model • Sequential code • Implemented in the usual way • Executes normally • Parallel code • Multiple threads created • Number of threads can be user-specified • Each thread executes the code in the parallel region

10. Using OpenMP • Compiler directives • Begin with #pragma omp • In C/C++, the code region is defined by curly braces following the directive • Should be ignored by compilers that don’t understand OpenMP • Define how regions of code should be executed • Define variable scope • Synchronization

11. Using OpenMP • Parallel region construct • #pragma omp parallel • Defines a region of parallel code • Causes a team of threads to be created • Threads execute code in the region in the same order • Threads join after the region ends

12. Using OpenMP • Work-sharing directives • For • Sections • Single

13. Using OpenMP • For construct • #pragma omp for • Loop parallelism • Iterations of the loop are divided amongst worker threads • Workload division can be user-specified • Branching out of the loop is not allowed

14. Using OpenMP • Sections construct • #pragma omp sections • Divides code into sections which are divided amongst worker threads • #pragma omp section • Used to define each section

15. Using OpenMP • Single construct • #pragma omp single • Only one thread executes the code • Useful when code is not thread-safe • All other threads wait until execution completes

16. Using OpenMP • Synchronization directives • Master • #pragma omp master • Code is executed only by the master thread • Critical • #pragma omp critical • Code is executed by only one thread at a time • Barrier • #pragma omp barrier • Threads will wait for all other threads to reach this pointbefore continuing • Atomic • #pragma omp atomic • The following statement (which must be an assignment)is executed by only one thread at a time.

17. Using OpenMP • Synchronization Directives • Flush • #pragma omp flush • Thread-visible variables are written back to memory to present a consistent view across all threads • Ordered • #pragma omp ordered • Forces iterations of a loop to be executed in sequential order • Used with the For directive • Threadprivate • #pragma omp threadprivate • Causes global variables to be local and persistent to a thread across multiple parallel regions

18. Using OpenMP • Data Scope • By default, most variables are shared • Loop index and subroutine stack variables are private

19. Using OpenMP • Data scoping attributes • Private • New object of the same type is created for each thread • Not initialized • Shared • Shared amongst all threads • Default • Allows specification of default scope (Private, Shared, or None) • Firstprivate • Variable is initialized with the value from the original object

20. Using OpenMP • Data scoping attributes • Lastprivate • Original object is updated with data from last section or loop iteration • Copyin • Variable in each thread is initialized with the data from the original object in the master thread • Reduction • Each thread gets a private copy of the variable, and the reduction clause allows specification of an operator for combining the private copies into the final result

21. Using OpenMP

22. OpenMP Example • A short OpenMP example…

23. References • http://www.openmp.org • http://www.llnl.gov/computing/tutorials/openMP/

24. MPI By Chris Van Horn

25. What is MPI? • Message Passing Interface • More specifically a library specification for a message passing interface

26. Why MPI? • What are the advantages of a message passing interface? • What could a message passing interface be used for?

27. History of MPI • MPI 1.1 • Before everyone had to implement own message passing interface • Committee formed of around 60 people from 40 organizations

28. MPI 1.1 • The standardization process began in April 1992 • Preliminary draft submitted November 1992 • Just meant to get the ball rolling

29. MPI 1.1 Continued • Subcommittees were formed for the major component areas • Goal to produce standard by Fall 1993

30. MPI Goals • Design an application programming interface (not necessarily for compilers or a system implementation library). • Allow efficient communication: Avoid memory-to-memory copying and allow overlap of computation and communication and offload to communication co-processor, where available. • Allow for implementations that can be used in a heterogeneous environment. • Allow convenient C and Fortran 77 bindings for the interface. • Assume a reliable communication interface: the user need not cope with communication failures. Such failures are dealt with by the underlying communication subsystem.

31. MPI Goals Continued • Define an interface that is not too different from current practice, such as PVM, NX, Express, p4, etc., and provides extensions that allow greater flexibility. • Define an interface that can be implemented on many vendor's platforms, with no significant changes in the underlying communication and system software. • Semantics of the interface should be language independent. • The interface should be designed to allow for thread-safety

32. MPI 2.0 • In March 1995 work began on extensions to MPI 1.1 • Forward Compatibility was preserved

33. Goals of MPI 2.0 • Further corrections and clarifications for the MPI-1.1 document. • Additions to MPI-1.1 that do not significantly change its types of functionality (new datatype constructors, language interoperability, etc.). • Completely new types of functionality (dynamic processes, one-sided communication, parallel I/O, etc.) that are what everyone thinks of as ``MPI-2 functionality.'' • Bindings for Fortran 90 and C++. This document specifies C++ bindings for both MPI-1 and MPI-2 functions, and extensions to the Fortran 77 binding of MPI-1 and MPI-2 to handle Fortran 90 issues. • Discussions of areas in which the MPI process and framework seem likely to be useful, but where more discussion and experience are needed before standardization (e.g. 0-copy semantics on shared-memory machines, real-time specifications).

34. How MPI is used • An MPI program consists of autonomous processes • The processes communicate via calls to MPI communication primitives

35. Features • Process Management • One Sided Communication • Collective Operations • I/O

36. What MPI Does not Do • Resource Control • Not able to design a portable interface that would be appropriate for the broad spectrum of existing and potential resource and process controllers.

37. Process Management • Can be tricky to implement properly • What to watch out for: • The MPI-2 process model must apply to the vast majority of current parallel environments. These include everything from tightly integrated MPPs to heterogeneous networks of workstations. • MPI must not take over operating system responsibilities. It should instead provide a clean interface between an application and system software.

38. Warnings continued • MPI must continue to guarantee communication determinism, i.e., process management must not introduce unavoidable race conditions. • MPI must not contain features that compromise performance. • MPI-1 programs must work under MPI-2, i.e., the MPI-1 static process model must be a special case of the MPI-2 dynamic model.

39. How Issues Addressed • MPI remains primarily a communication library. • MPI does not change the concept of communicator.

40. One Sided Communication • Functions that establish communication between two sets of MPI processes that do not share a communicator. • When would one sided communication be useful?

41. One Sided Communication • How are the two sets of processes going to communicate with each other? • Need some sort of rendezvous point.

42. Collective Operations • Intercommunicator collective operations • All-To-All • All processes contribute to the result. All processes receive the result. • * MPI_Allgather, MPI_Allgatherv • * MPI_Alltoall, MPI_Alltoallv • * MPI_Allreduce, MPI_Reduce_scatter • All-To-One • All processes contribute to the result. One process receives the result. • * MPI_Gather, MPI_Gatherv • * MPI_Reduce

43. Collective Operations • One-To-All • One process contributes to the result. All processes receive the result. • * MPI_Bcast • * MPI_Scatter, MPI_Scatterv • Other • Collective operations that do not fit into one of the above categories. • * MPI_Scan • * MPI_Barrier

44. I/O • Optimizations required for efficiency can only be implemented if the parallel I/O system provides a high-level interface

45. MPI Implementations • Many different implementations most widely used MPICH(1.1) and MPICH2(2.0) • Argonne National Laboratory

46. Examples • To run the program ``ocean'' with arguments ``-gridfile'' and ``ocean1.grd'' in C: • char command[] = "ocean"; • char *argv[] = {"-gridfile", "ocean1.grd", NULL}; • MPI_Comm_spawn(command, argv, ...); • To run the program ``ocean'' with arguments ``-gridfile'' and ``ocean1.grd'' and the program ``atmos'' with argument ``atmos.grd'' in C: • char *array_of_commands[2] = {"ocean", "atmos"}; • char **array_of_argv[2]; • char *argv0[] = {"-gridfile", "ocean1.grd", (char *)0}; • char *argv1[] = {"atmos.grd", (char *)0}; • array_of_argv[0] = argv0; • array_of_argv[1] = argv1; • MPI_Comm_spawn_multiple(2, array_of_commands, array_of_argv, ...);

47. More Examples • /* manager */ • #include "mpi.h" • int main(int argc, char *argv[]) • { • int world_size, universe_size, *universe_sizep, flag; • MPI_Comm everyone; /* intercommunicator */ • char worker_program[100]; • MPI_Init(&argc, &argv); • MPI_Comm_size(MPI_COMM_WORLD, &world_size); • if (world_size != 1) error("Top heavy with management"); • MPI_Attr_get(MPI_COMM_WORLD, MPI_UNIVERSE_SIZE, • &universe_sizep, &flag);

48. Example Continued • if (!flag) { • printf("This MPI does not support UNIVERSE_SIZE. How many\n\ • processes total?"); • scanf("%d", &universe_size); • } else universe_size = *universe_sizep; • if (universe_size == 1) error("No room to start workers"); • /* • * Now spawn the workers. Note that there is a run-time determination • * of what type of worker to spawn, and presumably this calculation must • * be done at run time and cannot be calculated before starting • * the program. If everything is known when the application is • * first started, it is generally better to start them all at once • * in a single MPI_COMM_WORLD. • */ • choose_worker_program(worker_program); • MPI_Comm_spawn(worker_program, MPI_ARGV_NULL, universe_size-1, • MPI_INFO_NULL, 0, MPI_COMM_SELF, &everyone, • MPI_ERRCODES_IGNORE); • /* • * Parallel code here. The communicator "everyone" can be used • * to communicate with the spawned processes, which have ranks 0,.. • * MPI_UNIVERSE_SIZE-1 in the remote group of the intercommunicator • * "everyone". • */ • MPI_Finalize(); • return 0; • }

49. Yet More Example • /* worker */ • #include "mpi.h" • int main(int argc, char *argv[]) • { • int size; • MPI_Comm parent; • MPI_Init(&argc, &argv); • MPI_Comm_get_parent(&parent); • if (parent == MPI_COMM_NULL) error("No parent!"); • MPI_Comm_remote_size(parent, &size); • if (size != 1) error("Something's wrong with the parent"); • /* • * Parallel code here. • * The manager is represented as the process with rank 0 in (the remote • * group of) MPI_COMM_PARENT. If the workers need to communicate among • * themselves, they can use MPI_COMM_WORLD. • */ • MPI_Finalize(); • return 0; • }

50. References • MPI Standards (http://www-unix.mcs.anl.gov/mpi/mpi-standard/mpi-report-2.0/mpi2-report.htm)