OpenMP

1. OpenMP A Standard for Shared Memory Parallel Programming Seyong Lee Purdue University School of Electrical and Computer Engineering

2. Overview • Introduction to OpenMP • OpenMP Programming Model • OpenMP Directives • Run-Time Library Routine • Environment Variables • Summary

3. I. Introduction to OpenMP What is OpenMP? • Application program interface (API) for shared memory parallel programming • A specification for a set of compiler directives, library routines, and environment variables • Make it easy to create multi-threaded (MT) programs in Fortran, C and C++ • Portable / multi-platform, including Unix platforms and Windows NT platforms • Jointly defined and endorsed by a group of major computer hardware and software vendors

4. OpenMP is not…. • Not Automatic parallelization - User explicitly specifies parallel execution - Compiler does not ignore user directives even if wrong • Not just loop level parallelism - Functionality to enable coarse grained parallelism • Not meant for distributed memory parallel systems • Not necessarily implemented identically by all vendors • Not Guaranteed to make the most efficient use of shared memory

5. Why OpenMP? • Parallel programming before OpenMP - Standard way to program distributed memory computers (MPI and PVM) - No standard API for shared memory programming • Several vendors had directive based API for shared memory programming - All different, vendor proprietary • Commercial users, high end software vendors have big investment in existing code - Not very eager to rewrite their code in new language • Portability possible only through MPI - Library based, good performance and scalability - But sacrifice the built in shared memory advantage of the hardware

6. Goals of OpenMP • Standardization : - Provide a standard among a variety of shared memory architectures/platforms • Lean and mean : - Establish a simple and limited set of directives for programming shared memory machines. • Ease of Use : - Provide capability to incrementally parallelize a serial program - Provide the capability to implement both coarse-grain and fine-grain parallelism • Portability : - Support Fortran (77, 90, and 95), C, and C++

7. Introduction to OpenMP • OpenMP Programming Model • OpenMP Directives • Run-Time Library Routine • Environment Variables • Summary

8. II. OpenMP Programming Model • Thread Based Parallelism • Explicit Parallelism • Fork-Join Model • Compiler Directive Based • Nested Parallelism Support • Dynamic Threads

9. User Interface Model • Compiler Directives - Most of the API : Control constructs, Data attribute constructs - Extends base language : f77, f90, C, C++ - Example : C$OMP PARALLEL DO • Library - Small set of functions to control threads and to implement unstructured locks - Example : call omp_set_num_threads(128) • Environment Variables - For end users to control run time execution - Example : setenv OMP_NUM_THREADS 8

10. Execution Model

11. Introduction to OpenMP • OpenMP Programming Model • OpenMP Directives • Run-Time Library Routine • Environment Variables • Summary

12. III. OpenMP Directives • OpenMP Directives Format • Directive Scoping • PARALLEL Region Construct • Work-Sharing Constructs • Synchronization Constructs • Data Environment Constructs

13. OpenMP Directives Format • Fortran Directives Format C$OMP construct [clause [clause]…] !$OMP construct [clause [clause]…] *$OMP construct [clause [clause]…] !$OMP PARALLEL DEFAULT (SHARED) PRIVATE (a, b) [structured block of code] !$OMP END PARALLEL cf. !$ a = OMP_get_thread_num() • C / C++ Directives Format #pragma omp construct [clause [clause]…] #pragma omp parallel default (shared) private(a,b) { [structured block of code] } /* all threads join master thread and terminate */

14. Structured blocks • Most OpenMP constructs apply to structured block - Structured block: A block of code with one point of entry at the top and one point of exit at the bottom. The only other branches allowed are STOP statements in Fortran and exit() in C/C++ C$OMP PARALLEL 10 wrk(id) = garbage (id) 30 res (id) = wrk(id) ** 2 if (conv(res(id)) goto 20 go to 10 C$OMP END PARALLEL if (not_DONE) goto 30 20 print *, id C$OMP PARALLEL 10 wrk(id) = garbage (id) res (id) = wrk(id) ** 2 if (conv(res(id)) goto 10 C$OMP END PARALLEL print *, id A structured block Not a structured block

15. III. OpenMP Directives • OpenMP Directives Format • Directive Scoping • PARALLEL Region Construct • Work-Sharing Constructs • Synchronization Constructs • Data Environment Constructs

16. Directive Scoping • Static (Lexical) Extent : - The block of code directly placed between the two directives !$OMP PARALLEL and !$OMP END PARALLEL • Dynamic Extent : - The code included in the lexical extent plus all the code called from inside the lexical extent • Orphaned Directive : - An OpenMP directive that appears independently from another enclosing directives. It exists outside of another directive’s static extent • Example PROGRAM TEST … !$OMP PARALLEL … !$OMP DO DO I = … … CALL SUB1 … ENDDO !$OMP END DO … CALL SUB2 … !$OMP END PARALLEL SUBROUTINE SUB1 … !$OMP CRITICAL … !$OMP END CRITICAL END SUBROUTINE SUB2 … !$OMP SECTIONS … !$OMP END SECTIONS … END Orphaned Directive Static Extent

17. III. OpenMP Directives • OpenMP Directives Format • Directive Scoping • PARALLEL Region Construct • Work-Sharing Constructs • Synchronization Constructs • Data Environment Constructs

18. PARALLEL Region Construct • A block of code that will be executed by multiple threads • Properties - Fork-Join Model - Number of threads won’t change inside a parallel region - SPMD execution within region - Enclosed block of code must be structured, no branching into or out of block • Format !$OMP PARALLEL clause1 clause2 … … !$OMP END PARALLEL

19. PARALLEL Region Construct !$OMP PARALLEL write (*,*) “Hello” !$OMP END PARALLEL

20. PARALLEL Region Construct • How many threads? 1. Use of the omp_set_threads() library function 2. Setting of the OMP_NUM_THREADS environment variable 3. Implementation default • Dynamic Threads : - By default, the same number of threads are used to execute each parallel region - Two methods for enabling dynamic threads 1. Use of the omp_set_dynamic() library function 2. Setting of the OMP_DYNAMIC environment variable

21. III. OpenMP Directives • OpenMP Directives Format • Directive Scoping • PARALLEL Region Construct • Work-Sharing Constructs • Synchronization Constructs • Data Environment Constructs

22. Work-Sharing Constructs • Divide the execution of the enclosed code region among the members of the team that encounter it • Do not launch new threads • Must be enclosed within a parallel region • No-implied barrier upon entry to a work-sharing construct • An implied barrier at the end of a work-sharing construct • Type of work-sharing constructs • DO Directive: !OMP DO / !$OMP END DO • SECTIONS Directive: !OMP SECTIONS / !$OMP END SECTIONS • SINGLE Directive: !OMP SINGLE / !$OMP END SINGLE

23. Work-Sharing Constructs • DO Directive • Format !$OMP DO clause1 clause2 … [do loop] !$ OMP END DO end_clause

24. Work-Sharing Constructs • How iterations of the loop are divided? • => use SHEDULE (type, chunk) clause • STATIC DYNAMIC GUIDED

25. Work-Sharing Constructs • SECTIONS Directive - Non-iterative work-sharing - Each section is executed once by a thread - Potential MIMD? • Format !$OMP SECTIONS clause1, clause2… !$OMP SECTION [block1] !$OMP SECTION [block2] … !$OMP END SECTIONS end_clause

26. Work-Sharing Constructs • SECTIONS Directive Example code !$OMP SECTIONS !$OMP SECTION write(*,*) “Hello” !$OMP SECTION write(*,*) “Hi” !$OMP SECTION write(*,*) “Bye” !$OMP ENT SECTIONS

27. Work-Sharing Constructs • SINGLE Directive - Encloses code to be executed by only one thread in the team - Threads in the team that do not execute the SINGLE directive, wait at the end of the enclosed code block unless a NOWAIT clause is specified • Format !$OMP SINGLE clause1 clause2… … !$OMP END SINGLE end_clause

28. Work-Sharing Constructs • SINGLE DIRECTIVE Example code !$OMP SINGLE write(*,*) “Hello” !$OMP END SINGLE

29. III. OpenMP Directives • OpenMP Directives Format • Directive Scoping • PARALLEL Region Construct • Work-Sharing Constructs • Synchronization Constructs • Data Environment Constructs

30. Synchronization Constructs • OpenMP has the following construct to support synchronization: - MASTER Directive - CRITICAL Directive - BARRIER Directive - ATOMIC Directive - ORDERED Directive - FLUSH Directive

31. Synchronization Constructs • MASTER Directive - Executed only by the master thread of the team - No implied barrier associated with this directive • Format !$OMP MASTER … !$OMP END MASTER

32. Synchronization Constructs • CRITICAL Directive - Specified a region of code that must be executed by only one thread at a time - The optional name enables multiple different CRITICAL regions to exist • Format !$OMP CRITICAL name … !$OMP END CRITICAL name

33. Synchronization Constructs • BARRIER Directive - Synchronize all threads in the team - When encountered, each thread waits until all the other threads have reached this point - Must be encountered by all threads in a team or by non at all: otherwise, deadlock • Format !$OMP BARRIER

34. Synchronization Constructs • FLUSH Directive - Explicit synchronization point at which the implementation is required to provide a consistent view of memory - Thread-visible variables are written to back to memory at this point • Format !$OMP FLUSH (variable1, variable2, …) • The FLUSH directives is implied for the directives shown in the table below. - The directive is not implied if NOWAIT clause is present BARRIER END SECTIONS CRITICAL and END CRITICAL END SINGLE END DO ORDERED and END ORDERED END PARALLEL

35. III. OpenMP Directives • OpenMP Directives Format • Directive Scoping • PARALLEL Region Construct • Work-Sharing Constructs • Synchronization Constructs • Data Environment Constructs

36. Data Environment Constructs • Define how and which data variables in the serial section of the program are transferred to the parallel sections of the program (and back) • Define which variables will be visible to all threads and which variable be private • Include - Directive THREADPRIVATE - Data scope attribute clause PRIVATE SHARED FIRSTPRIVATE LASTPRIVATE DEFAULT COPYIN REDUCTION

37. Data Environment Constructs • THREADPRIVATE Directive - Make global file scope variable or common blocks local and persistent to a thread - Use COPYIN clause to initialize data in THREADPRIVATE variables and common blocks • Format !$OMP THREADPRIVATE (a, b, …) • Data scope attribute clauses - PRIVATE clause !$OMP PARALLEL PRIVATE (a,b) - SHARED clause !$OMP PARALLEL SHARED (c,d)

38. Data Environment Constructs • Data scope attribute clauses - FIRSTPRIVATE clause : PRIVATE with automatic initialization !$OMP PARALLEL FIRSTPRIVATE (a,b) - LASTPRIVATE clause : PRIVATE with a copy from the last loop iteration or section to the original variable object !$OMP PARALLEL LASTPRIVATE (a,b) - DEFAULT clause : Specify a default PRIVATE, SHARED, or NONE scope for all variables in the lexical extent of any parallel region !$OMP PARALLEL DEFAULT (PRIVATE | SHARED | NONE) - COPYIN clause : Assign the same value to THREADPRIVATE variables for all thread in the team !$OMP PARALLEL COPYIN (a)

39. Introduction to OpenMP • OpenMP Programming Model • OpenMP Directives • Run-Time Library Routine • Environment Variables • Summary

40. IV. Run-Time Library Routine • API for library calls that performs a variety functions - For C/C++ : include “omp.h” - Fortran 95 : use “omp_lib” module • Runtime environment routines : - Modify/Check the number of threads - OMP_SET_NUM_THREADS(), OMP_GET_NUM_THREADS(), - OMP_GET_MAX_THREADS(), OMP_GET_THREAD_NUM() - Turn on/off nesting and dynamic mode - OMP_SET_NESTED(), OMP_GET_NESTED(), - OMP_SET_DYNAMIC(), OMP_GET_DYNAMIC() - Are we in a parallel region? - OMP_IN_PARALLEL() - How many processors in the system? - OMP_GET_NUM_PROCS()

41. Run-Time Library Routine • Lock routines - Lock : A flag which can be set or unset. - Ownership of the lock : A thread who sets a given lock to get some privileges - Lock differs from other synchronization directives Only threads related to the lock are affected by the status of the lock - Related functions : - OMP_INIT_LOCK(), OMP_SET_LOCK(), - OMP_UNSET_LOCK(), OMP_DESTROY_LOCK(), - OMP_TEST_LOCK()

42. Introduction to OpenMP • OpenMP Programming Model • OpenMP Directives • Run-Time Library Routine • Environment Variables • Summary

43. V. Environment Variables • Control how “OMP DO SCHEDULE(RUNTIME)” loop iterations are scheduled. - OMP_SCHEDULE “schedule,chunk_size” • Set the default number of threads to use - OMP_NUM_THREADS int_literal • Can the program use a different number of threads in each parallel region ? - OMP_DYNAMIC TRUE || FALSE • Will nested parallel region create new teams of threads? - OMP_NESTED TRUE || FALSE

44. VI. Summary • OpenMP is a directive based shared memory programming model • OpenMP API is a general purpose parallel programming API with emphasis on the ability to parallelize existing programs • Scalable parallel programs can be written by using parallel regions • Work-sharing constructs enable efficient parallelization of computationally intensive portions of program