Global View Languages in Parallel Programming

1. Chapter 7:MPI and Other Local View Languages Principles of Parallel Programming First Edition by Calvin Lin Lawrence Snyder

2. Figure 7.1 An MPI solution to the Count 3s problem.

3. Figure 7.1 An MPI solution to the Count 3s problem. (cont.)

4. Code Spec 7.1MPI_Init().

5. Code Spec 7.2MPI_Finalize().

6. Code Spec 7.3MPI_Comm_Size().

7. Code Spec 7.4MPI_Comm_Rank().

8. Code Spec 7.5MPI_Send().

9. Code Spec 7.6MPI_Recv().

10. Code Spec 7.7MPI_Reduce().

11. Code Spec 7.8MPI_Scatter().

12. Code Spec 7.8MPI_Scatter(). (cont.)

13. Figure 7.2 Replacement code (for lines 16–48 of Figure 7.1) to distribute data using a scatter operation.

14. Code Spec 7.9MPI_Gather().

15. Figure 7.3 Each message must be copied as it moves across four address spaces, each contributing to the overall latency.

16. Code Spec 7.10MPI_Scan().

17. Code Spec 7.11MPI_Bcast(). MPI routine to broadcast data from one root process to all other processes in the communicator.

18. Code Spec 7.12MPI_Barrier().

19. Code Spec 7.13MPI_Wtime().

20. Figure 7.4 Example of collective communication within a group.

21. Code Spec 7.14MPI_Comm_group().

22. Code Spec 7.15MPI_Group_incl().

23. Code Spec 7.16MPI_Comm_create().

24. Figure 7.5 A 2D relaxation replaces—on each iteration—all interior values by the average of their four nearest neighbors.

25. Figure 7.6 MPI code for the main loop of the 2D SOR computation.

26. Figure 7.6 MPI code for the main loop of the 2D SOR computation. (cont.)

27. Figure 7.6 MPI code for the main loop of the 2D SOR computation. (cont.)

28. Figure 7.7 Depiction of dynamic work redistribution in MPI.

29. Figure 7.8 A 2D SOR MPI program using non-blocking sends and receives.

30. Figure 7.8 A 2D SOR MPI program using non-blocking sends and receives. (cont.)

31. Figure 7.8 A 2D SOR MPI program using non-blocking sends and receives. (cont.)

32. Code Spec 7.17MPI_Waitall().

33. Figure 7.9 Creating a derived data type.

34. Partitioned Global Address Space Languages Higher level of abstraction Built on top of distributed memory clusters Considered a single address space Allows definition of global data structures Must consider local vs global data No longer consider message passing details or distributed data structures Use a more efficient one sided substrate

35. Main PGAS • Co-Array Fortran • https://bluewaters.ncsa.illinois.edu/caf • Unified Parallel C • http://upc.lbl.gov/ • Titanium • http://titanium.cs.berkeley.edu/

36. Co-Array Fortran (CAF) • Extends FORTRAN • Originally called F - - • Elegant and simple • Uses co-array (communication array) • Real, dimension (n,n)[p,*]:: a, b, c • a, b, c are co-arrays • Memory for co-array is dist across each process determined by the dimension statement

37. Unified Parallel C (UPC) • Global view of address space • Shared arrays are distributed in cyclic or block cyclic arrangement (aides load balancing) • Supports pointers (C) 4 types • private private • shared private • private shared • shared shared

38. C pointers • Private pointer pointing locally • int *p1; • Private pointer pointing to shared space • shared int *p2; • Shared pointer pointing locally • int *shared p3; • Shared pointer pointing into shared space • shared int *shared p4;

39. UPC • Has a forall verb • upc_forall • Distributes normal C for loop iterations across all processes • A global operation whereas most other operations are local

40. Titanium • Extends java • Object oriented • Adds regions • Supports safe memory management • Unordered iteration • Foreach • Allows concurrency over multiple indices in a block