CS 584

1. CS 584

2. Designing Parallel Algorithms • Designing a parallel algorithm is not easy. • There is no recipe or magical ingredient • Except creativity • We can benefit from a methodical approach. • Framework for algorithm design • Most problems have several parallel solutions which may be totally different from the best sequential algorithm.

3. PCAM Algorithm Design • 4 Stages to designing a parallel algorithm • Partitioning • Communication • Agglomeration • Mapping • P & C focus on concurrency and scalability. • A & M focus on locality and performance.

4. PCAM Algorithm Design • Partitioning • Computation and data are decomposed. • Communication • Coordinate task execution • Agglomeration • Combining of tasks for performance • Mapping • Assignment of tasks to processors

6. Partitioning • Ignore the number of processors and the target architecture. • Expose opportunities for parallelism. • Divide up both the computation and data • Can take two approaches • domain decomposition • functional decomposition

7. Domain Decomposition • Start algorithm design by analyzing the data • Divide the data into small pieces • Approximately equal in size • Then partition the computation by associating it with the data. • Communication issues may arise as one task needs the data from another task.

8. 1 4 ò + 2 1 x 0 Domain Decomposition • Evaluate the definite integral. 0 1

9. Split up the domain 0 1

10. Split up the domain 0 1

11. Split up the domain Now each task simply evaluates the integral in their range. All that is left is to sum up each task's answer for the total. 0 0.25 0.5 0.75 1

12. Domain Decomposition • Consider dividing up a 3-D grid • What issues arise? • Other issues? • What if your problem has more than one data structure? • Different problem phases? • Replication?

15. Functional Decomposition • Focus on the computation • Divide the computation into disjoint tasks • Avoid data dependency among tasks • After dividing the computation, examine the data requirements of each task.

16. Functional Decomposition • Not as natural as domain decomposition • Consider search problems • Often functional decomposition is very useful at a higher level. • Climate modeling • Ocean simulation • Hydrology • Atmosphere, etc.

17. Partitioning Checklist • Define a LOT of tasks? • Avoid redundant computation and storage? • Are tasks approximately equal? • Does the number of tasks scale with the problem size? • Have you identified several alternative partitioning schemes?

18. Communication • The information flow between tasks is specified in this stage of the design • Remember: • Tasks execute concurrently. • Data dependencies may limit concurrency.

19. Communication • Define Channel • Link the producers with the consumers. • Consider the costs • Intellectual • Physical • Distribute the communication. • Specify the messages that are sent.

20. Communication Patterns • Local vs. Global • Structured vs. Unstructured • Static vs. Dynamic • Synchronous vs. Asynchronous

21. Local Communication • Communication within a neighborhood. Algorithm choice determines communication.

22. Global Communication • Not localized. • Examples • All-to-All • Master-Worker 1 5 2 3 7

23. Avoiding Global Communication • Distribute the communication and computation 1 5 2 3 7 13 10 3 1 5 3 7 2 1

24. 7 8 3 5 1 5 3 2 1 Divide and Conquer • Partition the problem into two or more subproblems • Partition each subproblem, etc. Results in structured nearest neighbor communication pattern.

25. Structured Communication • Each task’s communication resembles each other task’s communication • Is there a pattern?

26. Unstructured Communication • No regular pattern that can be exploited. • Examples • Unstructured Grid • Resolution changes Complicates the next stages of design

27. Synchronous Communication • Both consumers and producers are aware when communication is required • Explicit and simple t = 1 t = 2 t = 3

28. Asynchronous Communication • Timing of send/receive is unknown. • No pattern • Consider: very large data structure • Distribute among computational tasks (polling) • Define a set of read/write tasks • Shared Memory

29. Problems to Avoid • A centralized algorithm • Distribute the computation • Distribute the communication • A sequential algorithm • Seek for concurrency • Divide and conquer • Small, equal sized subproblems

30. Communication Design Checklist • Is communication balanced? • All tasks about the same • Is communication limited to neighborhoods? • Restructure global to local if possible. • Can communications proceed concurrently? • Can the algorithm proceed concurrently? • Find the algorithm with most concurrency. • Be careful!!!

31. Agglomeration • Partition and Communication steps were abstract • Agglomeration moves to concrete. • Combine tasks to execute efficiently on some parallel computer. • Consider replication.

32. Agglomeration Goals • Reduce communication costs by • increasing computation • decreasing/increasing granularity • Retain flexibility for mapping and scaling. • Reduce software engineering costs.

33. Changing Granularity • A large number of tasks does not necessarily produce an efficient algorithm. • We must consider the communication costs. • Reduce communication by • having fewer tasks • sending less messages (batching)

34. Surface to Volume Effects • Communication is proportional to the surface of the subdomain. • Computation is proportional to the volume of the subdomain. • Increasing computation will often decrease communication.

35. How many messages total? How much data is sent?

36. How many messages total? How much data is sent?

37. Replicating Computation • Trade-off replicated computation for reduced communication. • Replication will often reduce execution time as well.

38. Summation of N Integers s = sum b = broadcast How many steps?

39. Using Replication (Butterfly)

40. Using Replication Butterfly to Hypercube

41. Avoid Communication • Look for tasks that cannot execute concurrently because of communication requirements. • Replication can help accomplish two tasks at the same time, like: • Summation • Broadcast

42. Preserve Flexibility • Create more tasks than processors. • Overlap communication and computation. • Don't incorporate unnecessary limits on the number of tasks.

43. Agglomeration Checklist • Reduce communication costs by increasing locality. • Do benefits of replication outweigh costs? • Does replication compromise scalability? • Does the number of tasks still scale with problem size? • Is there still sufficient concurrency?

44. Mapping • Specify where each task is to operate. • Mapping may need to change depending on the target architecture. • Mapping is NP-complete.

45. Mapping • Goal: Reduce Execution Time • Concurrent tasks ---> Different processors • High communication ---> Same processor • Mapping is a game of trade-offs.

46. Mapping • Many domain-decomposition problems make mapping easy. • Grids • Arrays • etc.

47. Mapping • Unstructured or complex domain decomposition based algorithms are difficult to map.

48. Other Mapping Problems • Variable amounts of work per task • Unstructured communication • Heterogeneous processors • different speeds • different architectures • Solution: LOAD BALANCING

49. Load Balancing • Static • Determined a priori • Based on work, processor speed, etc. • Probabilistic • Random • Dynamic • Restructure load during execution • Task Scheduling (functional decomp.)

50. Static Load Balancing • Based on a priori knowledge. • Goal: Equal WORK on all processors • Algorithms: • Basic • Recursive Bisection