Advanced Algorithms

1. Advanced Algorithms Piyush Kumar (Lecture 16: Parallel Algorithms) Courtesy Baker 05. Welcome to COT5405

2. Parallel Models • An abstract description of a real world parallel machine. • Attempts to capture essential features (and suppress details?) • What other models have we seen so far? RAM? External Memory Model?

3. RAM • Random Access Machine Model • Memory is a sequence of bits/words. • Each memory access takes O(1) time. • Basic operations take O(1) time: Add/Mul/Xor/Sub/AND/not… • Instructions can not be modified. • No consideration of memory hierarchies. • Has been very successful in modelling real world machines.

4. Parallel RAM aka PRAM • Generalization of RAM • P processors with their own programs (and unique id) • MIMD processors : At each point in time the processors might be executing different instructions on different data. • Shared Memory • Instructions are synchronized among the processors

5. PRAM Shared Memory EREW/ERCW/CREW/CRCW EREW: A program isnt allowed to access the same memory location at the same time.

6. Variants of CRCW • Common CRCW: CW iff processors write same value. • Arbitrary CRCW • Priority CRCW • Combining CRCW

7. Why PRAM? • Lot of literature available on algorithms for PRAM. • One of the most “clean” models. • Focuses on what communication is needed ( and ignores the cost/means to do it) • Most ideas translate to other models.

8. Problems with PRAM • Unrealistic • Constant time memory access? • Fixed number of processors

9. PRAM Algorithm design. • Problem 1: Produce the sum of an array of n numbers. • RAM = ? • PRAM = ?

10. 1st prefix 2nd prefix 3rd prefix ... (n-1)th prefix s0 s0Ä s1 s0Ä s1Ä s2 ... s0Ä s1Ä ... Ä sn-1 Problem 2: Prefix Computation Let X = {s0, s1, …, sn-1} be in a set S Let Ä be a binary, associative, closed operator with respect to S (usually Q(1) time – MIN, MAX, AND, +, ...) The result of s0Ä s1Ä…Ä sk is called the k-th prefix Computing all such n prefixes is the parallel prefix computation

11. Prefix computation • Suffix computation is a similar problem. • Assumes Binary op takes O(1) • In RAM = ?

12. Prefix Computation (Akl)

13. EREW PRAM Prefix computation • Assume PRAM has n processors and n is a power of 2. • Input: si for i = 0,1, ... , n-1. • Algorithm Steps: forj = 0 to (lg n) -1, do for i = 2j to n-1 do h = i - 2j si= shÄsi endfor endfor Total time in EREW PRAM?

14. Problem 3: Array packing • Assume that we have • an array of n elements, X = {x1, x2, ... , xn} • Some array elements are marked (or distinguished). • The requirements of this problem are to • pack the marked elements in the front part of the array. • place the remaining elements in the back of the array. • While not a requirement, it is also desirable to • maintain the original order between the marked elements • maintain the original order between the unmarked elements

15. In RAM? • How would you do this? • Inplace? • Running time? • Any ideas on how to do this in PRAM?

16. EREW PRAM Algorithm • Set si in Pi to 1 if xi is marked and set si = 0 otherwise. 2. Perform a prefix sum on S =(s1, s2 ,..., sn) to obtain destination di = si for each marked xi . 3. All PEs set m = sn , the total nr of marked elements. 4. Pi sets si to 0 if xi is marked and otherwise sets si = 1. 5. Perform a prefix sum on S and set di = si + m for each unmarked xi . 6. Each Pi copies array element xi into address di in X.

17. Array Packing • Assume n processors are used above. • Optimal prefix sums requires O(lg n) time. • The EREW broadcast of sn needed in Step 3 takes O(lg n) time using a binary tree in memory • All and other steps require constant time. • Runs in O(lg n) time and is cost optimal. • Maintains original order in unmarked group as well Notes: • Algorithm illustrates usefulness of Prefix Sums • There many applications for Array Packing algorithm

18. Problem 4: PRAM MergeSort • RAM Merge Sort Recursion? • PRAM Merge Sort recursion? • Can we speed up the merging? • Merging n elements with n processors can be done in O(log n) time. • Assume all elements are distinct • Rank(a, A) = number of elements in A smaller than a. For example rank(8, {1,3,5,7,9}) = 4

19. 1 2 3 8 10 12 14 15 16 19 PRAM Merging A = 2,3,10,15,16 B = 1,8,12,14,19 Rank(2)=1 Rank(3)=1 Rank(10)=2 Rank(15)=4 Rank(16)=4 +1 +2 +3 +4 +5 +1 +2 +3 +4 +5 Rank(1)=0 Rank(8)=2 Rank(12)=3 Rank(14)=3 Rank(19)=5

20. PRAM Merge Sort • T(n) = T(n/2) + O(log n) • Using the idea of pipelined d&c PRAM Mergesort can be done in O(log n). • D&C is one of the most powerful techniques to solve problems in parallel.

21. 21 12 Problem 5: Closest Pair • RAM Version ? L 7 6 5 4  = min(12, 21) 3 2 1

22. Closest Pair: RAM Version Closest-Pair(p1, …, pn) { Compute separation line L such that half the points are on one side and half on the other side. 1 = Closest-Pair(left half) 2 = Closest-Pair(right half) = min(1, 2) Delete all points further than  from separation line L Sort remaining points by y-coordinate. Scan points in y-order and compare distance between each point and next 11 neighbors. If any of these distances is less than , update . return. } O(n log n) 2T(n / 2) O(n) O(n log n) O(n)

23. Closest Pair: PRAM Version? Closest-Pair(p1, …, pn) { Compute separation line L such that half the points are on one side and half on the other side. 1 = Closest-Pair(left half) 2 = Closest-Pair(right half) = min(1, 2) Delete all points further than  from separation line L Sort remaining points by y-coordinate. Scan points in y-order and compare distance between each point and next 11 neighbors. Find min of all these distances, update . return. } O(1) Use sorted lists In parallel T(n / 2) Use presorting and prefix computation. O(log n) O(1) O(log n) Again use prefix computation. Recurrence : T(n) = T(n/2) + O(log n)

24. Problem 6: Planar Convex hulls MergeHull (P) • HL = MergeHull( Left of median) • HR = MergeHull( Right of median) • Return JoinHulls(HL,HR) Time complexity in RAM? Time complexity in PRAM?

25. Join_Hulls

26. Towards a betterPlanar Convex hull • Let Q = {q1, q2, . . . , qn} be a set of points in the Euclidean plane (i.e., E2-space). • The convex hull of Q is denoted by CH(Q) and is the smallest convex polygon containing Q. • It is specified by listing its corner points (which are from Q) in order (e.g., clockwise order). • Usual Computational Geometry Assumptions: • No three points lie on the same straight line. • No two points have the same x or y coordinate. • There are at least 4 points, as CH(Q) = Q for n  3.

27. PRAM CONVEX HULL(n,Q, CH(Q)) • Sort the points of Q by x-coordinate. • Partition Q into k =n subsets Q1,Q2,. . . ,Qkof k points each such that a vertical line can separate Qi from Qj • Also, if i <j, then Qi is left of Qj. • For i = 1 to k , compute the convex hulls of Qi in parallel, as follows: • if |Qi|  3, then CH(Qi) = Qi • else (using k=n PEs) call PRAM CONVEX HULL(k, Qi, CH(Qi)) • Merge the convex hulls in {CH(Q1),CH(Q2), . . . ,CH(Qk)}together.

28. Basic Idea

29. Last Step • The upper hull is found first. Then, the lower hull is found next using the same method. • Only finding the upper hull is described here • Upper & lower convex hull points merged into ordered set • Each CH(Qi) has n PEs assigned to it. • The PEs assigned to CH(Qi) (in parallel) compute the upper tangent from CH(Qi) to another CH(Qj) . • A total of n-1 tangents are computed for each CH(Qi) • Details for computing the upper tangents will be separately

31. Last Step • Among the tangent lines to CH(Qi) , and polygons to the left of CH(Qi), let Libe the one with the smallest slope. • Among the tangent lines to CH(Qi) and polygons to the right, let Ribe the one with the largest slope. • If the angle between Liand Riis less than 180 degrees, no point of CH(Qi) is in CH(Q). • See Figure 5.13 on next slide (from Akl’s Online text) • Otherwise, all points in CH(Q)between where Litouches CH(Qi) and where Ritouches CH(Qi) are in CH(Q). • Array Packing is used to combine all convex hull points of CH(Q) after they are identified.

33. Complexity • Step 1: The sort takes O(lg n) time. • Step 2: Partition of Q into subsets takes O(1) time. • Step 3: The recursive calculations of CH(Qi) for 1  i n in parallel takes t(n) time (using n PEs for each Qi). • Step 4: The big steps here require O(lgn)and are • Finding the upper tangent from CH(Qi) to CH(Qj) for each i, j pair. • Array packing used to form the ordered sequence of upper convex hull points for Q. • Above steps find the upper convex hull. The lower convex hull is found similarly. • Upper & lower hulls merged in O(1) time to ordered set

34. Complexity • Cost for Step 3: Solving the recurrance relation t(n) = t(n) + lg n yields t(n) = O(lg n) • Running time for PRAM Convex Hull is O(lg n) since this is maximum cost for each step. • Then the cost for PRAM Convex Hull is C(n) = O(n lg n).