How to Think Algorithmically in Parallel?

1. How to Think Algorithmically in Parallel? Uzi Vishkin

2. Commodity computer systems Chapter 1 19462003:Serial. 5KHz4GHz. Chapter 2 2004--: Parallel. #”cores”:~dy-2003 Apple 2004: 1 core 2008: 8 cores 2013: ?? cores Windows 7: scales to 256 cores… how to use the remaining 255? Is this the role of the OS? BIG NEWS Clock frequency growth: flat. If you want your program to run significantly faster … you’re going to have to parallelize it Parallelism: only game in town Since 1980: #Transistors/chip 29K~10sB! Bandwidth/Latency 300X [HP12] Programmer’s IQ? Flat.. 40 years of parallel computing The world is yet to see a successful general-purpose parallel computer: Easy to program & good speedups Intel Platform 2015, March05

3. How is many-core parallel computing doing? • Current-day system architectures allow good speedups on regular dense-matrix type programs, but are basically unable to do anything outside that What’s missing - Irregular problems/programs - Cost-effective programming for regular problems Sweat-to-gain ratio is (often too) high Though some progress with domain-specific languages - Strong scaling Missing items require revolutionary approach

4. Current systems/revolutionary changes Multiprocessors HP-12: Computer consisting of tightly coupled processors whose coordination and usage are controlled by a single OS and that share memory through a shared address space GPUsHW handles thread management. But, leave open missing items BACKUP: • Goal Fit as many FUs as you can into silicon. Now, use all of them all the time • Architecture, including memory, optimized for peak performance on limited workloads, rather than sustained general-purpose performance • Each thread is SIMD  limit on thread divergence (both sides of a branch) • HW uses parallelism for FUs and hiding memory latency • No: shared cache for general data, or truly all-to-all interconnection network to shared memory  Works well for plenty of “structured” parallelism • Minimal parallelism: just to break even with serial  • Cannot handle serial & low-parallel code.Leave open missing items: strong scaling, irregular, cost-effective regular Also: DARPA-HProductivityCS but still: “most programmers cannot exploit the vast parallelism in today’s machines” [GameOver’11] Revolutionary   high bar: Throw out what we have and replace it

5. Hardware-first threads Place holder OPINION Build-first, figure-out-how-to-program later  architecture Graphics cards Where to start so that GPUs.CUDA. GPGPU Parallel programming: MPI, Open MP ν Dense-matrix-type X Irregular,Cost-effective,Strong scaling ν Past Future? Heterogeneous  lowering the bar: Keep what we have, but augment it. Enabled by: increasing transistor budget, 3D VLSI & design of power Heterogeneous system

6. Hardware-first threads Algorithms-first thread OPINION Build-first, figure-out-how-to-program later  architecture Graphics cards How to think about parallelism? PRAM & Parallel algorithms Concept NYU-Ultracomputer? SB-PRAM, XMT Many-core.Quantitative XMT GPUs.CUDA. GPGPU Parallel programming: MPI, Open MP ν Dense-matrix-type X Irregular,Cost-effective,Strong scaling Fine, but more important: ν Past Future? Heterogeneous system

7. What about the missing items ? OPINION Evidence-based opinion SW alone is not enough.FeasibleOrders of magnitude better with different hardware. Evidence Broad portfolio; e.g., most advanced parallel algorithms; high-school students do PhD-thesis level work Who should care? - DARPA Opportunity for competitors to surprise the US military and economy - Vendors Confluence of mobile & wall-plugged processor market creates unprecedented competition. Quad-cores and architecture technique reached plateau. No other way to get significantly ahead But, - Chicken-and-egg effect Few end-user apps use missing items (since..missing) - My guess Under water, the “end-user application iceberg” is much larger than today’s parallel end-user applications. • Supporting evidence • Irregular problems: many and rising. Data compression. Computer Vision. Bio-related. Sparse scientific. Sparse sensing & recovery. EDA • In CS most algorithms we teach are irregular. How come that parallel ones have a different breakdown? Heard: so we teach the wrong things Can such ideas gain traction? Mental exercise on: • Naive answer: “Sure, since they can”. So, why not in the past? • Wall Street companies: risk averse. Too big for startup • Focus on fighting out GPUs (only competition) • Why not?

8. My conclusion OPINION A time bomb that will explode sooner or later Will take over domination of a core area of IT. How much more?

9. Example of a problem to be discussed:1 or 2 Paradigm Shifts? • Serial to parallel: widely agreed • Within parallel: Existing “decomposition-first” paradigms. Painful to program. Will there be a switch to a different (easier-to-program) paradigm?

10. Abstractions in CS • Any particular word of an indefinitely large memory is immediately available • A uniprocessor is serving the task that the user is currently working on exclusively. (i) abstracts away a hierarchy of memories, each has greater capacity, but slower access time, than the preceding one. (ii) abstracts way: virtual file systems that can be implemented in local storage or a local or global network, the (whole) web, and other tasks that may be concurrently using the same computer system. These abstractions have improved productivity of programmers and other users, and contributed towards broadening participation in computing. • The proposed addition to this consensus is as follows. That an indefinitely large number of operations available for concurrent execution executes immediately.

11. Who should produce the parallel code? Thanks: Prof. Barua Choices [state-of-the-art compiler research perspective] • Programmer only • Writing parallel code is tedious. • Good at ‘seeing parallelism’, esp. irregular parallelism. • But are bad at seeing locality and granularity considerations. • Have poor intuitions about compiler transformations. • Compiler only • Can see regular parallelism, but not irregular parallelism. • Great at doing compiler transformations to improve parallelism, granularity and locality.  Hybrid solution: Programmer specifies high-level parallelism, but little else. Compiler does the rest. Goals: • Ease of programming • Declarative programming (My) Broader questions Where will the algorithms come from? Is today’s HW good enough? This course relevant for all 3 questions

12. PreviewSerial versus parallel algorithmic thinking Serial RAM Step: 1 op (memory/etc). PRAM (Parallel Random-Access Model) Step: many ops. Serial doctrine Natural (parallel) algorithm time = #ops time << #ops What could I do in parallel at each step assuming unlimited hardware  . . # ops . . # ops . . .. .. .. .. time time

13. Saw this&BFS in 1st classFlavor of parallelism Exchange Problem Replace A and B. Ex. A=2,B=5A=5,B=2. Serial Alg: X:=A;A:=B;B:=X. 3 Ops. 3 Steps. Space 1. Fewer steps (FS): X:=A B:=X Y:=B A:=Y 4 ops. 2 Steps. Space 2. Array Exchange Problem Given A[1..n] & B[1..n], replace A(i) and B(i), i=1..n. Serial Alg: For i=1 to n do X:=A(i);A(i):=B(i);B(i):=X /*serial replace 3n Ops. 3n Steps. Space 1. Par Alg1: For i=1 to n pardo X(i):=A(i);A(i):=B(i);B(i):=X(i) /*serial replace in parallel 3n Ops. 3 Steps. Space n. Par Alg2: For i=1 to n pardo X(i):=A(i) B(i):=X(i) Y(i):=B(i) A(i):=Y(i) /*FS in parallel 4n Ops. 2 Steps. Space 2n. Discussion Parallelism requires extra space (memory). Par Alg 1 clearly faster than Serial Alg. Is Par Alg 2 preferred to Par Alg 1?

14. Example of Parallel algorithm Breadth-First-Search (BFS)

17. (i) “Concurrently” as in natural BFS: only change to serial algorithm (ii) Defies “decomposition”/”partition” Parallel complexity W = ~(|V| + |E|) T = ~d, the number of layers Average parallelism = ~W/T Mental effort 1. Sometimes easier than serial 2. Within common denominator of other parallel approaches. In fact, much easier

18. A-propos BFS [EduPar2011] 2011 NSF/IEEE-TCPP curriculum teach BFS using OpenMP Teaching experiment Joint F2010 UIUC/UMD class. 42 students Good news Easy coding (since no meaningful ‘decomposition’) Bad newsNone got speedup over serial on 8-proc SMP machine BFS alg was easy but .. no good: no speedups Speedups on 64-processor XMT 7x to 25x Fair to compare 64 processors to 8 since <1/4 of the silicon area Symptom of the bigger denial ‘Only problem Developers lack parallel programming skills’ Solution Education. False Teach then see that HW is the problem. HotPAR10 performance results include BFS: XMT/GPU Speed-up same silicon area, highly parallel input: 5.4X Small HW configuration, large diameter: 109X wrt same GPU

19. Discussion of BFS results • Contrast with smartest people, Stanford’11, Nvidia’12 .. BFS on multi-cores/GPUs, again only if the diameter is small, improving on a SC’10 IBM/GaTech and 6 other recent papers, all 1st rate conferences. BFS is bread & butter. Call the Marines each time you need bread? • ‘Decree’ Random graphs = ‘reality’. In the old days: Expander graphs taught in graph design. Planar graphs were real • Lots of parallelism  more HW design freedom. E.g., GPUs get decent speedup with lots of parallelism. But, not enough for general parallel algorithms. BFS (& max-flow): better speedups and easier programs on XMT

20. Experience with High School Students, Fall’07 1-day parallel algorithms tutorial to 12 HS students. Some (2 10th graders) managed 8 programming assignments, including 5 of the 6 in the grad course. Only help: 1 office hour/week by undergrad TA. No school credit. Part of a computer club after 8 periods/day. One of these 10th graders: “I tried to work on parallel machines at school, but it was no fun: I had to program around their engineering. With XMT, I could focus on solving the problem that I had to solve.”

21. Parallel Random-Access Machine/Model PRAM: • n synchronous processors all having unit time access to a shared memory. • Each processor has also a local memory. • At each time unit, a processor can: • write into the shared memory (i.e., copy one of its local memory registers into • a shared memory cell), • 2. read into shared memory (i.e., copy a shared memory cell into one of its local • memory registers ), or • 3. do some computation with respect to its local memory.

22. pardo programming construct - for Pi , 1 ≤ i ≤ n pardo - A(i) := B(i) This means The following n operations are performed concurrently: processor P1 assigns B(1) into A(1), processor P2 assigns B(2) into A(2), …. Modeling read&write conflicts to the same shared memory location Most common are: - exclusive-read exclusive-write (EREW) PRAM: no simultaneous access by more than one processor to the same memory location for read or write purposes • concurrent-read exclusive-write (CREW) PRAM: concurrent access for reads but not for writes • concurrent-read concurrent-write (CRCW allows concurrent access for both reads and writes. We shall assume that in a concurrent-write model, an arbitrary processor among the processors attempting to write into a common memory location, succeeds. This is called the Arbitrary CRCW rule. There are two alternative CRCW rules: (i) Priority CRCW: the smallest numbered, among the processors attempting to write into a common memory location, actually succeeds. (ii) Common CRCW: allows concurrent writes only when all the processors attempting to write into a common memory location are trying to write the same value.

23. Example of a PRAM algorithm: The summation problem Input An array A = A(1) . . .A(n) of n numbers. The problem is to compute A(1) + . . . + A(n). The summation algorithm works in rounds. Each round: add, in parallel, pairs of elements: add each odd-numbered element and its successive even-numbered element. If n = 8, outcome of 1st round is: A(1) + A(2), A(3) + A(4), A(5) + A(6), A(7) + A(8) Outcome of 2nd round: A(1) + A(2) + A(3) + A(4), A(5) + A(6) + A(7) + A(8) and the outcome of 3rd (and last) round: A(1) + A(2) + A(3) + A(4) + A(5) + A(6) + A(7) + A(8) B – 2-dimensional array (whose entries are B(h,i), 0 ≤ h ≤ log n and 1 ≤ i ≤ n/2h) used to store all intermediate steps of the computation (base of logarithm: 2). For simplicity, assume n = 2k for some integer k. ALGORITHM 1 (Summation) 1. for Pi , 1 ≤ i ≤ n pardo 2. B(0, i) := A(i) 3. for h := 1 to log n do 4. if i ≤ n/2h 5. then B(h, i) := B(h − 1, 2i − 1) + B(h − 1, 2i) 6. else stay idle 7. for i = 1: output B(log n, 1); for i > 1: stay idle Algorithm 1 uses p = n processors. Line 2 takes one round, Line 3 defines a loop taking log n rounds Line 7 takes one round.

24. Summation on an n = 8 processor PRAM Again Algorithm 1 uses p = n processors. Line 2 takes one round, line 3 defines a loop taking log n rounds, and line 7 takes one round. Since each round takes constant time, Algorithm 1 runs in O(log n) time. [When you see O (“big Oh”), think “proportional to”.] So, an algorithm in the PRAM model is presented in terms of a sequence of parallel time units (or “rounds”, or “pulses”); we allow p instructions to be performed at each time unit, one per processor; this means that a time unit consists of a sequence of exactly p instructions to be performed concurrently.

25. 2 drawbacks to PRAM mode: (i) Does not reveal how the algorithm will run on PRAMs with different number of processors; e.g., to what extent will more processors speed the computation, or fewer processors slow it? (ii) Fully specifying the allocation of instructions to processors requires a level of detail which might be unnecessary (a compiler may be able to extract from lesser detail) Work-Depth presentation of algorithms Alternative model and presentation mode. Work-Depth algorithms are also presented as a sequence of parallel time units (or “rounds”, or “pulses”); however, each time unit consists of a sequence of instructions to be performed concurrently; the sequence of instructions may include any number.

26. WD presentation of the summation example “Greedy-parallelism”: At each point in time, the (WD) summation algorithm seeks to break the problem into as many pair wise additions as possible, or, in other words, into the largest possible number of independent tasks that can performed concurrently. ALGORITHM 2 (WD-Summation) 1. for i , 1 ≤ i ≤ n pardo 2. B(0, i) := A(i) 3. for h := 1 to log n 4. for i , 1 ≤ i ≤ n/2h pardo 5. B(h, i) := B(h − 1, 2i − 1) + B(h − 1, 2i) 6. for i = 1 pardo output B(log n, 1) The 1st round of the algorithm (lines 1&2) has n operations. The 2nd round (lines 4&5 for h = 1) has n/2 operations. The 3rd round (lines 4&5 for h = 2) has n/4 operations. In general, the k-th round of the algorithm, 1 ≤ k ≤ log n + 1, has n/2k-1 operations and round log n +2 (line 6) has one more operation (use of a pardo instruction in line 6 is somewhat artificial). The total number of operations is 2n and the time is log n + 2. We will use this information in the corollary below. The next theorem demonstrates that the WD presentation mode does not suffer from the same drawbacks as the standard PRAM mode, and that every algorithm in the WD mode can be automatically translated into a PRAM algorithm.

27. The WD-presentation sufficiency Theorem Consider an algorithm in the WD mode that takes a total of x = x(n) elementary operations and d = d(n) time. The algorithm can be implemented by any p = p(n)-processor PRAM within O(x/p + d) time, using the same concurrent-write convention as in the WD presentation. [i.e., 5 theorems: EREW, CREW, Common/Arbitrary/Priority CRCW] Proof xi - # instructions at round i. [x1+x2+..+xd = x] p processors can simulate xiinstructions in ⌈xi/p⌉≤ xi/p + 1 time units. See next slide. Demonstration in Algorithm 2’ shows why you don’t want to leave this to a programmer. Formally: first reads, then writes. Theorem follows, since ⌈x1/p⌉+⌈x2/p⌉+..+⌈xd/p⌉≤ (x1/p +1)+..+(xd/p +1) ≤ x/p + d

28. Round-robin emulation of y concurrent instructions by p processors in ⌈y/p⌉ rounds. In each of the first ⌈y/p⌉ −1 rounds, p instructions are emulated for a total of z = p(⌈y/p⌉ − 1) instructions. In round ⌈y/p⌉, the remaining y − z instructions are emulated, each by a processor, while the remaining w − y processor stay idle, where w = p⌈y/p⌉

29. Corollary for summation example Algorithm 2 would run in O(n/p + log n) time on a p-processor PRAM. For p ≤ n/ log n, this implies O(n/p) time. Later called both optimal speedup & linear speedup For p ≥ n/ log n: O(log n) time. Since no concurrent reads or writes  p-processor EREW PRAM algorithm.

30. ALGORITHM 2’ (Summation on a p-processor PRAM) 1. for Pi , 1 ≤ i ≤ p pardo 2. for j := 1 to ⌈n/p⌉ − 1 do - B(0, i + (j − 1)p) := A(i + (j − 1)p) 3. for i , 1 ≤ i ≤ n − (⌈n/p⌉ − 1)p - B(0, i + (⌈n/p⌉ − 1)p) := A(i + (⌈n/p⌉ − 1)p) - for i , n − (⌈n/p⌉ − 1)p ≤ i ≤ p - stay idle 4. for h := 1 to log n 5. for j := 1 to ⌈n/(2hp)⌉ − 1 do (*an instruction j := 1 to 0 do means: - “do nothing”*) • B(h, i+(j −1)p) := B(h−1, 2(i+(j −1)p)−1) + B(h−1, 2(i+(j −1)p)) 6. for i , 1 ≤ i ≤ n − (⌈n/(2hp)⌉ − 1)p - B(h, i + (⌈n/(2hp)⌉ − 1)p) := B(h − 1, 2(i + (⌈n/(2hp)⌉ − 1)p) − 1) + - B(h − 1, 2(i + (⌈n/(2hp)⌉ − 1)p)) - for i , n − (⌈n/(2hp)⌉ − 1)p ≤ i ≤ p - stay idle • for i = 1 output B(log n, 1); for i > 1 stay idle Nothing more than plugging in the above proof. Main point of this slide: compare to Algorithm 2 and decide, which one you like better But is WD mode as easy as it gets? Hold on…Key question for this presentation

31. Measuring the performance of parallel algorithms A problem. Input size: n. A parallel algorithm in WD mode. Worst case time: T(n); work: W(n). 4 alternative ways to measure performance: 1. W(n) operations and T(n) time. 2. P(n) = W(n)/T(n) processors and T(n) time (on a PRAM). 3. W(n)/p time using any number of p ≤ W(n)/T(n) processors (on a PRAM). 4. W(n)/p + T(n) time using any number of p processors (on a PRAM). Exercise 1: The above four ways for measuring performance of a parallel algorithms form six pairs. Prove that the pairs are all asymptotically equivalent.

32. Goals for Designers of Parallel Algorithms Suppose 2 parallel algorithms for same problem: 1. W1(n) operations in T1(n) time. 2. W2(n) operations, T2(n) time. General guideline: algorithm 1 more efficient than algorithm 2 if W1(n) = o(W2(n)), regardless of T1(n) and T2(n); if W1(n) and W2(n) grow asymptotically the same, then algorithm 1 is considered more efficient if T1(n) = o(T2(n)). Good reasons for avoiding strict formal definition—only guidelines ExampleW1(n)=O(n),T1(n)=O(n); W2(n)=O(n log n),T2(n)=O(log n) Which algorithm is more efficient? Algorithm 1: less work. Algorithm 2: much faster. In this case, both algorithms are probably interesting. Imagine two users, each interested in different input sizes and in different target machines (different # processors). For one user Algorithm 1 faster. For second user Algorithm 2 faster. Known unresolved issues with asymptotic worst-case analysis.

33. Nicknaming speedups Suppose T(n) best possible worst case time upper bound on serial algorithm for an input of length n for some problem. (T(n) is serial time complexity for problem.) Let W(n) and Tpar(n) be work and time bounds of a parallel algorithm for same problem. The parallel algorithm is work-optimal, if W(n) grows asymptotically the same as T(n). A work-optimal parallel algorithm is work-time-optimal if its running time T(n) cannot be improved by another work-optimal algorithm. What if serial complexity of a problem is unknown? Still an accomplishment if T(n) is best known and W(n) matches it. Called linear speedup. Note: can change if serial improves. Recall main reasons for existence of parallel computing: - Can perform better than serial - (it is just a matter of time till) Serial cannot improve anymore

34. Default assumption regarding shared memory access resolution Since all conventions represent virtual models of real machines: strongest model whose implementation cost is “still not very high”, would be practical. Simulations results + UMD PRAM-On-Chip architecture • Arbitrary CRCW NC Theory Good serial algorithms: poly time. Good parallel algorithm: poly-log time, poly processors. Was much more dominant than what’s covered here in early 1980s. Fundamental insights. Limited practicality. In choosing abstractions: fine line between helpful and “defying gravity”

35. Technique: Balanced Binary Trees; Problem: Prefix-Sums Input: Array A[1..n] of elements. Associative binary operation, denoted ∗, defined on the set: a ∗ (b ∗ c) = (a ∗ b) ∗ c. (∗ pronounced “star”; often “sum”: addition, a common example.) The n prefix-sums of array A are: A(1) A(1) ∗ A(2) .. A(1) ∗ A(2) ∗ .. ∗ A(i) .. A(1) ∗ A(2) ∗ .. ∗ A(n) Prefix-sums is perhaps the most heavily used routine in parallel algorithms.

36. ALGORITHM 1 (Prefix-sums) 1. for i , 1 ≤ i ≤ n pardo - B(0, i) := A(i) 2. for h := 1 to log n 3. for i , 1 ≤ i ≤ n/2h pardo - B(h, i) := B(h − 1, 2i − 1) ∗ B(h − 1, 2i) 4. for h := log n to 0 5. for i even, 1 ≤ i ≤ n/2h pardo - C(h, i) := C(h + 1, i/2) 6. for i = 1 pardo - C(h, 1) := B(h, 1) 7. for i odd, 3 ≤ i ≤ n/2h pardo - C(h, i) := C(h + 1, (i − 1)/2) ∗ B(h, i) 8. for i , 1 ≤ i ≤ n pardo - Output C(0, i) } Summation (as before) } C(h,i) – prefix-sum of rightmost leaf of [h,i]

37. Prefix-sums algorithm Example • Complexity Charge operations to nodes. Tree has 2n-1 nodes. • No node is charged with more than O(1) operations. • W(n) = O(n). Also T(n) = O(log n) Theorem: The prefix-sums algorithm runs in O(n) work and O(log n) time.

38. Application - the Compaction Problem The Prefix-sums routine is heavily used in parallel algorithms. A trivial application follows: Input Array A = A[1. . N] of elements, and binary array B = B[1 . . n]. Map each value i, 1 ≤ i ≤ n, where B(i) = 1, to the sequence (1, 2, . . . , s); s is the (a priori unknown) numbers of ones in B. Copy the elements of A accordingly. The solution is order preserving. But, quite a few applications of compaction do not require that. For computing the mapping, simply find prefix sums with respect to array B. Consider an entry B(i) = 1. If the prefix sum of i is j then map A(i) into C(j). Theorem The compaction algorithm runs in O(n) work and O(log n) time.

39. Snapshot: XMT High-level language(same as earlier slide) XMTC: Single-program multiple-data (SPMD) extension of standard C. Includes Spawn and PS - a multi-operand instruction. Short (not OS) threads. Cartoon Spawn creates threads; a thread progresses at its own speed and expires at its Join. Synchronization: only at the Joins. So, virtual threads avoid busy-waits by expiring. New: Independence of order semantics (IOS).

40. XMT High-level language (cont’d) A D The array compaction problem Input: A[1..n]. Map in some order all A(i) not equal 0 to array D. Essence of an XMT-C program int x = 0; /*formally: psBaseReg x=0*/ spawn(0, n-1) /* Spawn n threads; $ ranges 0 to n − 1 */ { int e = 1; if (A[$] not-equal 0) { ps(e,x); D[e] = A[$] } } n = x; Notes: (i) PS is defined next (think F&A). See results for e0,e2, e6 and x. (ii) Join instructions are implicit. e0 e2 e6 e$ local to thread $; x is 3

41. XMT Assembly Language Standard assembly language, plus 3 new instructions: Spawn, Join, and PS. The PS multi-operand instruction New kind of instruction: Prefix-sum (PS). Individual PS, PS Ri Rj, has an inseparable (“atomic”) outcome: • Store Ri + Rj in Ri, and (ii) store original value of Ri in Rj. Several successive PS instructions define a multiple-PS instruction. E.g., the sequence of k instructions: PS R1 R2; PS R1 R3; ...; PS R1 R(k + 1) performs the prefix-sum of base R1 elements R2,R3, ...,R(k + 1) to get: R2 = R1; R3 = R1 + R2; ...; R(k + 1) = R1 + ... + Rk; R1 = R1 + ... + R(k + 1). Idea: (i) Several ind. PS’s can be combined into one multi-operand instruction. (ii) Executed by a new multi-operand PS functional unit.

42. Mapping PRAM Algorithms onto XMT(1st visit of this slide) (1) PRAM parallelism maps into a thread structure (2) Assembly language threads are not-too-short (to increase locality of reference) (3) the threads satisfy IOS How (summary): • Use work-depth methodology [SV-82] for “thinking in parallel”. The rest is skill. • Go through PRAM or not. For performance-tuning, in order to later teach the compiler. (To be suppressed as it is ideally done by compiler): Produce XMTC program accounting also for: (1) Length of sequence of round trips to memory, (2) QRQW. Issue: nesting of spawns.

43. Workflow from parallel algorithms to programming versus trial-and-error Option 2 Option 1 Domain decomposition, or task decomposition PAT Parallel algorithmic thinking (ICE/WD/PRAM) PAT Prove correctness Program Program Insufficient inter-thread bandwidth? Still correct Rethink algorithm: Take better advantage of cache Tune Compiler Still correct Hardware Hardware Is Option 1 good enough for the parallel programmer’s model? Options 1B and 2 start with a PRAM algorithm, but not option 1A. Options 1A and 2 represent workflow, but not option 1B. Not possible in the 1990s. Possible now: XMT@UMD Why settle for less?

44. Exercise 2 Let A be a memory address in the shared memory of a PRAM. Suppose all p processors of the PRAM need to “know” the value stored in A. Give a fast EREW algorithm for broadcasting A to all p processors. How much time will this take? Exercise 3 Input: An array A of n elements drawn from some totally ordered set. The minimum problem is to find the smallest element in array A. (1) Give an EREW PRAM algorithm that runs in O(n) work and O(log n) time. (2) Suppose we are given only p ≤ n/ log n processors numbered from 1 to p. For the algorithm of (1) above, describe the algorithm to be executed by processor i, 1 ≤ i ≤ p. The prefix-min problem has the same input as for the minimum problem and we need to find for each i, 1 ≤ i ≤ n, the smallest element among A(1),A(2), . . . ,A(i). (3) Give an EREW PRAM algorithm that runs in O(n) work and O(log n) time for the problem. Exercise 4 The nearest-one problem is defined as follows. Input: An array A of size n of bits; namely, the value of each entry of A is either 0 or 1. The nearest-one problem is to find for each i, 1 ≤ i ≤ n, the largest index j ≤ i, such that A(j) = 1. (1) Give an EREW PRAM algorithm that runs in O(n) work and O(log n) time. The input for the segmented prefix-sums problem, includes the same binary array A as above, and in addition an array B of size n of numbers. The segmented prefix-sums problem is to find for each i, 1 ≤ i ≤ n, the sum B(j) + B(j + 1) + . . . + B(i), where j is the nearest-one for i (if i has no nearest-one we define its nearest-one to be 1). (2) Give an EREWPRAM algorithm for the problem that runs in O(n) work and O(log n) time.

45. Recursive Presentation of the Prefix-Sums Algorithm Recursive presentations are useful for describing both serial and parallel algorithms. Sometimes they shed new light on a technique being used. PREFIX-SUMS(x1, x2, . . . , xm; u1, u2, . . . , um) 1. if m = 1 then u1 := x1; exit 2. for i, 1 ≤ i ≤ m/2 pardo - yi := x2i−1 ∗ x2i 3. PREFIX-SUMS(y1, y2, . . . , ym/2; v1, v2, . . . , vm/2) 4. for i even, 1 ≤ i ≤ m pardo - ui := vi/2 5. for i = 1 pardo - u1 := x1 6. for i odd, 3 ≤ i ≤ m pardo - ui := v(i−1)/2 ∗ xi To start, call: PREFIX-SUMS(A(1),A(2), . . . ,A(n);C(0, 1),C(0, 2), . . . ,C(0, n)). Complexity Recursive presentation can give concise and elegant complexity analysis. Excluding the recursive call in instruction 3, routine PREFIX-SUMS, requires: ≤ α time, and ≤ βm operations for some positive constants α and β. The recursive call is for a problem of size m/2. Therefore, T(n) ≤ T(n/2) + α W(n) ≤ W(n/2) + βn Their solutions are T(n) = O(log n), and W(n) = O(n).

46. Exercise 5: Multiplying two n × n matrices A and B results in another n × n matrix C, whose elements ci,j satisfy ci,j = ai,1b1,j + ..+ ai,kbk,j + ..+ ai,nbn,j. (1) Given two such matrices A and B, show how to compute matrix C in O(log n) time using n3 processors. (2) Suppose we are given only p ≤ n3 processors, which are numbered from 1 to p. Describe the algorithm of item (1) above to be executed by processor i, 1 ≤ i ≤ p. (3) In case your algorithm for item (1) above required more than O(n3) work, show how to improve its work complexity to get matrix C in O(n3) work and O(log n) time. (4) Suppose we are given only p ≤ n3/ log n processors numbered from 1 to p. Describe the algorithm for item (3) above to be executed by processor i, 1 ≤ i ≤ p.

47. Merge-Sort Input: Two arrays A[1. . n], B[1. . m]; elements from a totally ordered domain S. Each array is monotonically non-decreasing. Merging: map each of these elements into a monotonically non-decreasing array C[1..n+m] The partitioning paradigm n: input size for a problem. Design a 2-stage parallel algorithm: • Partition the input into a large number, say p, of independent small jobs AND size of the largest small job is roughly n/p. • Actual work - do the small jobs concurrently, using a separate (possibly serial) algorithm for each. Ranking Problem Input: Same as for merging. For every 1<=i<= n, RANK(i,B), and 1<=j<=m, RANK(j,A) Example: A=[1,3,5,7,9],B[2,4,6,8]. RANK(3,B)=2;RANK(1,A)=1

48. Merging algorithm (cnt’d) Observe Merging & Ranking: really same problem. Show MR in W=O(n),T=O(1) (say n=m): C(k)=A(i)  RANK(i,B)=k-i-1 Show RM in W=O(n),T=O(1): RANK(i,B)=jC(i+j+1)=A(i) “Surplus-log” parallel algorithm for the Ranking for 1 ≤ i ≤ n pardo • Compute RANK(i,B) using standard binary search • Compute RANK(i,A) using binary search Complexity: W=(O(n log n), T=O(log n)

49. Serial (ranking) algorithm SERIAL − RANK(A[1 . . ];B[1. .]) i := 0 and j := 0; add two auxiliary elements A(n+1) and B(n+1), each larger than both A(n) and B(n) while i ≤ n or j ≤ n do • if A(i + 1) < B(j + 1) • then RANK(i+1,B) := j; i := i + 1 • else RANK(j+1),A) := i; j := j + 1 In words Starting from A(1) and B(1), in each round: • compare an element from A with an element of B • determine the rank of the smaller among them Complexity: O(n) time (and O(n) work...)

50. Linear work parallel merging Partitioningfor 1 ≤ i ≤ n/p pardo [p <= n/log and p | n] • b(i):=RANK(p(i-1) + 1),B) using binary search • a(i):=RANK(p(i-1) + 1),A) using binary search Actual work Observe Ranking task can be broken into 2p independent “slices”. Example of a slice Start at A(p(i-1) +1) and B(b(i)). Using serial ranking advance till: Termination condition Either A(pi+1) or some B(jp+1) loses Parallel algorithm 2p concurrent threads