CS 515: Parallel Algorithms

1. 2010 - 02 -17 CS 515: Parallel Algorithms ChandrimaSarkar Atanu Roy

2. Agenda • Architecture • Parallel Programming Languages • Precedence Graph • Elementary Parallel Algorithms • Sorting • Matrix Multiplication Download :- http://www.cs.montana.edu/~atanu.roy/Classes/CS515.html

3. Architecture Flynn’s Classification S = single , M = multiple , I = instruction (stream), D = data (stream) SIMD SISD

4. Architecture Flynn’s Classification S = single , M = multiple , I = instruction (stream), D = data (stream) MISD MIMD

5. Linear Array Ring Ring arranged to use short wires Static Inter-connection Network Chordal ring Fully Connected Topology

6. Multidimensional Meshes and Torus Tree

7. Tree Cont. FAT TREE STAR

8. Hypercube 100 110 000 010 111 101 001 011 2-D 0-D 1-D 3-D 4-D 5-D

9. Parallel Programming Languages

10. Dijkstra’s High Level language construct • Degree of Parallelism is static Algol-68,CSP A parbegin C begin B parbegin D E parend G end parend H Precendence Graph

11. Elementary Parallel AlgorithmsFinding sum using a 2D mesh architecture

12. Finding sum of 16 values in a Shuffle Exchange SIMD Model

13. Parallel summation in a Hypercube SIMD Model

14. Broadcast in a Hypercube Algorithm 1 Algorithm 2

15. Odd Even Transposition Sort • (1) p = n • 14 – 5 – 15 – 8 – 4 – 11 – 13 – 12 • odd-even14 5 – 15 8 – 4 11 – 13 12 • even-odd14 – 5 15 – 4 8 – 11 13 – 12 • odd-even 5 14 – 4 15 – 8 11 – 12 13 • even-odd5 – 4 14 – 8 15 – 11 12 – 13 • odd-even 4 5 – 8 14 – 11 15 – 12 13 • even-odd4 – 5 8 – 11 14 – 12 15 – 13 • odd-even 4 5 – 8 11 – 12 14 – 13 15 • even-odd4 – 5 8 – 11 12 – 13 14 – 15

16. Odd Even Transposition Sort (contd…) • (2) p << n • S= {12, 7, 2, 4, 1, 11, 9, 5, 6, 3, 10, 8}, p = 4

17. Pseudocode • Proc MERGE-SPLIT(S) for i:= 1 to p do in parallel QUICKSORT(Si) end for for (i := 1 to ceil(p/2)) for odd-numbered processor do in parallel MERGE(Si , Si + 1) SPLIT end for for odd-numbered processor do in parallel MERGE(Si , Si + 1) SPLIT end for end for

18. 2 – D mesh with Snake Order Thompson and Kung (1977) Input : {23, 6, 1, 5, 11, 13, 55, 19, -3, 12, -5, -7, 9, 55, 28, -2}

19. Snake Order (contd.)

20. Bitonic Merge Sort • Bitonic Sequence :- 1, 3, 7, 8 6, 5, 4, 2 • Comparator • Note :- Batcher’s Bitonic Merge Sort compares elements whose indices differ by a single bit. + -

21. Bitonic Merge Sort

22. Shuffle-Exchange Network BitonicMergesort on Shuffle-Exchange Network • A list of n = 2k unsorted elements can be sorted in time θ(lg2 n) with a network 2k-1[k (k-1) + 1] comparators using the shuffle-exchange network.

23. Sorting Network

25. Odd Even Merging Network

26. Systolic Matrix Multiplication Multiply ai,kby ak,j Add the result to ri,j Send ai,kto cell ci+1,j Send bk,jto cell ci,j+1

28. Home Work • Show how the following 16 values would be sorted by Batcher’s Bitonic sort. 16, 7, 4, 12, 2, 10, 13, 9, 1, 8, 11, 3, 15, 6, 5, 14