Computability and Complexity

1. Computability and Complexity 30-1 Parallel Computation Computability and Complexity Andrei Bulatov

2. Computability and Complexity processor1 processor2 processor3 processorn memory1 memory2 memory3 memoryn shared memory 30-2 Multiprocessor Computation Obviously, several communicating processors often can solve problems faster than a single processor …

3. Computability and Complexity 30-3 Assumptions • Exchanging messages takes no time (one step in the other model) • Writing to and reading from the memory takes no time and can be shared • We can use as many processors as we wish (for different instances of • the same problem different number of processors can be used), but this • number is polynomial in the size of the input • from the point of view of complexity theory, fixed number of processors • makes no sense — it gives only constant speed up - we need to reveal “true” parallelism in algorithms • an algorithm with unlimited number of processors can be “scaled” to an • algorithm with fixed number of processors

4. Computability and Complexity Given two matrices compute their product Recall that Straightforward computation of C can be done in Obviously, yes: use a separate processor for each of the multiplications (denote it ); then use to compute the sum for Totally, we need time 30-4 Example Can we do better with parallel computation?

5. Computability and Complexity We seek for exponential drop in the time required, that is an algorithm that works in or at least 30-5 We have obtained polynomial speed-up This is not sufficient: changing the model of sequential computation it is possible to get some polynomial speed-up

6. Computability and Complexity Thus, C can be computed in 30-6 Fast Algorithm for Matrix Multiplication O(log n) steps

7. Computability and Complexity We solved the Matrix Multiplication problem using processors in steps As we parallelized an algorithm of time complexity the total amount of work done by a parallel algorithm cannot be less than So the minimal number of processors required is 30-7 Complexity Measures These two numbers constitute the complexity measure for parallel algorithms: • number of processors • time complexity

8. Computability and Complexity • compute in log n “shifts” using processors at each shift • use the same processors to compute the first log log n • parallel additions Brent’s Principle If there is a parallelization, A, of a sequential algorithm that works in O(f(n)), such that the time complexity of A is g(n), then there is a parallelization, B, of the same algorithm such that B works in O(g(n)) and uses processors 30-8 Brent’s Principle Our algorithm can be improved so that this theoretical bound is achieved • complete computation as before Since the total number of steps is at most 2 log n, this algorithm works in O(log n)

9. Computability and Complexity Reachability Instance: A finite directed graphG and vertices s and t Question: Is there a path from s to t? 30-9 Graph Reachability The natural algorithm for Reachability (breadth-first search) cannot be parallelized in an obvious way Even if many processors are arranged to process nodes from the stack, we cannot stop before a path is found that requires at least as many steps as the length of the path, that is up to n – 1 It is suspected that both searches in graphs are inherently sequential

10. Computability and Complexity 1 4 3 2 What is ? Therefore, to solve Reachability for all possible pairs of vertices we need to compute This can be done in parallel steps with total work 30-10 Adjacency Matrix G if and only if there is a path from i to j of length 2