A Pattern Language for Parallelizing Irregular Algorithms

1. A Pattern Language for Parallelizing Irregular Algorithms Pedro and Miguel Monteiro Universidade Nova de Lisboa

2. A lot of ideas come from the Ph.D. Dissertation of Milind V. Kulkarni (Cornell 2008) THE GALOIS SYSTEM: OPTIMISTIC PARALLELIZATION OF IRREGULAR PROGRAMS Also: Amorphous Data-parallelism in Irregular Algorithms Keshav Pingali, Milind Kulkarni, Donald Nguyen, Martin Burtscher, Mario Mendez-Lojo, Dimitrios Prountzos, Xin Sui and Zifei Zhong The University of Texas at Austin, Dept of Comp Sci Report# TR-09-05 February, 2009

3. Irregular Algorithm Unpredictability of Data Dependencies MOST GENERAL FORM: foreach element in dataStructure do dataElements = // identify neighbors lock dataElements; compute(dataStructure.get(dataElements)); unlock dataElements; endForeach Create a Taxonomy

4. KULKARNI DEFINITION: • for each Element e in Set S { B(e) } The loop body B(e) is executed for each element e of Set S. Since set elements are not ordered, this construct asserts that in a serial execution of the loop, the iterations can be executed in any order. There may be dependences between the iterations, but any serial order of executing iterations is permitted. When an iteration executes, it may add elements to S. • for each Element e in OrderedSet S { B(e) } This construct iterates over an ordered set S of active elements. It is similar to the Set iterator above, except that the execution order must respect the order on elements of S.

6. Data-Parallel Graph “The focus of this pattern is not to provide specific implementation solutions, merely to allow us to understand how graph characteristics influence irregular parallel problems.” Paper proposes a set of steps for decisions that need to be made, but they are all abstract

7. Optimistic Iteration Like speculative instruction execution

8. General optimistic implementation of a irregular algorithm. Graph graph; Worker worker; //worker thread Scheduler scheduler; while (true){ try{ Iteration it; it = scheduler.newIteration(worker); scheduler.nextElement(it); <result,work> = compute(graph,it); graph.replaceSubgraph(it, result); if(work.isNotNull()) scheduler.addWork(work); scheduler.commitIteration(it); }catch (violationException ve){ //do nothing //graph is only updated on commit } //check for termination }

9. In-order Iteration When there are some ordering constraints on data Don't want to reduce to sequential algorithm Put some knowledge of ordering constraints into the Optimistic Iteration engine so rollbacks are avoided where possible

10. KULKARNI TAXONOMY:

11. How to exploit structure: • Zero-copy implementation • Cautious operator, unordered • Data partitioning • Scheduling • Run-time • Just-in-time (compute once, reuse) • Compile-time

12. “Synthetic” approach, rather than a taxonomy. • Compare to • Mark Snir • Berkeley “dwarfs”