Design Patterns for Parallel Programming

1. Design Patterns for Parallel Programming Research work by:Kurt Keutzer, EECS,UC Berkeley Tim Mattson, Intel Corporation Presented to faculty at Tsinghua Multicore Workshop Michael Wrinn, Intel Corporation

2. Outline • A Programming Pattern Language • Motivation – patterns: why and what? • Structural patterns • Computational patterns • Composing patterns: examples • Concurrency patterns and PLPP • Examples: pattern language case studies • Examples: pattern language in Smoke Demo

3. Outline • Motivation – what is the problem we are trying to solve • How do programmers think? Psychology meets computer science. • Pattern Language for Parallel programming • Toy problems showing some key PLPP patterns • A PLPP example (molecular dynamics) • Expanding the effort: Patterns for engineering parallel software • A Survey of some key patterns • Case studies

4. Microprocessor trends Many core processors are the “new normal”. 160 cores 80 cores 240 cores ATI RV770 1 CPU + 6 cores NVIDIA Tesla C1060 Intel Terascale research chip IBM Cell 3rd party names are the property of their owners.

5. The State of the field • A harsh assessment … • We have turned to multi-core chips not because of the success of our parallel software but because of our failure to continually increase CPU frequency. • Result: a fundamental and dangerous (for the computer industry) mismatch • Parallel hardware is ubiquitous. • Parallel software is rare The multi-core Challenge: How do we make Parallel software as ubiquitous as parallel hardware? This could be the greatest challenge ever faced by the computer industry

6. We need to create a new generation of parallel programmers We need to focus on the needs of the programmer, not the needs of the computer • Parallel Programming is difficult, error prone and only accessible to small cadre of experts … Clearly we haven’t been doing a very good job at educating programmers. • Consider the following: • All known programmers are human beings. • Hence, human psychology, not hardware design, should guide our work on this problem.

7. Outline • Motivation – what is the problem we are trying to solve • How do programmers think? Psychology meets computer science. • Pattern Language for Parallel programming • Toy problems showing some key PLPP patterns • A PLPP example (molecular dynamics) • Expanding the effort: Patterns for engineering parallel software • A Survey of some key patterns • Case studies

8. Cognitive Psychology and human reasoning • Human Beings are model builders • We build hierarchical complexes of mental models. • Understand sensory input in terms of these models. • When input conflicts with models, we tend to believe the models. Consider the following slide …

10. Why did most of you see motion? • Your brain’s visual system contains a model that says this combination of geometric shapes and colors implies motion. • Your brain believes the model and not what your eyes see.. To understand people and how we work, you need understand the models we work with.

11. Programming and models • Programming is a process of successive refinement of a problem over a hierarchy of models. [Brooks83] • The models represent the problem at a different level of abstraction. • The top levels express the problem in the original problem domain. • The lower levels represent the problem in the computer’s domain. • The models are informal, but detailed enough to support simulation.

12. Models Domain Problem Specific: polygons, rays, molecules, etc. Specification OpenMP’s fork/join, Actors Programming Threads – shared memory Processes – shared nothing Computation Machine (AKA Cost Model) Registers, ALUs, Caches, Interconnects, etc. Model based reasoning in programming

13. Programming process: getting started. • The programmer starts by constructing a high level model from the specification. • Successive refinement starts by using some combination of the following techniques: • The problem’s state is defined in terms of objects belonging to abstract data types with meaning in the original problem domain. • Key features of the solution are identified and emphasized. These features, sometimes referred to as "beacons” [Wiedenbeck89] , emphasize key aspects of the solution.

14. The programming process • Programmers use an informal, internal notation based on the problem, mathematics, programmer experience, etc. • Within a class of programming languages, the program generated is only weakly dependent on the language. [Robertson90] [Petre88] • Programmers think about code in chunks or “plans”. [Rist86] • Low level plans code a specific operation: e.g. summing an array. • High level or global plans relate to the overall program structure.

15. Programming Process: Strategy + Opportunistic Refinement • Common strategies are: • Backwards goal chaining - start at result and work backwards to generate sub goals. Continue recursively until plans emerge. • Forward chaining: Directly leap to plans when a problem is familiar. [Rist86] • Opportunistic Refinement: [Petre90] • Progress is made at multiple levels of abstraction. • Effort is focused on the most productive level.

16. Programming Process: the role of testing • Programmers test the emerging solution throughout the programming process. • Testing consists of two parts: • Hypothesis generation: The programmer forms an idea of how a portion of the solution should behave. • Simulation: The programmer runs a mental simulation of the solution within the problem models at the appropriate level of abstraction. [Guindom90]

17. From psychology to software • Hypothesis: • A Design Pattern language provides the roadmap to apply results from the psychology of programming to software engineering: • Design patterns capture the essence of plans • The structure of patterns in a pattern language should mirror the types of models programmers use. • Connections between patterns must fit well with goal-chaining and opportunistic refinement.

18. References [Brooks83] R. Brooks, "Towards a theory of the comprehension of computer programs", International Journal of Man-Machine Studies, vol. 18, pp. 543-554, 1983. [Guindom90] R. Guindon, "Knowledge exploited by experts during software system design", Int. J. Man-machine Studies, vol. 33, pp. 279-304, 1990 [Hoc90] J.-M. Hoc, T.R.G. Green, R. Samurcay and D.J. Gilmore (eds.), Psychology of Programming, Academic Press Ltd., 1990. [Petre88] M. Petre and R.L. Winder, "Issues governing the suitability of programming languages for programming tasks. "People and Computers IV: Proceedings of HCI-88, Cambridge University Press, 1988. [Petre90] M. Petre, "Expert Programmers and Programming Languages", in [Hoc90], p. 103, 1990. [Rist86] R.S. Rist, "Plans in programming: definition, demonstration and development" in E. Soloway and S. Iyengar (Eds.), Empirical Studies of Programmers, Norweed, NF, Ablex, 1986. [Rist90] R.S. Rist, "Variability in program design: the interaction of process with knowledge", International Journal of Man-Machine Studies, Vol. 33, pp. 305-322,1990. [Robertson90] S. P. Robertson and C Yu, "Common cognitive representations of program code across tasks and languages", int. J. Man-machine Studies, vol. 33, pp. 343-360, 1990. [Wiedenbeck89] S. Wiedenbeck and J. Scholtz, "Beacons: a knowledge structure in program comprehension", In G. Salvendy and M.J. Smith (eds.) Designing and Using Human Computer interfaces and Knowledge-based systems, Amsterdam: Elsevier,1989.

19. People, Patterns, and Frameworks

20. Eventually End-user, application programs 1 Application patterns & frameworks + Domain Experts 2 Parallel patterns & programming frameworks Application frameworks + Domain literate programming gurus (1% of the population). Parallel programming frameworks 3 Parallel programming gurus (1-10% of programmers) The hope is for Domain Experts to create parallel code with little or no understanding of parallel programming. Leave hardcore “bare metal” efficiency layer programming to the parallel programming experts

21. Today End-user, application programs 1 Application patterns & frameworks + Domain Experts Parallel patterns & programming frameworks 2 Application frameworks + Domain literate programming gurus (1% of the population). Parallel programming frameworks 3 Parallel programming gurus (1-10% of programmers) • For the foreseeable future, domain experts, application framework builders, and parallel programming gurus will all need to learn the entire stack. • That’s why you all need to be here today!

22. Definitions - 1 Design Patterns: “Each design pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice.“ Page x, A Pattern Language, Christopher Alexander Structural patterns: design patterns that provide solutions to problems associated with the development of program structure Computational patterns: design patterns that provide solutions to recurrent computational problems

23. Definitions - 2 Library: The software implementation of a computational pattern (e.g. BLAS) or a particular sub-problem (e.g. matrix multiply) Framework: An extensible software environment (e.g. Ruby on Rails) organized around a structural pattern (e.g. model-view-controller) that allows for programmer customization only in harmony with the structural pattern Domain specific language: A programming language (e.g. Matlab) that provides language constructs that particularly support a particular application domain. The language may also supply library support for common computations in that domain (e.g. BLAS). If the language is restricted to maintain fidelity to a structure and provides library support for common computations then it encompasses a framework (e.g. NPClick).

24. Getting our software act together:First step … Define a conceptual roadmap to guide our work 13 dwarves

25. Alexander’s Pattern Language Christopher Alexander’s approach to (civil) architecture: "Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice.“ Page x, A Pattern Language, Christopher Alexander Alexander’s 253 (civil) architectural patterns range from the creation of cities (2. distribution of towns) to particular building problems (232. roof cap) A pattern language is an organized way of tackling an architectural problem using patterns Main limitation: It’s about civil not software architecture!!!

26. Alexander’s Pattern Language (95-103) • Layout the overall arrangement of a group of buildings: the height and number of these buildings, the entrances to the site, main parking areas, and lines of movement through the complex. • 95. Building Complex • 96. Number of Stories • 97. Shielded Parking • 98. Circulation Realms • 99. Main Building • 100. Pedestrian Street • 101. Building Thoroughfare • 102. Family of Entrances • 103. Small Parking Lots

27. Family of Entrances (102) • May be part of Circulation Realms (98). • Conflict: • When a person arrives in a complex of offices or services or workshops, or in a group of related houses, there is a good chance he will experience confusion unless the whole collection is laid out before him, so that he can see the entrance of the place where he is going. Resolution: Lay out the entrances to form a family. This means: • 1) They form a group, are visible together, and each is visible from all the others. • 2) They are all broadly similar, for instance all porches, or all gates in a wall, or all marked by a similar kind of doorway. • May contain Main Entrance (110), Entrance Transition (112), Entrance Room (130), Reception Welcomes You (149).

28. Family of Entrances http://www.intbau.org/Images/Steele/Badran5a.jpg

29. Computational Patterns The Dwarfs from “The Berkeley View” (Asanovic et al.) Dwarfs form our key computational patterns

30. Patterns for Parallel Programming • PLPP is the first attempt to develop a complete patternlanguage for parallel software development. •  PLPP is a great model for a pattern language for parallel software •  PLPP mined scientific applications that utilize a monolithic application style • PLPP doesn’t help us much with horizontal composition • Much more useful to us than: Design Patterns: Elements of Reusable Object-Oriented Software, Gamma, Helm, Johnson & Vlissides, Addison-Wesley, 1995.

31. Structural programming patterns • In order to create more complex software it is necessary to compose programming patterns • For this purpose, it has been useful to induct a set of patterns known as “architectural styles” • Examples: • pipe and filter • event based/event driven • layered • Agent and repository/blackboard • process control • Model-view-controller

32. Putting it all together… 13 dwarves

33. Elements of a Pattern Description • Name • Problem: • Classes of problems this pattern addresses • Context • Context in which this problem occurs • Forces • Trade-offs that crop up in this situation • Solution • Solution the pattern embodies • Invariants • Properties that need to always be true for this pattern to work • Examples • Known uses • Related Patterns 33

34. Programming Pattern Language 1.0 Keutzer& Mattson Applications Choose your high level architecture - Guided decomposition Identify the key computational patterns – what are my key computations?Guided instantiation Choose your high level structure – what is the structure of my application? Guided expansion Task Decomposition ↔ Data Decomposition Group Tasks Order groups data sharing data access Graph Algorithms Dynamic Programming Dense Linear Algebra Sparse Linear Algebra Unstructured Grids Structured Grids Graphical models Finite state machines Backtrack Branch and Bound N-Body methods Circuits Spectral Methods Model-view controller Iterator Map reduce Layered systems Arbitrary Static Task Graph Pipe-and-filter Agent and Repository Process Control Event based, implicit invocation Productivity Layer Refine the structure - what concurrent approach do I use? Guided re-organization Digital Circuits Task Parallelism Graph algorithms Event Based Divide and Conquer Data Parallelism Geometric Decomposition Pipeline Discrete Event Utilize Supporting Structures – how do I implement my concurrency? Guided mapping Master/worker Loop Parallelism BSP Distributed Array Shared-Data Fork/Join CSP Shared Queue Shared Hash Table Efficiency Layer Implementation methods – what are the building blocks of parallel programming? Guided implementation Thread Creation/destruction Process/Creation/destruction Message passing Collective communication Speculation Transactional memory Barriers Mutex Semaphores

35. Architecting Parallel Software • Decompose Data • Identify data sharing • Identify data access • Decompose Tasks • Group tasks • Order Tasks Identify the Key Computations Identify the Software Structure

36. Identify the SW Structure Structural Patterns • Pipe-and-Filter • Agent-and-Repository • Event-based coordination • Iterator • MapReduce • Process Control • Layered Systems These define the structure of our software but they do not describe what is computed

37. Analogy: Layout of Factory Plant

38. Identify Key Computations Computational Patterns • Computational patterns describe the key computations but not how they are implemented

39. Analogy: Machinery of the Factory

40. Architecting Parallel Software Decompose Tasks/Data Order tasks Identify Data Sharing and Access Identify the Key Computations Identify the Software Structure • Graph Algorithms • Dynamic programming • Dense/Spare Linear Algebra • (Un)Structured Grids • Graphical Models • Finite State Machines • Backtrack Branch-and-Bound • N-Body Methods • Circuits • Spectral Methods • Pipe-and-Filter • Agent-and-Repository • Event-based • Bulk Synchronous • MapReduce • Layered Systems • Arbitrary Task Graphs

41. Analogy: Architected Factory Raises appropriate issues like scheduling, latency, throughput, workflow, resource management, capacity etc.

42. Outline • A Programming Pattern Language • Motivation – patterns: why and what? • Structural patterns • Computational patterns • Composing patterns: examples • Concurrency patterns and PLPP • Examples: pattern language case studies • Examples: pattern language in Smoke Demo

43. Inventory of Structural Patterns • pipe and filter • iterator • MapReduce • blackboard/agent and repository • process control • model-view controller • layered • event-based coordination

44. Elements of a structural pattern • Components are where the computation happens • A configuration is a graph of components (vertices) and connectors (edges) • A structural pattern may be described as a family of graphs. Connectors are where the communication happens

45. Pattern 1: Pipe and Filter • Filters embody computation • Only see inputs and produce outputs Filter 1 • Pipes embody communication Filter 3 Filter 2 Filter 4 May have feedback Filter 5 Filter 6 Filter 7 Examples?

46. Examples of pipe and filter • Almost every large software program has a pipe and filter structure at the highest level Image Retrieval System Logic optimizer Compiler

47. Pattern 2: Iterator Pattern Initialization condition Variety of functions performed asynchronously iterate Synchronize results of iteration No Exit condition met? Examples? Yes

48. Example of Iterator Pattern:Training a Classifier: SVM Training Update surface Iterator Structural Pattern iterate Identify Outlier No All points within acceptable error? Yes 48 48

49. Pattern 3: MapReduce Map Map Reduce Reduce Examples? • To us, it means • A map stage, where data is mapped onto independent computations • A reduce stage, where the results of the map stage are summarized (i.e. reduced)

50. Examples of Map Reduce • General structure: • Map a computation across distributed data sets • Reduce the results to find the best/(worst), maxima/(minima) • Speech recognition • Map HMM computation to evaluate word match • Reduce to find the most-likely word sequences • Support-vector machines (ML) • Map to evaluate distance from the frontier • Reduce to find the greatest outlier from the frontier