1 / 34

Multiprocessors and Thread-Level Parallelism

Vincent Berk November 12 th , 2008 Reading for Friday: Sections 4.1-4.3 Reading for Monday: Sections 4.4 – 4.9. Multiprocessors and Thread-Level Parallelism. Parallel Computers.

ursa
Download Presentation

Multiprocessors and Thread-Level Parallelism

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ENGS 116 Lecture 15 Vincent Berk November 12th , 2008 Reading for Friday: Sections 4.1-4.3 Reading for Monday: Sections 4.4 – 4.9 Multiprocessors and Thread-Level Parallelism

  2. ENGS 116 Lecture 15 Parallel Computers • Definition: “A parallel computer is a collection of processing elements that cooperate and communicate to solve large problems fast.” Almasi and Gottlieb, Highly Parallel Computing, 1989 • Questions about parallel computers: • How large a collection? • How powerful are processing elements? • How do they cooperate and communicate? • How is data transmitted? • What type of interconnection? • What are HW and SW primitives for programmer? • Does it translate into performance?

  3. ENGS 116 Lecture 15 Parallel Processors “Religion” • The dream of computer architects since 1960: replicate processors to add performance vs. design a faster processor • Led to innovative organization tied to particular programming models since “uniprocessors can’t keep going” • e.g., uniprocessors must stop getting faster due to limit of speed of light: 1972, … , 1989, ... , 2008 • Borders religious fervor: you must believe! • Fervor damped some when 1990s companies went out of business: Thinking Machines, Kendall Square, ... • Argument instead is the “pull” of opportunity of scalable performance, not the “push” of uniprocessor performance plateau • Recent advancement: Sun Niagara “T1 Coolthreads” => 8 in-order simple execution cores. Switch thread on stalls...

  4. ENGS 116 Lecture 15 Opportunities: Scientific Computing • Nearly Unlimited Demand (Grand Challenge): App Perf (GFLOPS) Memory (GB) 48 hour weather 0.1 0.1 72 hour weather 3 1 Pharmaceutical design 100 10 Global Change, Genome 1000 1000 (Figure 1-2, page 25, of Culler, Singh, Gupta [CSG97]) • Successes in some real industries: • Petroleum: reservoir modeling • Automotive: crash simulation, drag analysis, engine • Aeronautics: airflow analysis, engine, structural mechanics • Pharmaceuticals: molecular modeling • Entertainment: full length movies (“Finding Nemo”)

  5. ENGS 116 Lecture 15 Flynn’s Taxonomy • SISD (Single Instruction Single Data) • Uniprocessors • MISD (Multiple Instruction Single Data) • ??? • SIMD (Single Instruction Multiple Data) • Examples: Illiac-IV, CM-2 • Simple programming model • Low overhead • Flexibility • All custom integrated circuits • MIMD (Multiple Instruction Multiple Data) • Examples: Sun Enterprise 10000, Cray T3D, SGI Origin • Flexible • Use off-the-shelf microprocessors

  6. ENGS 116 Lecture 15 Memory Module Memory Module Memory Module Memory Module Memory Module Memory Module Memory Module Memory Module Memory Module Instruction Stream 1 Data Data Data Data Data Stream Stream 4 Stream 1 Stream 2 Stream 3 Data Data Data Data Stream 1 Stream 2 Stream 3 Stream 4 Instruction Stream Processor Unit Processor Unit Processor Unit Processor Unit b) SIMD Computer a) SISD Computer Processor Unit Instruction Stream 2 Processor Unit Processor Unit Processor Unit Processor Unit Instruction Stream c) MIMD Computer Instruction Stream Instruction Stream 3 Instruction Instruction Stream Stream Instruction Instruction Instruction Instruction Stream 1 Stream 2 Stream 3 Stream 4 Control Unit Control Unit Control Unit Control Unit Control Unit Control Unit Instruction Stream 4 Flynn’s classification of computer systems

  7. ENGS 116 Lecture 15 Parallel Framework • Programming Model: • Multiprogramming: lots of jobs, no communication • Shared address space: communicate via memory • Message passing: send and receive messages • Data parallel: several agents operate on several data sets simultaneously and then exchange information globally and simultaneously (shared or message passing) • Communication Abstraction: • Shared address space: e.g., load, store, atomic swap • Message passing: e.g., send, receive library calls • Debate over this topic (ease of programming, scaling)

  8. ENGS 116 Lecture 15 P P P P Network  Network P P P  M M M M M  M Diagrams of Shared and Distributed Memory Architectures Distributed Memory system Shared Memory System

  9. ENGS 116 Lecture 15 Shared Address/Memory Multiprocessor Model • Communicate via Load and Store • Oldest and most popular model • Based on timesharing: processes on multiple processors vs. sharing single processor • Process: a virtual address space and ≥ 1 thread of control • Multiple processes can overlap (share), but ALL threads share a process address space • Writes to shared address space by one thread are visible to reads of other threads • Usual model: share code, private stack, some shared heap, some private heap (shared page table!)

  10. ENGS 116 Lecture 15 Processor Processor Processor Processor One or more levels of cache One or more levels of cache One or more levels of cache One or more levels of cache I/O System Main memory Example: Small-Scale MP Designs • Memory: centralized with uniform memory access (“UMA”) and bus interconnect, I/O • Examples: Sun Enterprise, SGI Challenge, Intel SystemPro

  11. ENGS 116 Lecture 15 SMP Interconnect • Processors connected to memory AND to I/O • Bus based: all memory locations equal access time so SMP = “Symmetric MP” • Sharing limited BW as we add processors, I/O • Crossbar: expensive to expand • Multistage network: less expensive to expand than crossbar with more BW • “Dance Hall” designs: all processors on the left, all memories on the right

  12. ENGS 116 Lecture 15 Processor + cache Processor + cache Processor + cache Processor + cache 1 cycle Memory I/O Memory I/O Memory I/O Memory I/O 40 cycles 100 cycles Low Latency High Reliability Interconnection Network I/O I/O I/O I/O Memory Memory Memory Memory Processor + cache Processor + cache Processor + cache Processor + cache Large-Scale MP Designs • Memory: distributed with nonuniform memory access (“NUMA”) and scalable interconnect (distributed memory)

  13. ENGS 116 Lecture 15 Shared Address Model Summary • Each processor can name every physical location in the machine • Each process can name all data it shares with other processes • Data transfer via load and store • Data size: byte, word, ... or cache blocks • Uses virtual memory to map virtual to local or remote physical • Memory hierarchy model applies: now communication moves data to local processor’s cache (as load moves data from memory to cache) • Latency, BW, scalability of communication?

  14. ENGS 116 Lecture 15 Message Passing Model • Whole computers (CPU, memory, I/O devices) communicate as explicit I/O operations • Essentially NUMA but integrated at I/O devices vs. memory system • Send specifies local buffer + receiving process on remote computer • Receive specifies sending process on remote computer + local buffer to place data • Usually send includes process tag and receive has rule on tag: match one, match any • Synchronization: when send completes, when buffer free, when request accepted, receive waits for send • Send + receive => memory-memory copy, where each supplies local address AND does pairwise synchronization!

  15. ENGS 116 Lecture 15 = processor = receiver’s buffer or queue = message = process Logical View of a Message Passing System Communications Network Processor 1 Processor 2 Port 2 Port 1

  16. ENGS 116 Lecture 15 Message Passing Model • Send + receive => memory-memory copy, synchronization on OS even on 1 processor • History of message passing: • Network topology important because could only send to immediate neighbor • Typically synchronous, blocking send & receive • Later DMA with non-blocking sends, DMA for receive into buffer until processor does receive, and then data is transferred to local memory • Later SW libraries to allow arbitrary communication • Example: IBM SP-2, RS6000 workstations in racks • Network Interface Card has Intel 960 • 8x8 Crossbar switch as communication building block • 40 MB/sec per link

  17. ENGS 116 Lecture 15 Message Passing Model • Special type of Message Passing: Pipe • Pipe is a point-to-point connection • Data is pushed in one way by first process, and • Received, in-order, by the second process. • Data can stay in pipe indefinitely • Asynchronous, simplex communication • Advantages: • Simple implementation • Works with Send-Receive calls • Good way of implementing producer consumers systems. • No latency, since communication is 1-way

  18. ENGS 116 Lecture 15 Communication Models • Shared Memory • Processors communicate with shared address space • Easy on small-scale machines • Advantages: • Model of choice for uniprocessors, small-scale MPs • Ease of programming • Lower latency • Easier to use hardware controlled caching • Message passing • Processors have private memories, communicate via messages • Advantages: • Less hardware, easier to design • Focuses attention on costly non-local operations • Can support either SW model on either HW base

  19. ENGS 116 Lecture 15 How Do We Develop a Parallel Program? • Convert a sequential program to a parallel program • Partition the program into tasks and combine tasks into processes • Coordinate data accesses, communication, and synchronization • Map the processes to processors • Create a parallel program from scratch

  20. ENGS 116 Lecture 15 Review: Parallel Framework • Programming ModelCommunication AbstractionInterconnection SW/OS Interconnection HW • Layers: • Programming Model: • Multiprogramming: lots of jobs, no communication • Shared address space: communicate via memory • Message passing: send and receive messages • Data parallel: several agents operate on several data sets simultaneously and then exchange information globally and simultaneously (shared or message passing) • Communication Abstraction: • Shared address space: e.g., load, store, atomic swap • Message passing: e.g., send, receive library calls • Debate over this topic (ease of programming, scaling)

  21. ENGS 116 Lecture 15 Fundamental Issues • Fundamental issues in parallel machines 1) Naming 2) Synchronization 3) Latency and Bandwidth

  22. ENGS 116 Lecture 15 Fundamental Issue #1: Naming • Naming: • what data is shared • how it is addressed • what operations can access data • how processes refer to each other • Choice of naming affects code produced by a compiler; via load where just remember address or keep track of processor number and local virtual address for message passing • Choice of naming affects replication of data; via load in cache memory hierarchy or via SW replication and consistency

  23. ENGS 116 Lecture 15 Fundamental Issue #1: Naming • Global physical address space: any processor can generate, address and access it in a single operation • Global virtual address space: if the address space of each process can be configured to contain all shared data of the parallel program • Segmented shared address space: locations are named < process number, address > uniformly for all processes of the parallel program • Independent local physical addresses

  24. ENGS 116 Lecture 15 Fundamental Issue #2: Synchronization • To cooperate, processes must coordinate • Message passing is implicit coordination with transmission or arrival of data • Shared address system needs additional operations to explicitly coordinate: e.g., write a flag, awaken a thread, interrupt a processor

  25. ENGS 116 Lecture 15 Fundamental Issue #3: Latency and Bandwidth • Bandwidth • Need high bandwidth in communication • Cannot scale, but stay close • Match limits in network, memory, and processor • Overhead to communicate is a problem in many machines • Latency • Affects performance, since processor may have to wait • Affects ease of programming, since requires more thought to overlap communication and computation • Latency Hiding • How can a mechanism help hide latency? • Examples: overlap message send with computation, prefetch data, switch to other tasks

  26. ENGS 116 Lecture 15 Processor Processor Processor Processor One or more levels of cache One or more levels of cache One or more levels of cache One or more levels of cache I/O System Main memory Small-Scale — Shared Memory • Caches serve to: • Increase bandwidth versus bus/memory • Reduce latency of access • Valuable for both private data and shared data • What about cache coherence?

  27. ENGS 116 Lecture 15 Cache Cache Cache A’ A’ A’ 100 550 100 B’ B’ B’ 200 200 200 Memory Memory Memory A A A 100 100 100 B B B 200 200 400 The Problem Cache Coherency CPU CPU CPU I/O I/O Output A gives 100 I/O Input 400 to B (a) Cache and memory coherent: A’ = A & B’ = B (b) Cache and memory incoherent: A’ ≠ A (A stale) (c) Cache and memory incoherent: B’ ≠ B (B’ stale)

  28. ENGS 116 Lecture 15 What Does Coherency Mean? • Informally: • “Any read must return the most recent write” • Too strict and too difficult to implement • Better: • “Any write must eventually be seen by a read” • All writes are seen in proper order (“serialization”) • Two rules to ensure this: • “If P writes x and P1 reads it, P’s write will be seen by P1 if the read and write are sufficiently far apart” • Writes to a single location are serialized: seen in one order • Latest write will be seen • Otherwise could see writes in illogical order (could see older value after a newer value)

  29. ENGS 116 Lecture 15 The cache-coherence problem for a single memory location (X), read and written by two processors (A and B).This example assumes a write-through cache.

  30. ENGS 116 Lecture 15 Potential HW Coherency Solutions • Snooping Solution (Snoopy Bus): • Send all requests for data to all processors • Processors snoop to see if they have a copy and respond accordingly • Requires broadcast, since caching information is at processors • Works well with bus (natural broadcast medium) • Dominates for small-scale machines (most of the market) • Directory-Based Schemes • Keep track of what is being shared in one centralized place • Distributed memory => distributed directory for scalability(avoids bottlenecks) • Send point-to-point requests to processors via network • Scales better than Snooping • Actually existed BEFORE Snooping-based schemes

  31. ENGS 116 Lecture 15 Basic Snoopy Protocols • Write Invalidate Protocol: • Multiple readers, single writer • Write to shared data: an invalidate is sent to all caches which snoop and invalidate any copies • Read Miss: • Write-through: memory is always up-to-date • Write-back: snoop in caches to find most recent copy • Write Broadcast Protocol (typically write through): • Write to shared data: broadcast on bus, processors snoop, and update any copies • Read miss: memory is always up-to-date

  32. ENGS 116 Lecture 15 An example of an invalidation protocol working on a snooping bus for a single cache block (X) with write-back caches.

  33. ENGS 116 Lecture 15 An example of a write update or broadcast protocol working on a snooping bus for a single cache block (X) with write-back caches.

  34. ENGS 116 Lecture 15 Basic Snoopy Protocols • Write Invalidate versus Broadcast: • Invalidate requires one transaction per write-run (writing to one location multiple times by the same CPU) • Invalidate uses spatial locality: one transaction per block • Broadcast has lower latency between write and read

More Related