Multiprocessors— Flynn Categories, Large vs. Small Scale, Cache Coherency

1. Multiprocessors—Flynn Categories, Large vs. Small Scale, Cache Coherency Professor Alvin R. Lebeck Computer Science 220 Fall 2001

2. What is Parallel Computer Architecture? • A Parallel Computer is a collection of processing elements that cooperate to solve large problems fast • how large a collection? • how powerful are the elements? • how does it scale up? • how do they cooperate and communicate? • how is data transmitted between processors? • what are the primitive abstractions? • how does it all translate to performance?

3. Parallel Computation: Why and Why Not? • Pros • Performance • Cost-effectiveness (commodity parts) • Smooth upgrade path • Fault Tolerance • Cons • Difficult to parallelize applications • Requires automatic parallelization or parallel program development • Software! AAHHHH!

4. Flynn Categories • SISD (Single Instruction Single Data) • Uniprocessors • MISD (Multiple Instruction Single Data) • ??? • SIMD (Single Instruction Multiple Data) • Examples: Illiac-IV, CM-2, Intel MMX • Simple programming model • Low overhead • Flexibility • All custom processors • MIMD (Multiple Instruction Multiple Data) • Examples: Intel 4-way SMP, SUN ES3000, SGI Origin, Cray T3D • Flexible • Use off-the-shelf microprocessors CPS 220

5. Communication Models • Shared Memory • Processors communicate with shared address space • Easy on small-scale machines • Advantages: • Model of choice for uniprocessors, small-scale MPs • Ease of programming • Lower latency • Easier to use hardware controlled caching • Message passing • Processors have private memories, communicate via messages • Advantages: • Less hardware, easier to design • Focuses attention on costly non-local operations • Can support either model on either HW base CPS 220

6. Simple Problem for i = 1 to N A[i] = (A[i] + B[i]) * C[i] sum = sum + A[i] • How do you make this loop parallel to run on many processors?

7. Simple Problem • Split the loops // Independent iterations // Run on up to N processors for i = 1 to N A[i] = (A[i] + B[i]) * C[i] // One last loop to run on one processor for i = 1 to N sum = sum + A[i]

8. Cache(s) and TLB P P P P P P P P Main Memory $ $ $ $ $ $ $ $ 0 N-1 Small Scale Shared Memory Multiprocessors • Small number of processors connected to one shared memory • Memory is equidistant from all processors (UMA) • Kernel can run on any processor (symmetric MP) • Intel dual/quad Pentium, IBM, SUN, Compaq, almost everyone • Some are moving on-chip (e.g., IBM)

9. P Mem P Mem P $ Mem P $ Cntrl/NI Mem $ Cntrl/NI $ Cntrl/NI Cntrl/NI Interconnect Large Scale Shared Memory Multiprocessors Shared • 100s to 1000s of nodes (processors) with single shared physical address space • Use General Purpose Interconnection Network • Still have cache coherence protocol • Use messages instead of bus transactions • No hardware broadcast • Communication Assist • Cray T3D, T3E, Compaq EV7, SUN ES3000 CPS 220

10. Mem Mem P P CA CA $ $ CA CA $ $ Mem Mem P P Message Passing Architectures • Cannot directly access memory on another node • IBM SP-2, Intel Paragon • Cluster of workstations Node 0 0,N-1 Node 1 0,N-1 Interconnect Node 2 0,N-1 Node 3 0,N-1

11. Important Communication Properties • Bandwidth • Need high bandwidth in communication • Cannot scale, but stay close • Limits may be in network, memory, and processor • Overhead to communicate is a problem in many machines • Latency • Affects performance, since processor may have to wait • Affects ease of programming, since requires more thought to overlap communication and computation • Latency Hiding • How can a mechanism help hide latency? • Examples: overlap message send with computation, prefetch CPS 220

12. Cache(s) and TLB P P P P P P P P Main Memory $ $ $ $ $ $ $ $ 0 N-1 Small-Scale—Shared Memory • Caches serve to: • Increase bandwidth versus bus/memory • Reduce latency of access • Valuable for both private data and shared data • What about cache coherence? CPS 220

13. Cache Coherence Problem (Initial State) P2 P1 Time Interconnection Network / Bus x Main Memory

14. Cache Coherence Problem (Step 1) P2 P1 ld r2, x Time Interconnection Network / Bus x Main Memory

15. P2 P1 ld r2, x ld r2, x Time Interconnection Network / Bus x Main Memory Cache Coherence Problem (Step 2)

16. P2 P1 ld r2, x ld r2, x add r1, r2, r4 st x, r1 Time Interconnection Network / Bus x Main Memory Cache Coherence Problem (Step 3)

17. The Problem of Cache Coherence (4) ld r2, x add r1, r3, r4 st x, r1 ld r4,y ld r2, x add r1, r2, r3 st y, r1 ld r5, x P2 P1 Time x y Interconnection Network / Bus y x Main Memory CPS 220

18. Coherence vs. Consistency • Intuition says loads should return latest value • what is latest? • Coherence concerns only one memory location • Consistency concerns apparent ordering for all locations • A Memory System is Coherent if • can serialize all operations to that location such that, • operations performed by any processor appear in program order • program order = order defined program text or assembly code • value returned by a read is value written by last store to that location

19. Why Coherence != Consistency /* initial A = B = flag = 0 */ P1P2 A = 1; while (flag == 0); /* spin */ B = 1; print A; flag = 1; print B; Intuition says printed A = B = 1 Coherence doesn’t say anything, why?

20. The Sequential Consistency Memory Model sequential processors issue memory ops in program order P3 P1 P2 switch randomly set after each memory op Memory

21. Sufficient Conditions for Sequential Consistency • Every processor issues memory ops in program order • Processor must wait for store to complete before issuing next memory operation • After load, issuing proc waits for load to complete, and store that produced value to complete before issuing next op • Easily implemented with shared bus.

22. Potential Solutions • Snooping Solution (Snoopy Bus): • Send all requests for data to all processors • Processors snoop to see if they have a copy and respond accordingly • Requires broadcast, since caching information is at processors • Works well with bus (natural broadcast medium) • Dominates for small scale machines (most of the market) • probably won’t scale beyond 2-4 processors • Directory-Based Schemes • Keep track of what is being shared in one centralized place • Distributed memory => distributed directory (avoids bottlenecks) • Send point-to-point requests to processors • Scales better than Snoop • Actually existed BEFORE Snoop-based schemes CPS 220

23. Basic Snoopy Protocols • Write Invalidate Protocol: • Multiple readers, single writer • Write to shared data: an invalidate is sent to all caches which snoop and invalidate any copies • Read Miss: • Write-through: memory is always up-to-date • Write-back: snoop in caches to find most recent copy • Write Broadcast (Update) Protocol: • Write to shared data: broadcast on bus, processors snoop, and update copies • Read miss: memory is always up-to-date • Write serialization: bus serializes requests • Bus is single point of arbitration CPS 220

24. Snoopy Cache-Coherence Protocols • Bus provides serialization point for consistency • but, but, what about write-buffers? Later in the semester…. • Each cache controller “snoops” all bus transactions • relevant transactions if for a block it contains • take action to ensure coherence • invalidate • update • supply value • depends on state of the block and the protocol • Simultaneous Operation of Independent Controllers

25. Snoopy Design Choices • Controller updates state of blocks in response to processor and snoop events and generates bus xactions • Often have duplicate cache tags • Snoopy protocol • set of states • state-transition diagram • actions • Basic Choices • write-through vs. write-back • invalidate vs. update Processor ld/st Cache State Tag Data . . . Snoop (observed bus transaction)

26. PrRd / PrWr Valid BusWr PrRd / BusRd Invalid PrWr / BusWr The Simple Invalidate Snoopy Protocol • Write-through, no-write-allocate cache • Actions: PrRd, PrWr, BusRd, BusWr

27. A 3-State Write-Back Invalidation Protocol • 2-State Protocol + Simple hardware and protocol • Bandwidth (every write goes on bus!) • 3-State Protocol (MSI) • Modified • one cache has valid/latest copy • memory is stale • Shared • one or more caches have valid copy • Invalid • Must invalidate all other copies before entering modified state • Requires bus transaction (order and invalidate)

28. MSI Processor and Bus Actions • Processor: • PrRd • PrWr • Writeback on replacement of modified block • Bus • Bus Read (BusRd) Read without intent to modify, data could come from memory or another cache • Bus Read-Exclusive (BusRdX) Read with intent to modify, must invalidate all other caches copies • Writeback (BusWB) cache controller puts contents on bus and memory is updated • Definition: cache-to-cache transfer occurs when another cache satisfies BusRd or BusRdX request • Let’s draw it!

29. M S PrRd BusRd I MSI State Diagram PrWr PrRd BusRdX / BusWB BusRd / BusWB PrWr / BusRdX BusRdX PrRd / BusRd PrWr / BusRdX

30. An example Proc ActionP1 StateP2 stateP3 stateBus ActData from 1. P1 read u S -- -- BusRd Memory 2. P3 read u S -- S BusRd Memory 3. P3 write u I -- M BusRdX Memory or not 4. P1 read u S -- S BusRd P3’s cache 5. P2 read u S S S BusRd Memory • Single writer, multiple reader protocol • Why Modified to Shared? • What if not in any cache? • Read, Write produces 2 bus transactions!

31. 4-State (MESI) Invalidation Protocol • Often called the Illinois protocol • Modified (dirty) • Exclusive (clean unshared) only copy, not dirty • Shared • Invalid • Requires shared signal to detect if other caches have a copy of block • Cache Flush for cache-to-cache transfers • Only one can do it though • What does state diagram look like?

32. 4-State Write-back Update Protocol • Dragon (Xerox PARC) • States • Exclusive (E): one copy, clean, memory is up-to-date • Shared-Clean (SC): could be two or more copies, memory unknown • Shared-Modified (SM): could be two or more copies, memory stale • Modified (M) • Adds Bus Update Transaction • Adds Cache Controller Update operation • Must obtain bus before updating local copy • What does state diagram look like? • let’s look at the actions first

33. Basic Snoopy Protocols • Write Invalidate versus Broadcast: • Invalidate requires one transaction per write-run • Invalidate uses spatial locality: one transaction per block • Broadcast has lower latency between write and read • Broadcast: BW (increased) vs. latency (decreased) tradeoff Name Protocol Type Memory-write policy Machines using Write Once Write invalidate Write back First snoopy protocol. after first write Synapse N+1 Write invalidate Write back 1st cache-coherent MPs Berkeley Write invalidate Write back Berkeley SPUR Illinois Write invalidate Write back SGI Power and Challenge “Firefly” Write broadcast Write back private, Write through shared SPARCCenter 2000 CPS 220

34. Larger MPs • Separate Memory per Processor • Local or Remote access via memory controller • Cache Coherency solution: non cached pages • Alternative: directory/cache that tracks state of every block in every cache • Which caches have a copies of block, dirty vs. clean, ... • Info per memory block vs. per cache block? • In memory => simpler protocol (centralized/one location) • In memory => directory is ƒ(memory size) vs. ƒ(cache size) • Prevent directory as bottleneck: distribute directory entries with memory, each keeping track of which Procs have copies of their blocks CPS 220

35. Directory Protocol • Similar to Snoopy Protocol: 3 states • Shared: > 1 processors have data, memory up to date • Uncached • Exclusive: 1 processor(owner) has data; memory out of date • In addition to cache state, must track which processors have data when in shared state • Terms: • Local node is the node where a request originates • Home node is the node where the memory location of an address resides • Remote node is the node that has a copy of a cache block, whether exclusive or shared. CPS 220

36. Example Directory Protocol • Message sent to directory causes 2 actions: • update the directory • more messages to satifty request • Block is in Uncached state: the copy in memory is the current value, & only possible requests for that block are: • Read miss: requesting processor is sent back the data from memory and the requestor is the only sharing node. The state of the block is made Shared. • Write miss: requesting processor is sent the value and becomes the Sharing node. The block is made Exclusive to indicate that the only valid copy is cached. Sharers indicates the identity of the owner. • Block is Shared, the memory value is up-to-date: • Read miss: requesting processor is sent back the data from memory & requesting processor is added to the sharing set. • Write miss: requesting processor is sent the value. All processors in the set Sharers are sent invalidate messages, & Sharers set is to identity of requesting processor. The state of the block is made Exclusive. CPS 220

37. Example Directory Protocol • Block is Exclusive: current value of the block is held in the cache of the processor identified by the set Sharers (the owner), & 3 possible directory requests: • Read miss: owner processor is sent a data fetch message, which causes state of block in owner’s cache to transition to Shared and causes owner to send data to directory, where it is written to memory and sent back to the requesting processor. Identity of requesting processor is added to set Sharers, which still contains the identity of the processor that was the owner (since it still has a readable copy). • Data write-back: owner processor is replacing the block and hence must write it back. This makes the memory copy up-to-date (the home directory essentially becomes the owner), the block is now uncached, and the Sharer set is empty. • Write miss: block has a new owner. A message is sent to old owner causing the cache to send the value of the block to the directory from which it is send to the requesting processor, which becomes the new owner. Sharers is set to identity of new owner, and state of block is made Exclusive. CPS 220