1 / 37

Shared Memory Without A Bus

Explore the challenges and protocols of implementing shared memory in distributed systems without a central bus. Learn about directory-based coherence schemes and how they maintain cache coherency.

rwakefield
Download Presentation

Shared Memory Without A Bus

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Shared Memory Without A Bus In SMPs the bus is a centralized point where the writes can be serialized. When no such point exists, as in large parallel computers, the situation gets very much more complicated. We continue our examination of shared memory implementations Source: Culler/Singh, Parallel Computer Architectures, MK ‘99

  2. Preliminaries • The computers implementing shared memory without a central bus are called “distributed shared memory” (DSM) machines • The subclass is the CC-NUMA machines, for cache coherent non-uniform memory access • On an access-fault by the processor • Find out information about the state of the cache block in other machines • Determine the exact location of copies, if necessary • Communicate with other machines to implement the shared memory protocol

  3. P3 P0 P1 P2 Cache Cache Cache Cache “Distributed” Applies to Memory • DSM computers have a CTA architecture with additional hardware to maintain coherency • Collectively, the controllers make the memory look shared $$$ Mem $$$ Mem $$$ Mem $$$ Mem Control Control Control Control Interconnection Network

  4. P0 P1 $ Mem $ Mem Directory Directory Directory Controller Directory Controller Interconnection Network Directory Based Cache-coherence Since broadcasting the memory references is impractical -- that’s what buses do -- a directory-based scheme is an alternative • A directory is a data structure giving the state of each cache block in the machine

  5. How Does It Work? • Using the directory it is possible to maintain cache coherency in a DSM, but its complex (and time consuming) • To illustrate, we work through the protocols to maintain memory coherency • Concepts • Events: A read or write access fault • Cache fields these for local data, controller fields these for remotely allocated data • Proc/Proc communication is by packets through the interconnection network

  6. Terminology • Node, a processor, cache and memory • Home node, node whose main memory has the block allocated • Dirty node, a node with a modified value • Owner, node holding a valid copy, usually the home or dirty node • Exclusive node, holds only valid cached copy • Requesting node, (local) node asking for the block

  7. Sample Directory Scheme • Local node has access fault • Sends request to home node for directory information • Read -- directory tells which node has the valid data and the data is requested • Write -- directory tells nodes with copies ... Invalidation or update requests are sent • Acknowledgments are returned • Processor waits for all ACKs before completion Notice that many transactions can be “in the air” at once, leading to possible races

  8. A Directory Entry • Directory entries don’t usually keep cache state • Use a P-length bit-vector to tell in which processors the block is present … presence bit • Clean/dirty bit implies exactly 1 presence bit on • Sufficient? • Determine who has valid copy for read-miss • Determine who has copies to be invalidated Dirty P1 P3 P5 P7 P0 P2 P4 P6 Presence Bits 1 1

  9. A Closer Look (Read) I • Postulate 1 processor per node, 1 level cache, local MSI protocol [from last week] • On a read access fault at Px, the local directory controller determines if block is locally/remotely allocated • If local, it delivers data • If remote it finds the home … by high order bits probably • Controller sends request to home node for blk • Home controller looks up directory entry for blk • Dirty bit OFF, controller finds blk in memory, sends reply, sets xth presence bit ON

  10. A Closer Look (Read) II • Dirty bit ON -- controller sends reply to Px of the processor ID of Py, the owner • Px requests data from owner Py • Owner Py controller, sets state to “shared,” forwards data to Px and sends data to home • At home, data is updated, dirty bit is turned OFF and the xth presence bit is set ON and yth presence bit remains ON This is basically the protocol for the LLNL S-1 multicomputer from the late ‘70s

  11. A Closer Look (Write) I On a write access fault at Px, the local directory controller checks if the block is locally/remotely allocated; if remote it finds the home • Controller sends request to home node for blk • Home controller looks up directory entry of blk • Dirty bit OFF, the home has a clean copy • Home node sends data to Px w/presence vector • Home controller clears directory, sets xth bit ON and sets dirty bit ON • Px controller sends invalidation request to all nodes listed in the presence vector

  12. A Closer Look (Write) II • Px controller awaits ACKs from all those nodes • Px controller delivers blk to cache in dirty state • Dirty bit is ON • Home notifies owner Py of Px’s write request • Py controller invalidates its blk, sends data to Px • Home clears yth presence bit, turns xth bit ON and dirty bit stays ON • On writeback, home stores data, clears both presence and dirty bits

  13. Detailed Example • Consider the example similar to last week • The assumptions are … • a is globally allocated • a has it’s home at P1 • P0 previously read a P0 P1 P2 P3 P1 reads a into its cache P3 reads a into its cache P3 changes a to 5 {P2reads a into its cache P2 writes a in its cache} a:V:4 $ a:4 01000 $ $ Ctlr Ctlr Ctlr Ctlr Interconnection Network

  14. P1 Reads a Into Cache • The local directory controller determines if block is locally/remotely allocated • If remote it finds the home … by high order bits probably • Controller asks home node for blk: No-op • Home controller looks up directory entry for blk • Dirty bit OFF, controller finds blk in memory, sends reply, sets xth presence bit ON P0 P1 P2 P3 In the special case that a processor references it’s own globally allocated data no communication is required, only manage the presence bits a:V:4 a:V:4 a:4 01100 $ $ Ctlr Ctlr Ctlr Ctlr Interconnection Network

  15. P3 Reads a Into Cache • The local directory controller determines if block is locally/remotely allocated • If remote it finds the home … by high order bits probably • Controller asks home node for blk: Message to P1 • Home controller looks up directory entry for blk • Dirty bit OFF, controller finds blk in memory, sends message to P3, sets xth presence bit ON P0 P1 P2 P3 Msg: P3 to P1, Read a Msg: P1 to P3, Here’s a a:V:4 a:V:4 a:4 01101 $ a:V:4 Controller Controller Controller Controller Interconnection Network

  16. P3 Writes a Changing It To 5 Part I • On a write access fault at Px, local controller checks and finds it remote; finds the home • Controller sends request to home node for blk • Home controller looks up directory entry of blk • Dirty bit OFF, the home has a clean copy • Home node sends data to Px w/presence vector • Home controller clears directory, sets xth bit and dirty ON • Px controller sends invalidation request to all nodes listed P0 P1 P2 P3 Stalled Msg: P3 to P1, Write a Msg: P1 to P3, a:01101 Msg: P3 to P0, Invalid a Msg: P3 to P1, Invalid a a:V:4 a:V:4 a:4 10001 $ a:V:4 Controller Controller Controller Controller Interconnection Network

  17. P3 Writes a Changing It To 5 Part II • Processor continues to be stalled • Px controller awaits ACKs from all those nodes • Px controller delivers blk to cache in dirty state • Total messages when clean copy exists: ToHome, FromHome, (Invalidate, ACK)*s P0 P1 P2 P3 Msg: P0 to P3, ACK Msg: P1 to P3, ACK a:I:4 a:I:4 a:4 10001 $ a:M:5 Controller Controller Controller Controller Interconnection Network

  18. P2 Reads a Into Cache Dirty bit ON -- home controller sends reply to Px of the processor ID of Py, the owner; Px asks Py for data • Owner Py controller, sets state to “shared,” forwards data to Px and sends data to home • At home, data is updated, dirty bit is turned OFF and the xth presence bit is set ON and yth presence bit remains ON P0 P1 P2 P3 Msg: P2 to P1, Read a Msg: P1 to P2, P3 has it Msg: P2 to P3, Read a Msg: P3 to P2, Here’s a Msg: P3 to P1, Here’s a a:I:4 a:I:4 a:5 00011 a:V:5 a:V:5 Controller Controller Controller Controller Interconnection Network

  19. Instead Let P2’s Request Be Write 6 • That is … this action replaces the previous slide • Dirty bit is ON • Home notifies owner Py of Px’s write request • Py controller invalidates its block, sends data to Px • Home clears yth presence bit, turns xth bit ON and dirty bit stays ON P0 P1 P2 P3 Msg: P2 to P1, Write a Msg: P1 to P3, P2 asking Msg: P3 to P2, Here’s a a:I:4 a:I:4 a:4 10010 a:M:6 a:I:5 Controller Controller Controller Controller Interconnection Network

  20. Summarizing The Example • The controller sends out a series messages to keep the writes to the memory locations coherent • The scheme differs from the bus solution in that all processors get the information at the same time using the bus, but at different times using the network • The number of messages is potentially large if there are many sharers

  21. Homework Assignment Suppose a 100x100 array S is distributed by blocks across four processors, so that each contains a 50x50 subarray. At each position S[i,j] is updated by the sum of its 8 nearest neighbors: S’[i,j]=(S[i-1,j-1]+S[i-1,j]+S[i-1,j+1]+S[i,j+1]+S[i+1,j+1] +S[i+1,j]+S[i+1,j-1]+S[i,j-1])/8; If each processor updates its own elements, how many messages must are produced to maintain concurrency for a directory based CC-NUMA? HINT: Assume that an extra row and column, initialized to zero, surrounds A, A is allocated in rmo, and storage for S alternates with S’

  22. Break

  23. Alternative Directory Schemes • The “bit vector directory” is storage-costly • Consider improvements to Mblk*P cost • Increase block size, cluster processors • Just keep list of Processor IDs of sharers • Need overflow scheme • Five slots probably suffice • Link the shared items together • Home keeps the head of list • List is doubly-linked • New sharer adds self to head of list • Obvious protocol suffices, but watch for races

  24. Assessment • An obvious difference between directory and bus solutions is that for directories, the invalidate request grows as the number of processors that are sharing • Directories take memory • 1 bit per block per processor + c • If a block is B bytes, 8B processors imply 100% overhead to store the directory

  25. Performance Data • To see how much sharing takes place and how many invalidations must be sent, experiments were run • Summarizing the data • Usually there are few shares • The mode is 1 other processor(s) sharing ~ 60 • The “tail” of the distribution stretches out for some applications • Remote activity increases as the number of processors • Larger block sizes increase traffic, 32 is good

  26. Protocol Optimizations I • Read Request to Exclusively Held Block 1: Request 2: Response 3: Intervention 4a: Revise 4b: Response 3 Strict Request/ Response 1 L H R 2 4a 4b 1: Request 2: Intervention 3: Revise 4: Response 1 2 L H R Intervention Forwarding 4 3 1: Request 2: Intervention 3a: Revise 3b: Response 1 2 Reply Forwarding L H R 3a 3b

  27. Protocol Optimizations II (Lists) • Improved Invalidation 5:I ACK includes next sharer on list 3:I 1:I L S1 S2 S3 2:A 6:A 4:A ACK and next invalidate in parallel 1:I 2a:I 3a:I L S1 S2 S3 2b:A 4:A 3b:A 1:I 2:I 3:I ACK comes from the last sharer L S1 S2 S3 4:A

  28. P0 P1 P2 P3 Cache Cache Cache Cache Directory Directory Controller Interconnection Network Higher Level Optimization • Organizing nodes as SMPs with one coherent memory and one directory controller can improve performance since one processor might fetch data that the next processor wants … it is already present • The main liability is that the controller resource and probably its channel into the network are shared

  29. Serialization • The bus defines the ordering on writes in SMPs • For directory systems, memory (home) does • If home always has value, FIFO would work • Consider a block in modified state and two nodes requesting exclusive access in an invalidation protocol: The requests reach home in one order, but they could reach the owner in a different order; which order prevails? • Fix: Add “busy state” indicating transaction in flight

  30. Four Solutions To Ensure Serialization • Buffer at home -- keep request at home, service in order … lower concurrency, overflow • Buffer at requesters with linked list; follow Py • NACK and retry -- when directory is busy, just “return to sender” • Forward to dirty node -- serialize at home for clean, serialize at owner otherwise

  31. Coherency != Memory Consistency A=1; while (A==0)do; B=1; while (B==0)do; print A; Assume A and B initially 0 P0 P1 P2 Print 0 $ Mem $ Mem $ Mem Directory Controller Directory Controller Directory Controller Interconnection Network A=1; B=1; Delay A=1;

  32. Sequential Consistency • Sequential Consistency is a very strict form of memory consistency • A MP is sequentially consistent if the result of any execution is the same as some sequential order and operations of each processor are in program order A=1; while (A==0)do; B=1; while (B==0)do; print A;

  33. Relaxed Consistency Models • Since sequential consistency is so strict, alternative schemes allow reordering of reads and writes to improve performance • total store ordering (TSO) • partial store ordering (PSO) • relaxed memory ordering (RMO) • processor consistency (PC) • weak ordering (WO) • release consistency (RC) • Many are difficult to use in practice

  34. Relaxing Write-to-Read Program Order • While a write miss is in the write buffer and not yet visible to other processors, the processor can issue and complete reads that hit in its cache or even a single read that misses in its cache. TSO and PSO allow this. • This matches intuition often … • This code works as expected P0 P1 P0 P1 A=1; while (Flag==0)do; A=1; print B; Flag=1; print A; B=1; print A;

  35. Less Intuitive • Some programs don’t work as expected We expect to get one of the following: • A=0, B=1 • A=1, B=0 • A=1, B=1 • But not A=0, B=0 … but TSO would permit it • Solution: Insert a memory barrier after write P0 P1 A=1; B=1; print B; print A;

  36. Origin 2000 • Intellectual descendant of Stanford DASH • Two processors per node • Caches use MESI protocol • Directory has 7 states: • Stable: unowned, shared, exclusive (cl/dirty in $) • Busy: Processor not ready to handle new requests to block, read, readex, uncached • Generally O2000 follows protocols discussed • Proves basic ideas actually apply • Shows that simplifying assumptions must be revisited to get a system built and deployed

  37. Summary • Shared memory support is much more difficult when there is no bus • A directory scheme achieves the same result, but the protocol requires a substantial number of messages, proportional to the amount of sharing • Coherency applies to individual locations • Consistent memory requires additional software or hardware to assure that updates or invalidations are complete

More Related