Shared Memory – Consistency of Shared Variables

1. CPU0 CPU1 CPU2 CPU3 Read/ Write Shared Memory Symmetric Multi-Processor (SMP): Distributed Shared Memory (DSM): CPU0 CPU1 CPU2 CPU3 CPU0 CPU1 CPU2 CPU3 Read/ Write Local Cache Local Cache Local Cache Local Cache Local Memory Module Local Memory Module Local Memory Module Local Memory Module R/W of Misses + Cache Invalidate Shared Memory Network Shared Memory – Consistency of Shared Variables The actual architecture of shared memory systems: The ideal picture of shared memory:

2. CPUi CPUj W V,x Write value x to local copy of shared variable V Read V from local copy R V,0? R V,x? The Million $$s Question:How/When Does One ProcessRead Other Process’s Writes? Assumption: Initial value of shared variables is always 0. Why is this a question? Because temporal order relations like “before/after” do not necessarily hold in a distributed system.

3. Non-Atomic writes/reads(also called loads/stores) • A read by Pi is considered performedwith respect to Pk at a point in time when the issuing of a write to the same address by Pk cannot affect the value returned by the read. • A write by Pi is considered performedwith respect to Pk at a point in time when an issued read to the same address by Pk returns the value defined by this write (or a subsequent write to the same location). • An access is performed when it is performed with respect to all processors. • A read is globally performed if it is performed and if the write that is the source of the returned value has been performed. In what follows, we will think of atomic read/write but these definitions can be used to generalize.

4. a=0,b=0 Print(b) Print(a) a=1 b=1 Printed:0,0? Printed:1,0? Printed:1,1? Why Memory Model? Answers the question: “Which writes by a process are seen by which reads of the other processes?”

5. Example execution: Pi: R V,0; W V,7; R V,7; R V,13 Pj: R V,0; W V,13; R V,13; R V,7 Order of writes to V as seen to Pi: (1) W V,7; (2) W V,13 Order of writes to V as seen to Pj: (1) W V,13; (2) W V,7 Memory Consistency Models Example program: Pi: R V; W V,7; R V; R V Pj: R V; W V,13; R V; R V A consistency/memory model is an “agreement” between the execution environment (H/W, OS, middleware) and the processes. Runtime guarantees to the application certain properties on the way values written to shared variables become visible to reads. This determines the memory model, what’s valid, what’s not.

6. Pi: W V,7 R V,7 R V,7 Pj: W V,13 R V,13 R V,7 Pi: W V,7 R V,7 R V,7 Pj: W V,13 R V,7 R V,7 Pi: W V,7 R V R V Pj: W V,13 R V R V Pi: W V,7 R V,7 R V,13 Pj: W V,13 R V,13 R V,13 Pi: W V,7 R V,13 R V,13 Pj: W V,13 R V,13 R V,13 Pi: W V,7 R V,7 R V,7 Pj: W V,13 R V,13 R V,13 Memory Model: Coherence Example program: All valid executions under Coherence: Coherence is the memory model in which (the runtime guarantees to the program that) writes performed by the processes for every specific variable are viewed by all processes in the same full order. Note: the view of a process consists of the values it “sees” in its reads, and the writes it performs. Thus, if a R V in P which is later than a W V,x in P sees a value different than x, then a later R V cannot see x.

7. Formal definition of Coherence Program Order: The order in which instructions appear in each process. This is a partial order on all the instructions in the program. A serialization:A full order on all the instructions (reads/writes) of all the processes, which is consistent with the program order. A legal serialization:A serialization in which eachreadXreturns the value written by the latestwriteXin the full order. Let P be a program; let PX be the “sub-program” of P which contains all the read X/write Xoperations on X only. Coherence:Pis said to be coherent if for every variableXthere exists a legal serialization of PX. (Note: a process cannot distinguish one such serialization from another for a given execution)

8. Process 1 read x,1 write y,1 Process 1 write x,1 write x,2 Process 2 read x,2 read x,1 Process 2 read y,1 write x,1 Process 2 read y,1 write x,1 Examples Coherent. Serializations: x: write x,1, read x,1 y: write y,1, read y,1 Process 1 read x,1 write x,2 Process 2 read x,2 write x,1 Not Coherent. Cycle of dependencies. Cannot be serialized. Not Coherent. Cannot be serialized.

9. Process 1 read x,1 write y,1 Process 2 read y,1 write x,1 Process 1 write x,1 write y,1 Process 2 read y,1 read x,0 Sequential Consistency [Lamport 1979] Coherent. Not Sequentially consistent. Sequential Consistency is the memory model in which all reads/writes performed by the processes are viewed by all processes in the same full order. Coherent. Not Sequentially consistent.

10. a=0,b=0 Print(b) Print(a) a=1 b=1 Printed:0,0 or 0,1 or 1,0 Printed:1,1 Strict (Strong) Memory Models Sequential Consistency: Given an execution, there exists an order of reads/writes which is consistent with all program orders. Coherence: For any variable x, there exists an order ofread x/write x consistent with all p.o.s.

11. Formal definition of Sequential Consistency Observation: Every program which is sequentially consistent is also coherent. Conclusion: Sequential Consistency has stronger requirements and we thus say that it is stronger than Coherence. In general: A consistency model A is said to be (strictly) stronger than B if all executions which are valid under A are also valid under B. Let P be a program. Sequential Consistency:Pis said to be sequentially consistent ifthere exists a legal serialization of all reads/writes in P.

12. P2: Print(V) Write U,1 P1: Print(U) Write V,1 The problem of strong consistency models The runtime system should ensure the existence of legal serialization, and the same consistent view for all processes. This requires lots of expensive coordination  degrades performance! SC: Hardware cannot reorder locally in each thread for this will result in a possible printing 1,1. HW may reorder anyway and postpone writes, but then why reorder in the first place?

13. p.x = 0 a=p.x p.x=1 b=q.x assert(ab) Coherence Forbids Reordering q.x is aliased to p.x. Reordering may make assignment to B early (seeing 0) and that to A late (seeing 1). The right thread see order of writes different from left thread. Once thread sees an update – cannot “forget” it has seen it. • Cannot reorder two reads of the same memory location.

14. p and q might point to same object p.x = 0 a=p.x p.x=1 b=q.x reads can make a process see writes by another process. The read “kills” later reuse of local values. c=p.x Cannot put c=a assert(p==q  a  b  c) Coherence makes reads preventcommon compiler optimizations

15. Causal Consistency If event B is caused or influenced by an earlier event A, Causal Consistency requires that all the processes first see A, then see B. Causally related events: Events A and B are causally related if A causes or influences B. Concurrent events: Events A and B are concurrent/independent if they are not causally related. Formally: Write After Read: When a Readoperation is followed in program order by a Write operation, then the two events are potentially causally related. Read After Write: A Read(x) operation is causally related to the Write(x) operation that provided the data the Read got. Transitive closure: when there is a C so A is [potentially] causally related to C and C is [potentially] causally related to B then A is causally related to B. Casual Consistency: An execution obeys causal consistency if Writes that are potentially casually related are seen by all processes in the same order. Concurrent writes may be seen in a different order on different machines.

16. Example – Causally Consistent • W2(x),2 and W1(x),3 are concurrent events (so it is not required that all processes see them in the same order). • The only writes that are potentially causally related are W1(x),1and W2(x),2. • The above sequence of events is Causally Consistent but not Sequentially Consistent (the readings of P3 – R3(x),3 and R3(x),2 are not allowed in a sequentially consistent memory model).

17. Example – Causally Inconsistent • W2(x),2is potentially causally related to W1(x),1(because the writing of 2 may result from the value read by R2(x),1). • Since the two writes are potentially causally related all processes must see them in the same order, however, P3 and P4 see them in a different order.

18. Example – Causally Consistent • W2(x),2and W1(x),1are concurrent events (the read R2(x),1 was removed). • Since the two writes are concurrent they may be seen in a different order by different processes.