Chapter 6 Consistency and Replication

1. Chapter 6Consistency and Replication Shyh-In Hwang in Yuan Ze University

2. Reasons for Replication • To increase the reliability of a system • Survive after one replica crashes • Better protection against corrupted data • Performance • Scalability • Numerous clients • The size of geographical area Shyh-In Hwang in Yuan Ze University

3. Price of Replication • Consistency of the replicas • When and how to carry out modifications to all copies Shyh-In Hwang in Yuan Ze University

4. Object Replication (1) • Organization of a distributed remote object shared by two different clients. Shyh-In Hwang in Yuan Ze University

5. Object Replication (2) How to protect the object against simultaneous access by multiple clients • A remote object capable of handling concurrent invocations on its own. • An object adapter is required to handle concurrent invocations Shyh-In Hwang in Yuan Ze University

6. Object Replication (3) • A distributed system for replication-aware distributed objects. • A distributed system responsible for replica management (More common. Simplicity for AP developers) Shyh-In Hwang in Yuan Ze University

7. Data-Centric Consistency Models • The general organization of a logical data store, physically distributed and replicated across multiple processes. Shyh-In Hwang in Yuan Ze University

8. A contract between the processes and the data store If the processesagree to obey certain rules, the store promises to work correctly. Consistency Models Shyh-In Hwang in Yuan Ze University

9. Wi(x) a A write by process Pi to data item x with the value a Ri(x) b A read from data item x by process Pi returning the value b Data items are initially NIL Denotation Shyh-In Hwang in Yuan Ze University

10. Strict Consistency • The most stringent consistency model • Definition • Any read on a data item x returns a value corresponding to the result of the most recent write on x. Shyh-In Hwang in Yuan Ze University

11. P1 P2 Strictly consistent memory Not strictly consistent R(x)a R(x)a R(x)NIL Strict Consistency W(x)a Shyh-In Hwang in Yuan Ze University

12. Strict Consistency P2 P1 3 meters Machine A Machine B T2: R2(x) T1: W1(x), where T2 = T1 + 1ns C = 3 * 108 m/sec Propagation speed: 3 m / 10-9s = 3 * 109 m/sec In order to meet the strict consistency requirement Shyh-In Hwang in Yuan Ze University

13. Strict consistency is an ideal programming model; however, it’s nearly impossible to implement in a distributed system. Strict consistency may violate the laws of physics (Einstein’s special theory of relativity) It is impossible in a distributed system to assign a unique timestamp to each operation that corresponds to actual global time Strict Consistency Shyh-In Hwang in Yuan Ze University

14. Programmers can often live with weaker models. • When the order of events is essential, semaphores or other synchronization tools should be used. Shyh-In Hwang in Yuan Ze University

15. Definition(Lamport, 1979) The result of any execution is the same as if the (read and write) operations by all processes on the data store were executed in some sequential order, and the operations of each individual process appear in this sequence in the order specified by its program. Sequential Consistency Shyh-In Hwang in Yuan Ze University

16. That is, when processes run concurrently on (possibly) different machines, any valid interleaving of read and write operations is acceptable behavior, butall processes see the same interleaving of operations. Nothing is said about time. There is no reference to the “most recent” write operation on an object. A process sees writes from all processes but only its own reads. Sequential Consistency Shyh-In Hwang in Yuan Ze University

17. W(x)a P1 P1 R(x)NIL R(x)a P2 P2 W(x)a R(x)a R(x)a Sequential Consistency Two possible results of running the same program Shyh-In Hwang in Yuan Ze University

18. P1 P3 P2 P4 Sequential Consistency W(x)a W(x)b R(x)b R(x)a R(x)b R(x)a Sequentially consistent data store Shyh-In Hwang in Yuan Ze University

19. P1 P1 P2 P2 Sequential Consistency W(x)a W(x)b R(x)b R(x)a R(x)a R(x)b Not sequentially consistent Shyh-In Hwang in Yuan Ze University

20. Definition Definition of Sequential Consistency, plus If tsop1(x) < tsop2(y) , then operation OP1(x) should precede OP2(y) in this sequence Ordering according to a set of loosely synchronized clocks with only finite precision Stronger than sequential consistency Linearizability Shyh-In Hwang in Yuan Ze University

21. Linearizability • Why linearizability? • To assist formal verification of concurrent algorithms Shyh-In Hwang in Yuan Ze University

22. Sequential Consistency Example: consider the following three parallel processes, assume that all statements are atomic. How many possible execution sequences are there? Each statement is assumed to be indivisible Shyh-In Hwang in Yuan Ze University

23. Sequential Consistency 6! / 2 / 2 / 2 = 90 x, y, and z are initially 0. Shyh-In Hwang in Yuan Ze University

24. Four Valid Results Accepted by Sequential Consistency P3 must complete beforeP1 starts P1 executes both statements beforeP2 orP3 starts Shyh-In Hwang in Yuan Ze University

25. Sequential Consistency • Signature: a 6-bit string of the output of P1, P2, P3 in that order, which characterizes a particular interleaving of statements • 90 different valid statement orderings • less than 64 valid program results under sequential consistency • 000000 is not permitted • 001001 is not permitted Shyh-In Hwang in Yuan Ze University

26. To Express Sequential Consistency • Ahamad et all (1993) • E1: W1(x)a (execution on x) • E2: W2(x)b • E3: R3(x)b, R3(x)a • E4: R4(x)b, R4(x)a • H: history • Program order must be maintained • Data coherence must be respected • R(x) must return the value most recently written to x (指排序非指時間) • H = W2(x)b, R3(x)b, R4(x)b, W1(x)a, R3(x)a, R4(x)a Shyh-In Hwang in Yuan Ze University

27. Problem of sequential consistency: Poor performance (Lipton & Sandberg, 1988) r: read time w: write time t: minimal packet transfer time between nodes r + w  t For any sequentially consistent memory, changing the protocol to improve the read performance makes the write performance worse, and vice versa. Sequential Consistency Shyh-In Hwang in Yuan Ze University

28. Causal Consistency Consider USENET posting example: The “answer message” comes before the “question message,” thus violate the causal consistency rules. Examples When there is any read followed later by any write, the two events are potentially causally related. A read is causally related to the write that provided the data the read got. If two processes spontaneously and simultaneously write two variables, these are not causally related. Operations that are not causally related are said to be concurrent. Shyh-In Hwang in Yuan Ze University

29. Causal Consistency Hutto and Ahamad, 1990, a causally consistent memory should obey following condition: Writes that are potentiallycausally related must be seen by all processesin the same order. Concurrent writes may be seen in a different order on different machines. Overhead: implementing causal consistency requires keeping track of which processes have seen which writes. Shyh-In Hwang in Yuan Ze University

30. Causal Consistency P1 W(x)a W(x)c P2 R(x)a W(x)b concurrent P3 R(x)a R(x)c R(x)b R(x)a R(x)b R(x)c P4 This sequence is allowed with causally consistent memory, but not with sequentially consistent memory or strictly consistent memory. Shyh-In Hwang in Yuan Ze University

31. Causal Consistency Potentially causally related W(x)a P1 Wrong order R(x)a W(x)b P2 R(x)b R(x)a P3 R(x)a R(x)b P4 A violation of causal memory Shyh-In Hwang in Yuan Ze University

32. W(x)a P1 W(x)b P2 R(x)b R(x)a P3 R(x)a R(x)b P4 Causal Consistency Concurrent A correct sequence of events in causal memory(concurrent writes can be seen in a different order on different machines) Shyh-In Hwang in Yuan Ze University

33. PRAM (Pipelined RAM) Consistency Writes done by a single process are seen by all other processes in the order in which they were issued, but writes from different processes may be seen in a different order by different processes. Easy to implement Require that writes originating in a single process be seen everywhere in order. All writes generated by different processes are concurrent FIFO Consistency Shyh-In Hwang in Yuan Ze University

34. FIFO Consistency • Key difference between sequential & PRAM consistency • Sequential consistency: although the order of statement execution and memory references is non-deterministic, at least all processes agree what it is; • PRAM consistency: they need not agree Shyh-In Hwang in Yuan Ze University

35. FIFO Consistency P1 W(x)a R(x)a W(x)b W(x)c P2 R(x)b R(x)a P3 R(x)c P4 R(x)a R(x)b R(x)c A valid sequence of events for PRAM consistency Shyh-In Hwang in Yuan Ze University

36. FIFO Consistency Seen by P1 Seen by P2 Seen by P3 001001 is impossible with sequential consistency Shyh-In Hwang in Yuan Ze University

37. P1: P2: x = 1; if (y == 0) kill (P2) y = 1; if (x == 0) kill (P1) FIFO Consistency • Consider two parallel processes P1 & P2 running under different consistency model • Sequential consistent: P1 is killed, P2 is killed, or neither is killed • PRAM consistent: Both processes can be killed Shyh-In Hwang in Yuan Ze University

38. Problem of FIFO Consistency • Not all applications require even seeing all writes, let alone seeing them in order • E.g.: a process inside a critical section reading and writing some variables in a tight loop. • Other processes are not supposed to touch the variables until the first process has left its critical section • The memory has no way of knowing when a process is in a critical section and when it is not, so it has to propagate all writes to all memories in the usual way • A better solution is to let the process finish its critical section and then make sure that the final results were sent everywhere. => Synchronization variable Shyh-In Hwang in Yuan Ze University

39. Synchronization variable used to synchronize memory When a synchronization completes, all writes done on that machine are propagated outward and all writes done on other machines are brought in. All of shared memory is synchronized Weak Consistency Shyh-In Hwang in Yuan Ze University

40. Weak Consistency • Property 1: Accesses to synchronization variables associated with a data store are sequentially consistent • All processes see all accesses to synchronization variables in the same order. Shyh-In Hwang in Yuan Ze University

41. Weak Consistency • Property 2: No operation on a synchronization variable is allowed to be performed until all previous writes have been completed everywhere • Accessing a synchronization variable flushes the pipeline (writes). Shyh-In Hwang in Yuan Ze University

42. Weak Consistency • Property 3: No read or write operation on data items are allowed to be performed until all previous operations to synchronization variables have been performed. • By doing a synchronization before reading shared data, a process can be sure of getting the most recent values. Shyh-In Hwang in Yuan Ze University

43. Most useful when isolated accesses to shared variables are rare, with most coming in clusters (many accesses in a short period, then none for a long time). We limit only the time when consistency holds With Weak Consistency, Sequential consistency is enforced on groups of operations instead of on individual operations. Weak Consistency Shyh-In Hwang in Yuan Ze University

44. Weak Consistency int a, b, c, d, e, x, y; /* variables */int *p, *q; /* pointers */int f( int *p, int *q); /* function prototype */ a = x * x; /* a stored in register */b = y * y; /* b as well */c = a*a*a + b*b + a * b; /* used later */d = a * a * c; /* used later */p = &a; /* p gets address of a */q = &b /* q gets address of b */e = f(p, q) /* function call */ • Having memory be wrong is acceptable. Only when the function f is called does the compiler have to put the current values of a and b back in memory. Shyh-In Hwang in Yuan Ze University

45. W(x)a W(x)b S P1 R(x)a R(x)b S P2 R(x)b R(x)a S P3 W(x)a W(x)b S P1 S R(x)a P2 Weak Consistency A valid sequence of events for weak consistency Should be b An invalid sequence of events for weak consistency Shyh-In Hwang in Yuan Ze University

46. Gharachorloo et al., 1990 Two kinds of synchronization variables or operations are provided instead of one in weak consistency. acquire: a critical region is about to be entered. release: a critical region has just been exited. Able to tell the difference between entering/leaving a critical region, thus more efficient than weak consistency. Acquire & release do not have to apply to all of memory. Instead, they may only guard specific shared variables. The shared variables that are kept consistent are said to be protected. Release Consistency Shyh-In Hwang in Yuan Ze University

47. Acq(L) W(x)a W(x)b Rel(L) P1 Acq(L) R(x)b Rel(L) P2 R(x)a P3 Release Consistency A valid event sequence for release consistency Shyh-In Hwang in Yuan Ze University

48. It is also possible to use barriers instead of critical regions with release consistency A barrier is a synchronization mechanism that prevents any process from starting phase n + 1 of a program until all processes have finished phase n. When a process arrives at a barrier, it must wait until all other processes get there too. When the last one arrives, all shared variables are synchronized and then all processes are resumed. Departure from the barrier is done on an acquire and arrival is done on a release. Release Consistency Shyh-In Hwang in Yuan Ze University

49. When the software does an acquire: All the local copies of the protected variables are brought up to date to be consistent with the remote ones if need be. Doing an acquire does not guarantee that locally made changes will be sent to other machines immediately When the release is done: Protected variables that have been changed are propagated out to other machines. Doing a release does not necessarily import changes from other machines. Release Consistency Shyh-In Hwang in Yuan Ze University

50. Release Consistency • Rules: • Before a read or write operation on shared data is performed, all previous acquires done by the process must have completed successfully. • Before a release is allowed to be performed, all previous reads and writes by the process must have completed • Accesses to synchronization variables are FIFO consistent (sequential consistency is not required). Shyh-In Hwang in Yuan Ze University