Consistency and Replication

1. Consistency and Replication Chapter 7

2. Giving credit where credit is due: CSCE455/855Distributed Operating Systems • Most of the lecture notes are based on slides by • Prof. Jalal Y. Kawash at Univ. of Calgary • Prof. Kenneth Chiu at SUNY Binghamton • Prof. Sanjeev Setia at George Mason University • I have modified them and added new slides

3. Consistency and Replication Chapter 7 Part I Consistency Models

4. Reasons for Replication • Reliability: • Mask failures • Mask corrupted data • Performance: • Scalability (size and geographical) • Examples: • Web caching • Horizontal server distribution

5. Cost of Replication ? • Replicas must be kept consistent Dilemma: • Replicate data for better performance • Modification on one copy triggers modifications on all other replicas • Propagating each modification to each replica can degrade performance

6. Consistency Issues – Access/Update Ratio User accesses to the page … Updates to the Web page time

7. Consistency Model • When and how the modifications are made = consistency model: • Weak versus strong consistency model

8. Consistency Models (cont.) The general organization of a logical data store, physically distributed and replicated across multiple processes.

9. Consistency Models (cont) read2 read1 • A process performs a read operation on a data item, expects the operation to return a value that shows the result of the last write operation on that data • No global clock  difficult to define the last write operation • Consistency models provide other definitions • Different consistency models have different restrictions on the values that a read operation can return

10. Summary of Consistency Models • Consistency models not using synchronization operations. • Models with synchronization operations.

11. Framework for Consistency Partial and Total Orders Let S be a set, and R  S  S • R is anti-reflexive if x  S, (x,x)  R • R is transitive if x, y, z  S, if (x,y)  R and (y,z)  R then (x,z)  R • A PO is an anti-reflexive, transitive relation • A PO is denoted by (S,R) • xRy means (x,y)  R • A TO is a PO (S,R) such that x, y S x  y, either xRy or yRx

12. Framework for Consistency Operations and Data Items • Operations are either writes or reads • A write is denoted wp(x)v • A read is denoted rp(x)v • A read-write data item is the set of all sequences <o1, o2, … on> such that • Each oi is either a read or a write • Each read returns the same value written by the most recent preceding write in the sequence

13. Framework for Consistency Operations and Processes • Each operation can be decomposed into two components: • Invocation and response • wp(x)v: invocation = wp(x)v; response = empty • rp(x)v: invocation = rp(x)?; response = v • A process is a sequence of operation invocations • A process computation is a sequence of operations obtained by augmenting each invocation in the process by its response

14. Framework for Consistency Multiprocess Systems • A (multiprocess) system (P,D) is a set of processes, P, and a set of data items, D, such that all operation invocations of processes in P are applied to items in D • A (multiprocess) system (P,D) computation is a collection of process computations one for each process in P

15. Framework for Consistency Example Program p: x = y Program q: y = x System (P,D): P = {p,q} D = {x,y} Process p: r(y)v? w(x)v? Process q: r(x)v? w(y)v? System (P,D) Computation: p: r(y)5 w(x)5 q: r(x)0 w(y)0 Process p Comp: r(y)5 w(x)5 Process q Comp: r(x)0 w(y)0

16. Framework for Consistency Program Order Program p: x = y Program q: y = x • rp(y)5 <po wp(x)5 • rq(x)0 <po wq(y)0 • All of program order for the example Process p: r(y)v? w(x)v? Process q: r(x)v? w(y)v? Process p Comp: r(y)5 w(x)5 Process q Comp: r(x)0 w(y)0 • Define program order, denoted (O, <po), by o1<po o2 iff o2 follows o1 in p’s computation

17. Framework for Consistency Consistency Models • A consistency model is a set of constraints on system computations • A system computation of (P,D) satisfies a consistency model CM if the computation meets all the constraints in CM

18. Relation of Consistency Models • Sequential All processes see all shared accesses in the same order • Strict Absolute time ordering of all shared accesses matters. • For two consistency models CM1 and CM2 CM1 is stronger than CM2 if the constraints of CM1 imply those of CM2 • CM2 is weaker than CM1 • Sequential consistency is weaker than strict consistency

19. Framework for Consistency – Validity • Given a set of operations O • O|w indicates all the write operations in O • O|r indicates all the read operations in O • O|p is the subset of O containing p’s operations, for some process p • O|x is the subset of O containing operations on x, for some data item x • Let (O,<) be a total order of O • (O,<) is valid if for each data item x, the subsequence (O|x,<) is valid for x

20. Framework for Consistency Valid Total Orders Valid for x: rq(x)0 wq(y)5 wp(x)5 rq(x)5 rp(y)5 Valid for y: rq(x)0 wq(y)5 wp(x)5 rq(x)5 rp(y)5 Computation: p: w(x)5 r(y)5 q: r(x)0 w(y)5 r(x)5 x and y are initially 0 Valid Total Order: rq(x)0 wq(y)5 wp(x)5 rq(x)5 rp(y)5 Invalid Total Order: wp(x)5 rq(x)0 wq(y)5 rq(x)5 rp(y)5

21. Sequential Consistency (SC) • Two constraints: • the result of any execution is the same as if the operations of all the processes were executed in some sequential order, and • the operations of each individual process appear in this sequence in the order specified by its program • Let O be the set of all the operations of a computation C of a system (P,D). Then, C satisfies SC if there is a valid total order (O,<) such that (O,<po)  (O,<)

22. SC – Intuition … process process process FIFO Channels Switch All Data Items (the set D)

23. Sequential Consistency – Example p: w(x)1 r(x)2 q: r(x)1 w(x)2 C1 C1 satisfies SC (O,<) = <wp(x)1, rq(x)1, wq(x)2, rp(x)2> (O,<po) = { (wp(x)1, rp(x)2), (rq(x)1, wq(x)2) } • C satisfies SC if there is a valid total order (O,<) such that (O,<po)  (O,<)

24. Sequential Consistency – Examples p: w(x)1 r(x)2 q: w(x)2 r(x)1 p: w(x)1 w(y)2 q: r(y)2 r(x)0 C2 C3 C2 does not satisfy SC (O, <po) = { (wp(x)1, rp(x)2), (wq(x)2, rq(x)1) } <wp(x)1, wq(x)2, rp(x)2, rq(x)1> (is not valid) <wp(x)1, rq(x)1, wq(x)2, rp(x)2> (violates PO) Exercise: Does C3 satisfy SC? (x and y are initially 0)

25. Coherence [Goodman] • SC per data item • Let O be the set of all the operations of a computation C of a system (P,D). Then, C satisfies Coherence if for each x  D there is a valid total order (O|x,<x) such that (O|x,<po)  (O|x,<x)

26. Coherence – Intuition … process process process FIFO Channels … One Data Item One Data Item One Data Item

27. Coherence – Examples p: w(x)1 r(x)2 q: r(x)1 w(x)2 p: w(x)1 r(x)2 q: w(x)2 r(x)1 p: w(x)1 w(y)2 q: r(y)2 r(x)0 p: w(x)3 w(x)2 r(y)3 q: w(y)3 w(y)1 r(x)3 C1 C2 C3 C4 C1 satisfies Coherence (O|x,<x) = <wp(x)1, rq(x)1, wq(x)2, rp(x)2> C2 does not satisfy Coherence C3 satisfies Coherence but not SC Does C4 satisfy Coherence? SC?

28. SC versus Coherence C3 All Computations satisfying consistency model CM = C(CM) C(Coherence) • If Computation C satisfies SC, then it satisfies Coherence + PO • If a Computation C satisfies Coherence, then it does not necessarily satisfy SC • Proof: Computation C3 is an example C(SC)

29. FIFO [Lipton & Sandberg] • Writes done by a single process are seen by all other processes in the order in which they were issued, but writes from different processes may be seen in a different order by different processes • When would we like to use FIFO consistency model

30. Review: SC – Intuition … process process process FIFO Channels Switch All Data Items (the set D)

31. Review: Coherence – Intuition … process process process FIFO Channels … One Data Item One Data Item One Data Item

32. FIFO – Intuition process process process process All Data Items (D) All Data Items (D) All Data Items (D) All Data Items (D) FIFO Channels

33. FIFO [Lipton & Sandberg] • Writes done by a single process are seen by all other processes in the order in which they were issued, but writes from different processes may be seen in a different order by different processes • Let O be the set of all the operations of a computation C of a system (P,D). Then, C satisfies FIFO if for each p  P there is a valid total order (O|p  O|w,<p) such that (O|p  O|w,<po)  (O|p  O|w,<p)

34. FIFO – Examples p: w(x)1 r(x)2 q: r(x)1 w(x)2 p: w(x)1 r(x)2 q: w(x)2 r(x)1 p: w(x)1 w(y)2 q: r(y)2 r(x)0 C1 C2 C3 C1 satisfies FIFO (also SC and Coherence) (O|p  O|w,<p) = <wp(x)1, wq(x)2, rp(x)2> (O|q  O|w,<q) = <wp(x)1, rq(x)1, wq(x)2> C2 satisfies FIFO but not Coherence C3 satisfies Coherence but not SC nor FIFO

35. FIFO – Examples (cont) p: w(x)3 w(x)1 w(y)2 q: r(y)2 r(x)3 p: w(x)3 w(x)2 r(y)3 q: w(y)3 w(y)1 r(x)3 C5 C4 Does C4 satisfy FIFO? Coherence? SC? Does C5 satisfy FIFO? Coherence? SC?

36. SC versus FIFO • If Computation C satisfies SC, does it satisfy FIFO? • If Computation C satisfies FIFO, does it satisfy SC?

37. SC versus FIFO p: w(x)3 w(x)2 r(y)3 q: w(y)3 w(y)1 r(x)3 C4 C4 C(FIFO) • If Computation C satisfies SC, then it satisfies FIFO • If a Computation C satisfies FIFO, then it does not necessarily satisfy SC • Proof: Computation C4 is an example C(SC)

38. Coherence versus FIFO • If Computation C satisfies Coherence, does it satisfy FIFO?

39. Coherence versus FIFO p: w(x)3 w(x)1 w(y)2 q: r(y)2 r(x)3 C5 • If Computation C satisfies Coherence, then it does not necessarily satisfy FIFO • Proof: Computation C5 is an example

40. Coherence versus FIFO • If a Computation C satisfies FIFO, does it satisfy Coherence?

41. Coherence versus FIFO p: w(x)1 r(x)2 q: w(x)2 r(x)1 C2 • If a Computation C satisfies FIFO, then it does not necessarily satisfy Coherence • Proof: Computation C2 is an example

42. Coherence versus FIFO p: w(x)3 w(x)2 r(y)3 q: w(y)3 w(y)1 r(x)3 C4 C: satisfies FIFO and Coherence, but not SC C(FIFO) C(Coherence) C(SC) • There are computations that satisfy both Coherence and FIFO, but not SC • Proof: Computation C4

43. Weak Consistency • Consider Critical Section • If a process is in a critical section, its intermediate results of operations are not necessarily propagated to others. • Idea • Enforce consistency on a Group of Operations • Limit the time when consistency holds • Let programmer explicitly specify this

44. Synchronization Operations • In addition to reads and writes, introduce synchp() operation, which • synchronizes all local copies of the data store • Propagate local updates • Bring in other’s updates

45. Weak Consistency (cont.) • Three conditions • No operation on a synchronization variable is allowed to be performed until all previous writes have completed everywhere. • No read or write operation on data items are allowed to be performed until all previous operations to synchronization variables have been performed. • Accesses to synchronization variables associated with a data store, are sequentially consistent.

46. Weak Consistency (cont.) Weak Consistency Not Weak Consistency P3 P2 P3 P1 P2 P1 W(x, a) W(x, a) W(x, b) W(y, c) bR(x) W(y, c) W(x, b) Or aR(x) S3 NilR(y) S3 S1 S1 S2 S2 bR(x) aR(x) bR(x) cR(y) cR(y) cR(y) cR(y) bR(x) • No read or write operation on data items are allowed to be performed until all previous operations to synchronization variables have been performed. • No operation on a synchronization variable is allowed to be performed until all previous writes have completed everywhere.

47. WC – Example p: w(x)3 s() q: r(x)0 s() w(y)1 s’() r(x)3 m: w(x)5 r(y)1 s() r(x)3 C6 • Accesses to synchronization variables associated with a data store, are sequentially consistent. • All of p, q, and m must agree on a total order of synch operations consistent with program order; for example: • <sq(), sp(), s’q(), sm()> • (O|p  O|w  O|s , <p) = • < wm(x)5,sq(), wp(x)3, sp(), wq(y)1, s’q(), sm() > • (O|q  O|w  O|s, <q) = • < rq(x)0,wm(x)5,sq(), wp(x)3,sp(), wq(y)1, s’q(), sm(), rq(x)3> • (O|m  O|w  O|s, <m) = • < wm(x)5,sq(), wp(x)3, sp(), wq(y)1, s’q(), rm(y)1, sm(), rm(x)3 >

48. Summary of Consistency Models • Consistency models not using synchronization operations. • Models with synchronization operations.

49. Weaker Models • Sometimes strong models are needed, if the result of race conditions are very bad. • Banks • Sometimes the result of races are just inefficiency, or inconvenience, etc. • How strong is Orbitz’s model? • If it shows that a flight ticket with a certain price is available, is it really? • One kind of weaker model is eventual consistency • It eventually becomes consistent

50. Lazy Consistency Models • When updates are scarce • When updates are not conflicting • Examples: DNS and WWW • Eventual Consistency (EC): Lazy propagation of updates to all replicas • If no updates take place for a long time, all replicas will become consistent • Cheap to implement • If a client always accesses the same replica, the same or newer data will be read as time passes. EC works.