Logic and Lattices for Distributed Programming Neil Conway UC Berkeley

1. Logic and Lattices for Distributed Programming Neil ConwayUC Berkeley Joint work with:Peter Alvaro, Peter Bailis,David Maier, Bill Marczak,Joe Hellerstein, SriramSrinivasan Basho Chats #004June 27, 2012

2. Programming

3. Distributed Programming

4. Dealing with Disorder Introduce order • Paxos, Zookeeper, Two-Phase Commit, … • “Strong Consistency” Tolerate disorder • Correct behavior in the face of many possible network orders • Typical goal: replicas converge to same final state • “Eventual Consistency”

5. Eventual Consistency Popular Hard toprogram

6. Help developers buildreliable programs on top ofeventual consistency

7. This Talk • Theory • CRDTs, Lattices, and CALM • Practice • Programming with Lattices • Case Study: KVS

8. Write: {Alice, Bob, Carol} Read: {Alice, Bob} Students{Alice, Bob, Carol} Students{Alice, Bob} Client0 How to resolve? Write: {Alice, Bob, Dave} Read: {Alice, Bob} Students{Alice, Bob, Dave} Students{Alice, Bob} Client1

10. Students{Alice, Bob, Carol, Dave} Client0 Merge = Set Union Students{Alice, Bob, Carol, Dave} Client1

11. Commutative Operations • Used by Dynamo, Riak, Bayou, etc. • Formalized as CRDTs: Convergent and Commutative Replicated Data Types • Shapiro et al., INRIA (2009-2012) • Based on join semilattices • Commutative, associative, idempotent • Practical libraries: Statebox, Knockbox

12. Time “Growth”: Larger Sets “Growth”: Larger Numbers “Growth”: false  true Set (Union) Integer(Max) Boolean(Or)

13. Students{Alice, Bob, Carol} Students{Alice, Bob, Carol, Dave} Read: {Alice, Bob, Carol, Dave} Client0 Teams{<Alice, Bob>} Teams{<Alice, Bob>,<Carol, Dave>} Teams{<Alice, Bob>,<Carol, Dave>} Read: {<Alice,Bob>} Write: {<Alice,Bob>, <Carol,Dave>} Replica Synchronization Students{Alice, Bob, Carol} Students{Alice, Bob, Carol, Dave} Remove: {Dave} Client1 Teams{<Alice, Bob>,<Carol, Dave>} Teams{<Alice, Bob>} Teams{<Alice, Bob>,<Carol, Dave>}

14. Students{Alice, Bob, Carol, Dave} Students{Alice, Bob, Carol} Read: {Alice, Bob, Carol} Client0 Teams{<Alice, Bob>} Teams{<Alice, Bob>} Read: {<Alice,Bob>} Replica Synchronization Nondeterministic Outcome! Students{Alice, Bob, Carol} Students{Alice, Bob, Carol, Dave} Remove: {Dave} Client1 Teams{<Alice, Bob>} Teams{<Alice, Bob>}

15. Possible Solution:Wrap both replicated valuesina single complex CRDT

16. Goal:Compose larger applicationusing “safe” mappingsbetween simple lattices

17. Time Monotone functionfrom set  max Monotone functionfrom max boolean size() >= 5 Set(merge = Union) Integer(merge = Max) Boolean(merge = Or)

18. Monotonicity in Practice “The more you know, the more you know” Never retractprevious outputs(“mistake-free”) • Typical patterns: • immutable data • accumulate knowledge over time • threshold tests (“if” w/o “else”)

19. Monotonicity and Determinism Agents strictly learn more knowledge over time Monotone: different learning order, same final outcome Result:Program is deterministic!

20. A program is confluentif it produces the same results regardless of network nondeterminism

21. A program is confluentif it produces the same results regardless of network nondeterminism

22. CALM Analysis All monotone programs are confluent Simple syntactic test for monotonicity Result: Simple static analysis for eventual consistency ConsistencyAs LogicalMonotonicity

23. Handling Non-Monotonicity … is not the focus of this talk  Basic choices: • Nodes agree on an event order using a coordination protocol (e.g., Paxos) • Allow non-deterministic outcomes • If needed, compensate and apologize

24. Putting It Into Practice What we’d like: • Collection of agents • No shared state( message passing) • Computation over arbitrary lattices

27. Quorum Vote in BloomL QUORUM_SIZE=5 RESULT_ADDR="example.org" classQuorumVote includeBud state do channel :vote_chn, [:@addr, :voter_id] channel :result_chn, [:@addr] lset:votes lmax:vote_cnt lbool:got_quorum end bloom do votes <=vote_chn {|v|v.voter_id} vote_cnt<=votes.size got_quorum<=vote_cnt.gt_eq(QUORUM_SIZE) result_chn<~got_quorum.when_true { [RESULT_ADDR] } end end Annotated Ruby class Communication interfaces Program state Lattice state declarations Accumulate votesinto set Map set ! max Map max !bool Program logic Merge function for set lattice Threshold test on bool

28. Builtin Lattices

29. Case Study

30. Goal:Provably eventually consistentkey-value store (KVS) Assumption:Map keys to lattice values (i.e., values do not decrease) Solution:Use a map lattice

31. Time Nested lattice value Replica 2 Replica 1

32. Time Add new K/V pair Replica 2 Replica 1

33. Time “Grow” value in extant K/V pair Replica 2 Replica 1

34. Time Replica Synchronization Replica 2 Replica 1

35. Goal:Provably eventually consistent KVS that stores arbitrary values Solution:Assign a version to eachkey-value pair Each replica stores increasingversions, not increasing values

36. Object Versions in Dynamo/Riak • Each KV pair has a vector clock version • Given two versions of a KV pair, prefer the one with the strictly greater version • If versions are incomparable, invoke user-defined merge function

37. Vector Clock:Map from node IDs  logical clocks Solution:Use a map lattice Logical Clock:Increasing counter Solution:Use an increasing-int lattice

38. Version-Value Pairs Pair = <fst, snd> Pair merge(Pair o) { if self.fst > o.fst: self elsifself.fst < o.fst: o else new Pair(self.fst.merge(o.fst), self.snd.merge(o.snd)) }

39. Time Replica 2 Replica 1

40. Time Version increase;NOT value increase Replica 2 Replica 1

41. Time R1’s version replaces R2’s version Replica 2 Replica 1

42. Time New version @ R2 Replica 2 Replica 1

43. Time Concurrent writes! Replica 2 Replica 1

44. Merge VC (automatically),value merge via user’s lattice(as in Dynamo) Time Replica 2 Replica 1

45. Lattice Composition in KVS

46. Conclusion

47. Questions Welcome Please try Bloom! http://www.bloom-lang.org Or: gem install bud

48. Backup Slides

49. Lattices hS,t,?iis a bounded join semi-latticeiff: • S is a partially ordered set • t is a binary operator (“least upper bound”) • For all x,y2 S, xty = z where x·Sz, y·Sz, and there is no z’ z2S such that z’ ·Sz. • Associative, commutative, and idempotent • ? is the “least” element in S(8x2S: ?t x= x) Example: increasing integers • S = Z, t = max, ? = -∞

50. Monotone Functions f : ST is a monotone function iff 8a,b2S : a·Sb) f(a)·Tf(b) Example: size(Set) !Increasing-Intsize({A, B}) = 2size({A, B, C}) = 3