Replicated Data Consistency Explained Through Baseball

1. Replicated Data Consistency Explained Through Baseball slides by Landon Cox with some others from elsewhere prepended and appended (cultural history lesson)

2. Preview/overview • K-V stores are a common data tier for mega-services. • They evolved during the Web era, starting with DDS. • Today they often feature geographic replication. • Multiple replicas in different data centers. • Geo-replication offers better scale and reliability/availability. • But updates are slower to propagate, and network partitions may interfere. • A read might not see the “latest” write. • So we have to think carefully about what consistency properties we need: “BASE” might be “good enough”. • FLP and CAP tell us that there are fundamental limits on what we can guarantee….but many recent innovations in this space.

3. Over the next couple of years, Amazon transformed internally into a service-oriented architecture. They learned a tremendous amount… - pager escalation gets way harder….build a lot of scaffolding and metrics and reporting. - every single one of your peer teams suddenly becomes a potential DOS attacker. Nobody can make any real forward progress until very serious quotas and throttling are put in place in every single service. - monitoring and QA are the same thing. You'd never think so until you try doing a big SOA. But when your service says "oh yes, I'm fine", it may well be the case that the only thing still functioning in the server is the little component that knows how to say "I'm fine, roger roger, over and out" in a cheery droid voice. In order to tell whether the service is actually responding, you have to make individual calls. The problem continues recursively until your monitoring is doing comprehensive semantics checking of your entire range of services and data, at which point it's indistinguishable from automated QA. So they're a continuum. - if you have hundreds of services, and your code MUST communicate with other groups' code via these services, then you won't be able to find any of them without a service-discovery mechanism. And you can't have that without a service registration mechanism, which itself is another service. So Amazon has a universal service registry where you can find out reflectively (programmatically) about every service, what its APIs are, and also whether it is currently up, and where. - debugging problems with someone else's code gets a LOT harder, and is basically impossible unless there is a universal standard way to run every service in a debuggable sandbox. That's just a very small sample. There are dozens, maybe hundreds of individual learnings like these that Amazon had to discover organically. There were a lot of wacky ones around externalizing services, but not as many as you might think. Organizing into services taught teams not to trust each other in most of the same ways they're not supposed to trust external developers. This effort was still underway when I left to join Google in mid-2005, but it was pretty far advanced. From the time Bezos issued his edict through the time I left, Amazon had transformed culturally into a company that thinks about everything in a services-first fashion. It is now fundamental to how they approach all designs, including internal designs for stuff that might never see the light of day externally. Key-value stores • Many mega-services are built on key-value stores. • Store variable-length content objects: think “tiny files” (value) • Each object is named by a “key”, usually fixed-size. • Key is also called a token: not to be confused with a crypto key! Although it may be a content hash (SHAx or MD5). • Simple put/get interface with no offsets or transactions (yet). • Goes back to literature on Distributed Data Structures [Gribble 2000] and Distributed Hash Tables (DHTs). [image from Sean Rhea, opendht.org, 2004]

4. ACID vs. BASE Eric Brewer ACM SIGOPS Mark Weiser Award 2009 Jim Gray ACM Turing Award 1998

5. ACID Strong consistency Isolation Focus on “commit” Nested transactions Availability? Conservative (pessimistic) Difficult evolution(e.g. schema) “small” Invariant Boundary The “inside” BASE Weak consistency stale data OK Availability first Best effort Approximate answers OK Aggressive (optimistic) “Simpler” and faster Easier evolution (XML) “wide” Invariant Boundary Outside consistency boundary ACID vs. BASE but it’s a spectrum

6. Dr. Werner Vogels is Vice President & Chief Technology Officer at Amazon.com. Prior to joining Amazon, he worked as a researcher at Cornell University.

7. Vogels on consistency The scenario A updates a “data object” in a “storage system”. Consistency “has to do with how observers see these updates”. Strong consistency: “After the update completes, any subsequent access will return the updated value.” Eventual consistency: “If no new updates are made to the object, eventually all accesses will return the last updated value.”

8. Brian F. Cooper, Raghu Ramakrishnan, Utkarsh Srivastava, Adam Silberstein, Philip Bohannon, Hans-Arno Jacobsen, Nick Puz, Daniel Weaver and Ramana Yerneni Yahoo! Research PNUTS: Yahoo!’s Hosted Data Serving Platform

9. What are my friends up to? Sonja: Brandon: Example: social network updates Brian Sonja Jimi Brandon Kurt

10. Example: social network updates 6 Jimi <ph.. 8 Mary <re.. 12 Sonja <ph.. 15 Brandon <po.. 16 Mike <ph.. <photo> <title>Flower</title> <url>www.flickr.com</url> </photo> 17 Bob <re..

11. Asynchronous replication

12. Consistency model • Goal: make it easier for applications to reason about updates and cope with asynchrony • What happens to a record with primary key “Brian”? Record inserted Delete Update Update Update Update Update Update Update v. 2 v. 5 v. 1 v. 3 v. 4 v. 6 v. 7 v. 8 Time Time Generation 1

13. Consistency model Read Stale version Current version Stale version v. 2 v. 5 v. 1 v. 3 v. 4 v. 6 v. 7 v. 8 Time Generation 1

14. Consistency model Read up-to-date Stale version Current version Stale version v. 2 v. 5 v. 1 v. 3 v. 4 v. 6 v. 7 v. 8 Time Generation 1

15. Consistency model Read-critical(required version): Read ≥ v.6 Stale version Current version Stale version v. 2 v. 5 v. 1 v. 3 v. 4 v. 6 v. 7 v. 8 Time Generation 1

16. Consistency model Write if = v.7 Test-and-set-write(required version) ERROR Stale version Current version Stale version v. 2 v. 5 v. 1 v. 3 v. 4 v. 6 v. 7 v. 8 Time Generation 1

17. Don’t Settle for Eventual: Scalable Causal Consistency for Wide-Area Storage with COPS Wyatt Lloyd* Michael J. Freedman* Michael Kaminsky† David G. Andersen‡ *Princeton, †Intel Labs, ‡CMU

18. Wide-Area Storage Stores: Status Updates Likes Comments Photos Friends List Stores: Posts +1s Comments Photos Circles Stores: Tweets Favorites Following List

19. Wide-Area Storage Serves Requests Quickly

20. Inside the Datacenter Web Tier Storage Tier Web Tier Storage Tier Replication A-F A-F Remote DC G-L G-L M-R M-R S-Z S-Z

21. Desired Properties: ALPS • Availability • Low Latency • Partition Tolerance • Scalability “Always On”

22. Scalability A-Z A-Z Increase capacity and throughput in each datacenter A-C M-O M-O A-F A-C A-F P-S A-L G-L A-L D-F D-F P-S G-L G-J T-V M-Z M-Z M-R T-V M-R G-J W-Z K-L S-Z K-L S-Z W-Z

23. Desired Property: Consistency • Restricts order/timing of operations • Stronger consistency: • Makes programming easier • Makes user experience better

24. Consistency with ALPS Strong Sequential Causal Eventual Impossible [Brewer00, GilbertLynch02] Impossible [LiptonSandberg88, AttiyaWelch94] COPS Amazon LinkedIn Facebook/Apache Dynamo Voldemort Cassandra

26. Replicated-data consistency • A set of invariants on each read operation • Which writes are guaranteed to be reflected? • What write orders are guaranteed? • Consistency is an application-level concern • When consistency is too weak, applications break • Example: auction site must not tell two people they won • What are consequences of too-strong consistency? • Worse performance (for reads and writes) • Worse availability (for reads and writes)

27. The following are slides on the Doug Terry paper by Landon Cox. • We went through these pretty fast in class, but you should understand these models and why we might use them.

28. Assumptions for our discussion • Clients perform reads and writes • Data is replicated among a set of servers • Writes are serialized (logically, one writer) • Performed in the same order at all servers • Write order consistent with write-request order • Reads result of one or more past writes

29. Consistency models • Strong consistency • Reader sees effect of all prior writes • Eventual consistency • Reader sees effect of subset of prior writes • Consistent prefix • Reader sees effect of initial sequence of writes • Bounded staleness • Reader sees effect of all “old” writes • Monotonic reads • Reader sees effect of increasing subset of writes • Read my writes • Reader sees effect of all writes performed by reader

30. Setting: baseball game Write (“visitors”, 0); Write (“home”, 0); for inning = 1..9 outs = 0; while outs < 3 visiting player bats; for each run scored score = Read (“visitors”); Write (“visitors”, score + 1); outs = 0; while outs < 3 home player bats; for each run scored score = Read (“home”); Write (“home”, score + 1); end game; Primary game thread. Only thread that issues writes.

31. Visitors’ score Home score S1 S2 S3 S4 S5 S6 V V V H H H W R W R R R R Reader Writer (also reads) Reader

32. Example 1: score keeper score = Read (“visitors”); Write (“visitors”, score + 1); … score = Read (“home”); Write (“home”, score + 1);

33. Example 1: score keeper What invariant is the score keeper maintaining on the game’s score? Write (“home”, 1); Write (“visitors”, 1); Write (“home”, 2); Write (“home”, 3); Write (“visitors”, 2); Write (“home”, 4); Write (“home”, 5); Both values increase monotonically Visitors = 2 Home = 5

34. Example 1: score keeper What invariant must the store provide so the score keeper can ensure monotonically increasing scores? Write (“home”, 1); Write (“visitors”, 1); Write (“home”, 2); Write (“home”, 3); Write (“visitors”, 2); Write (“home”, 4); Write (“home”, 5); Reads must show effect of all prior writes (strong consistency) Visitors = 2 Home = 5

35. Example 1: score keeper Under strong consistency, what possible scores can the score keeper read after this write completes? Write (“home”, 1); Write (“visitors”, 1); Write (“home”, 2); Write (“home”, 3); Write (“visitors”, 2); Write (“home”, 4); Write (“home”, 5); 2-5 Visitors = 2 Home = 5

36. Example 1: score keeper Under read-my-writes, what possible scores can the score keeper read after this write completes? Write (“home”, 1); Write (“visitors”, 1); Write (“home”, 2); Write (“home”, 3); Write (“visitors”, 2); Write (“home”, 4); Write (“home”, 5); 2-5 Visitors = 2 Home = 5

37. Visitors’ score Home score S1 S2 S3 S4 S5 S6 V V V H H H W W W Writer (also reads) Writer (also reads) Reader

38. Visitors’ score Home score S1 S2 S3 S4 S5 S6 V V’ V’ H H’ H Under strong consistency, who must S3 have spoken to (directly or indirectly) to satisfy read request? R S2, S5 Writer (also reads) Writer (also reads) Reader

39. Visitors’ score Home score S1 S2 S3 S4 S5 S6 V V’ V’ H H’ H When does S3 have to talk to S2 and S5? Before writes return or before read returns? R Implementation can be flexible. Guarantee is that inform-flow occurs before read completes. Writer (also reads) Writer (also reads) Reader

40. Visitors’ score Home score S1 S2 S3 S4 S5 S6 V V’ V’ H H’ H Under read-my-writes, who must S3 have spoken to (directly or indirectly) to satisfy read request? R S5 Writer (also reads) Writer (also reads) Reader

41. Visitors’ score Home score S1 S2 S3 S4 S5 S6 V V’ V’ H H’ H For baseball, why is read-my-writes equivalent to strong consistency, even though it is “weaker”? R Application only has one writer. Not true in general. Reader Writer (also reads) Reader

42. Example 1: score keeper Write (“home”, 1); Write (“visitors”, 1); Write (“home”, 2); Write (“home”, 3); Write (“visitors”, 2); Write (“home”, 4); Write (“home”, 5); Common theme: Consider application invariants Reason about what store must ensure to support application invariants Visitors = 2 Home = 5

43. Example 2: umpire if first half of 9th inning complete then vScore = Read (“visitors”); hScore = Read (“home”); if vScore < hScore end game; Idea: home team doesn’t need another chance to bat if they are already ahead going into final half inning

44. Example 2: umpire if first half of 9th inning complete then vScore = Read (“visitors”); hScore = Read (“home”); if vScore < hScore end game; What invariant must the umpire uphold? Game should end if home team leads going into final half inning.

45. Example 2: umpire if first half of 9th inning complete then vScore = Read (“visitors”); hScore = Read (“home”); if vScore < hScore end game; What subset of writes must be visible to the umpire to ensure game ends appropriately? Reads must show effect of all prior writes (strong consistency)

46. Example 2: umpire if first half of 9th inning complete then vScore = Read (“visitors”); hScore = Read (“home”); if vScore < hScore end game; Would read-my-writes work as it did for the score keeper? No, since the umpire doesn’t issue any writes

47. Example 3: radio reporter do { vScore = Read (“visitors”); hScore = Read (“home”); report vScore, hScore; sleep (30 minutes); } Idea: periodically read score and broadcast it to listeners

48. Example 3: radio reporter do { vScore = Read (“visitors”); hScore = Read (“home”); report vScore, hScore; sleep (30 minutes); } What invariants must the radio reporter uphold? Should only report scores that actually occurred, and score should monotonically increase

49. Example 3: radio reporter do { vScore = Read (“visitors”); hScore = Read (“home”); report vScore, hScore; sleep (30 minutes); } Do we need strong consistency? No, since listeners can accept slightly old scores.

50. Example 3: radio reporter do { vScore = Read (“visitors”); hScore = Read (“home”); report vScore, hScore; sleep (30 minutes); } Can we get away with eventual consistency (some subset of writes is visible)? No, eventual consistency can return scores that never occurred.