Amazon’s Dynamo

1. Amazon’s Dynamo Simple Cloud Storage

2. Foundations • 1970 – E.F. Codd“A Relational Model of Data for Large Shared Data Banks” • Idea of tabular data • SQL Foundations • Codd’s 12 rules • How database structured and what is available to user • Application not dependent on physical or logical levels of database • Insert, Update, Delete operators

3. Foundations, continued • First Relational database management systems (RDBMS) • Oracle in 1979, first SQL based system • Microsoft SQL server, etc • Open source software would follow later (mySQL) • Follows Codd’s ideas • Complexity on the server side, let the query do all the work • Very stringent requirements

4. Drawbacks • Licensing fees on a per processor rate • High end Oracle system is mind-numbingly expensive • Load Distribution requires specific nodes to handle • Some servers have specific roles, failure point in network • Complexity on servers creates difficultly with maintenance, upgrades • Upgrades often all at once as result, not incremental

5. A New Direction • Simplify services that the database provides • Easier scaling and error handling • Take a more pragmatic approach • Tailor system to sacrifice some aspects of the traditional RDBMS to gain performance in others • Systems less general, specific end requirements in mind when creating

6. Examples • Amazon Dynamo • Simple primary key • Highly available, end user based model • Low cost virtualized nodes • Facebook Cassandra • Similar goals to Amazon’s Dynamo • Highly avaiable, incremental scalablilty • Google File System • Master node • Data distributed across low cost nodes

7. Dynamo Goals • Scale – adding systems to network causes minimal impact • Symmetry – No special roles, all features in all nodes • Decentralization – No Master node(s) • Highly Available – Focus on end user experience • SPEED – A system can only be as fast as the lowest level • Service Level Agreements – System can be adapted to an application’s specific needs, allows flexibility

8. Dynamo Assumptions • Query Model – Simple interface exposed to application level • Get(), Put() • No Delete() • Atomicity, Consistency, Isolation, Durability • Operations either succeed or fail, no middle ground • System will be eventually consistent, no sacrifice of availability to assure consistency • Conflicts can occur while updates propagate through system • System can still function while entire sections of network are down • Efficiency – Measure system by the 99.9th percentile • Important with millions of users, 0.1% can be in the 10,000s • Non Hostile Environment - No need to authenticate query, speed boost

9. Wanted Results • Deliver requests in a bounded time • Always writable • Highly available to users • No dedicated roles • Work split between nodes fairly

10. Techniques

11. Partitioning • Consistent Hashing • Changing the number of slots in hash table results in only a small number of keys to remap • More info • A ring of virtual nodes • Node responsible for region between it and its predecessor

12. Virtual Node • Physical Machine has # of virtual nodes based on performance • Can adapt load more easily if a machine goes down • Likewise, assign nodes to a new machine in network

13. Replication • Application provided parameter N • Replication on different physical nodes • Data still available if nodes go down • Makes part of preference list for query

14. Versioning and Vector Clocks • Updates propagate asynchronously, need a way of distinguishing conflicts • Possible reason for absence of Delete() • Vector Clock • List of (node, counter) • Limited size, limit overhead for data • If all fields are less than or equal, first can be updated by second

15. Sloppy Quorum and Hinted Handoff • W and R parameter set min # of nodes in a read or write • Read and write on the first N healthy nodes, no strict membership, can vary over time • Hint in metadata for intended node, will update once that node is again available • Allows for temporary failure in nodes or entire networks

16. Synchronization and Gossip • Merkle Trees - Info • Use common key values between two nodes • Traverse tree and check vector clocks to see if updates are needed • Exchange information on most current version of the data if inconsistencies are found • Gossip • Nodes select neighbors at random and reconcile membership change histories • Use seed nodes to initialize • Detect failures

17. Routing get() and put() • Two Techniques • Route request through a load balancer • Slower • Simpler application level code • Partition aware client, route directly to appropriate nodes • Faster • More complicated application level • First node routed to is “coordinator” node • Generates vector clock for put and gives data to N highest healthy nodes • Queries N highest nodes for all versions, returned all versions found

18. Implementation • Java based • Hardware independent, JVM • Allows different back-end systems to be used, based on size of data needed to be stored • Berkeley Database Transactional Data Store • BDB Java Edition • MySQL, can handle large objects • Coordinator node is a state machine for read/writes for client • Coordinator for a write determined by fastest read

19. Flexibility • Changing W, R, N • Business logic specific reconciliation • Data replicated over nodes • Application level reconciliation fro conflicting objects • Timestamp Reconciliation • Similar to above, last write wins • High performance read engine • By setting R = 1, W = N • Reads fast and numerous, few updates

20. Observed Results - Speed

21. Observed Results – Load Balancing • Higher traffic causes load to be balanced more evenly • Requests of popular keys let system to balance more easily • In lower traffic, less important to balance load

22. Observed Results - Coordination • Client coordination can provide a speed boost • Read and write latency nearly identical • Results as expected

23. Observed Results - Versions • Measured over 24 hour period for shopping cart • 99.94% of users saw 1 version • 0.00057% saw 2 versions • 0.00047% saw 3 versions • 0.00009% saw 4 versions • Increase caused by increase in number of concurrent writers, most likely

24. Conclusions • Dynamo allows Amazon’s customers to have a consistent experience even in face of server and network errors • Gives a scalable solution with millions of data points to be queried quickly and efficiently • Offloads complexity to the application to provide a simple, flexible, and fast server-side implementation

25. Thanks for listening!