1 / 29

Amazon's Archeticture

Amazon’s Dynamo System The material is taken from “Dynamo: Amazon’s Highly Available Key-value Store,” by G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels. Amazon's Archeticture. What is Dynamo?.

dwillard
Download Presentation

Amazon's Archeticture

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Amazon’s Dynamo SystemThe material is taken from “Dynamo: Amazon’s Highly Available Key-value Store,”by G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S.Sivasubramanian, P. Vosshall, and W. Vogels.

  2. Amazon's Archeticture

  3. What is Dynamo? • The platform for Amazon's e-commerce services: shopping chart, best seller list, produce catalog, promotional items etc. • A highly avaliable key-value storage system: put() & get() interfaces • Aims to provide "always on" guarantee (at the cost of losing some consistency)

  4. Requirement • Reliability: Customers should be able to edit their shopping cart even when there are: • disk failures • network failures • tornados! • Efficiency: • latency measurement is done at 99.9th percentile of the distribution.

  5. Requirement • Query Model: only simple read/write to small data (less than 1MB ), uniquely identified by a key • ACID Properties: • atomicity: important! • consistency: weak is sufficient • isolation: not at all • durability: important!

  6. Requirement Service Level Agreement(SLA): • client and server agree on several characteristics related to the system • e.g 300 ms response time for peak load of 500 requests/second • putting together a single webpage can require responses from 150 services for typical request

  7. Techniques Used by Dynamo

  8. Consistent Hashing

  9. What if... • Unbalanced load distribution • Oblivious to the heterogeneity of nodes ?

  10. Virtual Node

  11. Virtual Node

  12. Node Joining/Leaving

  13. What if...

  14. Replication N = 3

  15. Putting Together coordinator N = 3

  16. Techniques Used by Dynamo

  17. Quorums • put(): • coordinator writes new data locally • send to next n-1 nodes • when w-1 respond, considered successful • get(): • coordinator requests next n-1 nodes • when r-1 respond, considered successful • r + w > n

  18. Quorums • A put() call may return successfully to caller before the update has been applied to all the replications • That being said...... • A get() call may return many versions of the same object.

  19. Delayed Reconciliation • The goal: • "add to cart" operation should never be forgotten or rejected. • When a customer wants to add an item to (or remove from) a shopping cart and the latest version is not available, the item is added to (or removed from) the older version • and the divergent versions are reconciled later.

  20. Vector Clock • The coordinator assigns a logical timestamp(an integer) to each version of key/value pair, update on each operation • Reconciliation • easy reconciliation if values are causally ordered • otherwise, applications handle them

  21. Vector Clock • size of vector clock can get arbitrarily long • bounded by N in pratice • drop oldest one when beyond certain threshold

  22. Reconciliation • One sentence summary: • An "add to cart" operation will never be lost, • but removed items may reappear...

  23. Reconciliation • In one 24 hour period, 99.94% of requests saw exactly one version • Divergent version usually not caused by failure but concurrent writers....(rarely human beings, usually automated client programs)

  24. Techniques Used by Dynamo

  25. Sloppy Quorum • "'add to cart' operation should never be forgotten or rejected." • put() request requires at least w-1 responses • What if the number of availiables nodes in the next N-1 node is less than w-1?

  26. Sloppy Quorum • Use next N healthy nodes for read/write • Data tagged with the node it should go to • Transfer the data to the node when it becomes avaliable • Hinted Handoff

  27. Techniques Used by Dynamo

  28. Handling Permanent Failures • Replica Synchronization: • Node synchronizes with another node • A Merkle Tree is used to detect inconsistensy and minimize the data that needs to be transfered • leaves are hash of objects • parents are hash of children

  29. Membership and Failure Detection • Gossip-based protocol propagates membership change and maintains an eventually consistent view • Seeds are used to prevent parititioned ring • Use timeout to for failure discovery

More Related