1 / 23

Zookeeper

ZooKeeper is a service for coordinating distributed processes, providing high-performance server implementation for handling hundreds of thousands of transactions per second. It is a distributed system that enables efficient implementation of other distributed systems.

badamson
Download Presentation

Zookeeper

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Zookeeper Wait-Free Coordination for Internet-Scale Systems

  2. What is ZooKeeper • Service for coordinating distributed processes • Wait-free coordination • Enables high-performance server implementation • Can handle hundreds of thousands of transactions per second • Distributed system for implementing distributed systems!

  3. What distributed processes entail • Large number of processes • Heterogeneous hardware • Inter-Process Communication • Asynchronous systems • Network delays

  4. Some Examples • Search engines • Crawling • Indexing • Query Processing • Large-scale data processing • Map-reduce • Hadoop • Dryad

  5. Why is it necessary • Distributed systems need • Configuration Maintenance • Distributed Synchronization • Group Membership • Because • Race Conditions • Deadlocks • Bugs

  6. Introduction • ZooKeeper – Coordination service • Database of meta-data • Relieves distributed systems of its distributed responsibilities • How?

  7. Elements of ZooKeeper • Replicated in-memory database • Hierarchical DHT • Coarse-grained lock service • Event queue server • Hierarchical Pub/Sub server

  8. Guarantees of ZooKeeper • Serializability • Serializable Reads • All reads from a client are processed in order • Linearizability • Linearizable Writes • All writes from all clients are processed in order

  9. Data Model • File system supporting full reads and writes • Uses znodes • Data objects • Hierarchical ordering • Znodes are unlike files • Does support storing metadata

  10. Data Model

  11. The API • create(path, data, flags) • delete(path, version) • exists(path, watch) • getData(path, watch) • setData(path, data, version) • getChildren(path, watch) • sync(path)

  12. Why multiple functions for a function • Atomicity • Message passing • Three notifications • Exists -> znode insertion at a path • getData -> znode data updates • getChildren -> znode group broadcasts • Failure detection • Synchronization

  13. The many guarantees of ZooKeeper • Sequential consistency • Atomicity • Reliability • Group revision • Linearizable reads

  14. ZooKeeper Implementation

  15. ZooKeeper ImplementationRequest Processor • Provides high availability by replication • Use atomic broadcast for coordination in case of writes • If read request, simply generate response

  16. ZooKeeper ImplementationRequest Processor • Replicated database contains entire tree • Maintains logs for recoverability • Clients connect to one server to submit requests • Transactions are idempotent. • Writes forwarded to one server – leader

  17. ZooKeeper Implementation

  18. ZooKeeper Primitives • Configuration Management • Rendezvous • Group membership • Simple locks • Read / Write locks • Double barrier

  19. Evaluation of ZooKeeper • Variable number of servers, fixed number of clients. • 35 machines simulating 250 simultaneous clients, which all use the asynchronous API. • Read/write payloads all 1KB in size. • Benchmarking done on the client side.

  20. Evaluation of ZooKeeper

  21. Evaluation of ZooKeeper

  22. Evaluation of ZooKeeper 1. Failure and recovery of a follower; 2. Failure and recovery of a different follower; 3. Failure of the leader; 4. Failure of two followers (a, b) in the first two marks, and recovery at the third mark (c); 5. Failure of the leader.

  23. Conclusion • Wait-free approach towards coordinating processes • Used in several applications • Yahoo Message Broker (Pub/Sub) • Hadoop • Katta – Distributed Indexer

More Related