1 / 30

Apache Zookeeper

Apache Zookeeper. Talentica DevOps Engineering Team. What is distributed system ?. A distributed systems provides single coherent view of collection of independent computing machines. Multiple software components on multiple computers, which run as a single system.

emerritt
Download Presentation

Apache Zookeeper

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Apache Zookeeper TalenticaDevOps Engineering Team

  2. What is distributed system ? • A distributed systems provides single coherent view of collection of independent computing machines. • Multiple software components on multiple computers, which run as a single system. • It can be locally or geographically connected.

  3. Characteristics of Distributed System • Concurrency • Fault Tolerance • Scalability • Transparency • Heterogeneity

  4. Delusion of Distributed System • Reliable Network • Zero Latency • Infinite Bandwidth • Secure Network • Topology Doesn’t change • Homogeneous Network • There is one administrator

  5. Introducing Zookeeper Zookeeper provides coordination between distributed process via shared hierarchical namespace.

  6. Why Zookeeper ? • Naming Service • Locking • Synchronization • Configuration Management • Leader Election

  7. Zookeeper Architecture

  8. Zookeeper Architecture • Zookeeper follows client server architecture • Collection of Zookeeper servers is called an ensemble • At given time, a Zookeeper client is connected to one Zookeeper server • Client read requests are catered by connected Zookeeper server • Client write requests are forwarded to Zookeeper leader. • Write is considered successful, if quorum is achieved.

  9. Zookeeper Architecture • Client updates are sequentially consistent • Updates are atomic • Updates are reliable • Timeliness

  10. ZAB Protocol • Zookeeper Atomic Broadcast • Propagates state changes from leader to followers • Guarantees the state change

  11. Znodes • Organized similar to standard Unix and Linux file system • Similar to files and directories • Both directory and leaf znode holds data • Stores data <= 1MB

  12. Znode - Hierarchical Namespace

  13. Time in Zookeeper Zxid : Every change to ZK state receives a stamp in form of zxid Version Number : Change to Znode will cause change in Version Ticks : Ticks define timing of events Real Time : To put timestamp of stat structure and znode modification

  14. Znode Stat Structure Czxid : Change that caused Znode to be created Mzxid : Change that last modified Znode Pzxid : Change that last modified children of the Znode Ctime : Millisecond from epoch, when this Znode was created Mtime : Millisecond from epoch, when this Znode was last modified Version : Number of changes to data of Znode Cversion : Number of changes to children of Znode Aversion : Number of changes to ACL of Znode EphemeralOwner : SessionId of owner of this Znode DataLength : Length of data field Numchildren : Number of children

  15. Zookeeper Operations

  16. Zookeeper Operations

  17. Types of Znodes • Ephemeral Znode : Deleted with termination of client session, also can’t have any child znode. • Persistent Znode : Can have child znode, It is explicitly required to delete it. • Sequential Znode : Append a monotonicly increasing counter to the end of path. It is unique to parent znode.

  18. Znode Watchers • One time triggers • Sent to client which sets the watch • Maintained locally at zookeeper • Client won’t receive watches after getting disconnected • Two lists of watches • Data watches : create(),setData(), delete() • Child watches : create(), delete()

  19. Zookeeper ACL • ACL Permission • Create • Read • Write • Delete • Admin

  20. Zookeeper ACL • ACL Scheme • World • Auth • Digest • Host • IP

  21. Zookeeper Recipes/Higher Order Functions/Apache Curator • Naming Service • Service Discovery • Configuration Management • Locks • 2-phase commit • Leader Election • Barriers • Queues

  22. Netflix/SoabaseExhibitor • Zookeeper server is fail fast • Zookeeper is self-healing, (i.e. crashed/failed server if brought online, will automatically join the cluster) • Instance Monitoring • Log Cleanup • Backup/Restore • Cluster-wide configuration • Rolling Ensemble Change • Rich REST API

  23. Backup/Exhibitor • Online Backup - Use Netflix Exhibitor • Offline Backup - Bring down the old ensemble cluster and copy the zookeeper data directory to the new ensemble cluster

  24. Zookeeper Monitoring • 4 Letter Words • JMX

  25. Zookeeper Monitoring - 4 Letter Words

  26. Zookeeper Monitoring - JMX • Mbeans

  27. Data Directory Cleanup • autopurge.snapRetainCount: Maintains n number of recent snapshots along with transaction log • autopurge.purgeInterval : TimeInterval at which purge event will trigger

  28. Zookeeper Future Releases - 3.5.X • Dynamic Reconfiguration w/o Rolling restarts • Separate Dynamic configuration file • Adding/Removing Servers • Modifying server parameters • Rebalance Client Connections • Backward Compatible • Modifying Server Roles[Obersver/Participants]

  29. Zookeeper in Real World • Apache Kafka • Apache Storm • Apache HBase • Apache Solr • Apache Mesos • Apache Hive

  30. Questions ? Thank You

More Related