Providing High Availability Using Lazy Replication

1. Providing High Availability Using Lazy Replication Rivaka Ladin, Barbara Liskov, Liuba Shrira, Sanjay Ghemawat Presented by Huang-Ming Huang

2. Outline • Model • Algorithm • Performance Analysis • Discussion

3. Replication Model RM client FE RM Service Front ends RM client FE Replication Manager Excerpt from “Distributed Systems – Concept and Design” by Coulouris, Dollimore and Kindberg

4. System Guarantees • Each client obtains a consistent service over time • Relaxed consistency between replicas • Updates are applied with ordering guarantees that make the replicas sufficiently similar.

5. Operation Classification RM RM RM gossip Update, prev Update id Query, prev Val, new FE FE query val update Client Client Excerpt from “Distributed Systems – Concept and Design” by Coulouris, Dollimore and Kindberg

6. Update operation classification • Causal update • Forced update : performed in the same order (relative to one another) at all replicas. • Immediate update : performed at all replicas in the same order relative to all other operations.

7. Vector timestamp • Given two timestamps • T = (t1,t2,,tn) • S = (s1,s2,,sn) • T ≤ S ≡ti≤si for all i • merge(T,S)= (max(t1,s1),…,max(tn,sn)) • Each part of the vector timestamp corresponds to each replica manager in the system.

8. Update log RM components Other replicas Replica Timestamp Replica log Gossip Messages Timestamp table stable Value Timestamp Replica timestamp updates Value Executed operation table Updates Operation prev id FE FE Excerpt from “Distributed Systems – Concept and Design” by Coulouris, Dollimore and Kindberg

9. Query • The replica manager blocks the query q operation until the condition holds: • q.prev <= valueTS • The replica manger returns valueTS back to FE. • FE updates its own timestamp • frontEndTS := merge(frontEndTS, new)

10. Causal Update Replication Manageri ValueTS merge(ValueTS, r.ts) (r1,r2,…,ri+1,…,rn) (r1,r2,…,ri,…,rn) Value apply(value.r.u.op) Update log executed  r.u.id Executed operation table ts=(p1,p2,…,pi+1,…,pn) r.u.prev ≤ valueTS logRecord =(i, ts, u.op, u.prev, u.id) ts operation (p1,p2,…pn,) id FE

11. Gossip messages • Goal : bring the states of replication managers up to date. • Consists of : • Replication timestamp • Update log • Upon receiving gossip • Merge the arriving log with its own • Apply any unexecuted stable updates • Eliminate redundant log and executed operation table entries

12. Control the size of update log • Timestamp table • keeps recent timestamps • from messages sent by all other replicas. • A log record r can be removed from the log when • r.tsr.i < timestamp_table[j] r.i , for all j

13. Control the size of executed operation table • Each update carries an extra time field • FE returns an ACK • Contains FE’s clock time after receiving the response for an update from RM. • RM inserts the received ACK to the log.

14. Control the size of executed operation table (con’t) • A message m from FE is late if • m.time + δ< replica’s clock time • An update is discard if it is late • An ACK is kept at least until it is late • Remove an entry c in executed operation table when • an ACK for c’s update is received • all records for c’s update have been discarded.

15. Forced Update • Use the primary to assign a global unique identifier. • The primary carries out a two phase protocol for updates.

16. Two phase protocol • Upon receiving an update, the primary sends it to all other replicas. • Upon receiving responses from all most half of the backups, • the primary commit the update by insert the record to its log. • Backups know the commitment from gossip messages.

17. Fail Recovery • New coordinator informs participants about the failure. • Participants inform coordinator about most recent forced updates • Coordinator assign UID with the largest it knows after the sub-majority of replicas has responded.

18. Immediate Update • Primary use 3 phase protocol. • Pre-prepare • Prepare • Commit

19. Update log 3 phase protocol primary update FE Update id logRecord Give me your log and timestamp backup backup backup

20. Number of Messages for different operations • Query : 2 • Casual : 2 + (N-1)/K • Forced : 2N/2+ (N-1)/K • Immediate : 2N +2(N/2-1)+(N-1)K • N : the number of replicas • K : the number of update/ack pairs in a gossip.

21. Capacity of a 3-replica system Excerpt from “Providing high Availability Using Lazy Replication” by Ladin, Liskov, Shrira and Ghemawat

22. Capacity of the Unreplicated System Excerpt from “Providing high Availability Using Lazy Replication” by Ladin, Liskov, Shrira and Ghemawat

23. Discussion • No time guarantee for gossip messages • Not generally suitable for real-time application such as • realtime conference • updating shared document. • Scalability • Timestamp space grows as number of replicas grow. • can be increased by making most of the replicas read-only

24. Qustions?