Efficient Message Ordering in Dynamic Networks

1. Efficient Message Ordering in Dynamic Networks PODC 1996 talk slides Idit Keidar and Danny Dolev The Hebrew University Transis Project

2. The System Model • Asynchronous • Network partitions and site failures • Transport layer (Transis) provides: • Locally ordered multicast within connected components • Membership

3. An Algorithm for Totally Ordering Messages • A connected majority can always make progress • Previous protocols sometimes block • All processes may initiate messages, that may eventually be ordered • A majority orders messages within two rounds

4. COReL - Consistent Object Replication Layer Application totally ordered messages multicast messages COReL locally ordered messages send messages membership Transis physical network

5. Reliable Multicast • Logging all messages • ACKing logged messages • Retransmission upon recovery • Gossip-like dissemination

6. Message Ordering • Transis orders messages within each network component • This order is not atomic • COReL uses this order within a primary component: Members of a primary component P totally order a message once it was ACKed by all the members of P

7. The Colors Model • Red: • No knowledge of order • Messages from minority • Yellow: • Message from primary • Green: • Totally ordered • All members of primary marked as yellow (ACKed) Message Queue (MQ)

8. 6 5 4 4 3 3 2 2 1 1 Knowledge Levels in MQ Process s partitioned from p, q and r 5 4 3 x 2 2 1 1 s p q r

9. Invariants of MQ • FIFO and Causal Question is ordered before answer • No changes in green • Agreed green prefix • Yellow If p marked m as green in P, and q knows of P: • q has m as yellow or green • q and p have the same messages before m

10. Recovery:(upon membership change) • Retransmission of missing messages • Green and yellow messages from the last primary component are ordered before concurrent red messages • Agreeing upon MQ • Preserving the invariants • If majority: establish new primary

11. Recovery Majority? Attempt All Attempted? Commit All Committed? Establish Establishing a New Primary If a membership change occurs - restart the Recovery

12. 6 5 y 4 4 4 x 3 3 3 2 2 2 2 1 1 1 1 During Recovery - Need to Know the Last Primary q, r and s become connected BEFORE RECOVERY AFTER y x 4 3 2 1 Last Primary {p, q, r} {q, r, s} {s}

13. Numbering the Primary • How can we determine which primary is latest? • Use two variables: • LAST COMMITTED PRIMARY • Used to know which is later • Unique number • LAST ATTEMPTED PRIMARY • Guarantees uniqueness of LAST COMMITTED PRIMARY

14. A Majority Can Always Make Progress When a majority becomes connected: • All members recover state of last primary • The members establish a new primary • The members of an established primary order messages ACKed by all of them

15. Conclusions • COReL orders messages within connected primary components in two communication rounds • Extends local order determined by Transis within primary components • Messages from minorities diffuse through the system and eventually become ordered

16. Thanks to ... • Yair Amir, Dalia Malki, Catriel Beeri