Ordered Communication

1. Ordered Communication

2. Ordered Communication • Define guarantees about the order of deliveries inside group of processes • Type of ordering: • Deliveries respect the FIFO ordering of the corresponding sendings • Deliveries respect the Causal ordering of the corresponding sendings • Delivery respects a total ordering of deliveries (atomic communication)

3. Advantages of ordered communication • Orthogonality wrt reliable communication. Reliable broadcast does not have any property on ordering deliveries of messages • This can cause anomalies in many applicative contexts • “Reliable ordered communication” are obtained adding one or more ordering properties to reliable communication Example: flight booking system. Consider the message pattern depicted in the figure. The server cansel a reservation that has never been done! t “reserve” “cancel” client server “Prices 15% off”

4. FIFO Broadcast\ specification • Messages sent by the a process has to be delivered in the sending order. • FIFO Reliable broadcast specification is given by properties reliable (regular) broadcast plus an additional property of Safety that captures the notion of order (example): • FIFO Order: if a process sends a broadcast message m before m’, then no correct process delivers m’ if it has not already delivered m. • FIFO order can be uniform/non uniform FIFO Broadcast = Reliable Broadcast + FIFO Order

5. FIFO Broadcast\algorithm • Each process q holds: • S p a count of messages broadcast by p • Rp the sequence number of the latest message sent by p and delivered by q • For p to FO-multicast a message to g, it piggybacks S pon the message, rbBroadcasts it and increments S pby 1 • On receipt of a message from q sent by p with sequence number S, p checks whether S = Rp+ 1. If so, q FO-delivers it • if S > Rp+ 1 then q places message in hold-back queue until intervening messages have been delivered. (note that rbBroadcast does eventually deliver messages unless the sender crashes)

6. Advantages of Ordered Communication (2) • FIFO does not prevent all the anomalies due to order of deliveries. • Es: newsgroup. • Even though the computation satisfies FIFO order, student 2 can be disoriented by the order in which it receives messages m1 and m2. • m1 causally precede m2, then m2 can be delivered only if m1 is delivered. • Causal Broadcast m1: “Fri exam cancelled” Prof. m2: “let’s party on Thu night” Student 1 m3: “but we have an exam on Fri!” Student 2

7. Causal Broadcast\specification • Causal Reliable broadcast specification is given by properties reliable (regular) broadcast plus an additional property of Safety that captures the notion of order (example): • Causal Order: if the sending of a message m causally precedes the sending of a message m’, then every correct process has to deliver m before delivering m’. Causal Broadcast = Reliable Broadcast+Causal Order Causal Order FIFO Order, But FIFO OrderCausal Order thus, Causal Order = FIFO Order + ?

8. Causal Broadcast\specification • Causal Order = FIFO Order + Local Order. • Local Order: if a process delivers a message m before sending a msg m’, then no correct process deliverm’ if it has not already delivered m. • Example: t m p m’ q r The delivery of m’ is delaied till the arrival and the delivery of m

9. m1 m1,m2 ,m3 m2 Filter out Causal Broadcast\implementations • Two implementations • blocking algorthm using vector clocks (already discussed) • non-blocking algorithm using piggybacking of causal past COBcast(m1) COdelv(m3) COdelv(m1) COdelv(m2) p1 COBcast(m3) COdelv(m2) COdelv(m3) p2 COdelv(m1) p3 CObcast(m2) COdelv(m1) COdelv(m2) m2già COdelivered! COdelv(m3)

10. Advantages of Ordered Communication (3) • Causal Order is not enough strong to avoid anomalies • Es. banking. Bank account replicated on two sites Deposit £20 A:£100 A:£120 A:£132 R1 m1 A:£100 m2 R2 Add 10% interest A:£110 A:£130 Despite the fact that replicas initially share the same state, the state reaches a different value in the two sites at the end of the exexution as shown in Figure. Note that the computation is Causally Ordered To guarantee values of replicas be the same, one has to ensure that the order of delivery be the same at each process. In the above example R1 delivers m1 before m2 and R2 delivers m2 before m1. Note that ensuring the same delivery order at each replicas does not look at the sending order of messages

11. Atomic Broadcast\specification • Atomic (Total) Reliable broadcast specification is given by properties reliable (regular) broadcast plus an additional property of Safety that captures the notion of total order (example): • Total Order: if two correct processes p and q deliver m and m’, then p delivers m before m’ if, and only if, q delivers m before m’ • Total order is orthogonal with respect to FIFO and Causal Order. Total order would accept indeed a computation in which a process sends n messages to a group, and each of the processes of the group delivers such messages in the reverse order of their sendings. The computation is totally ordered but it is not FIFO.

12. Total Order Reliable broadcast Atomic broadcast FIFO Order FIFO Order Total Order FIFO broadcast FIFO Atomic broadcast Causal Order Causal Order Local Order Local Order Total Order Causal broadcast Hierarchy of Broadcast Specifications Causal Atomic broadcast

13. TOcast(m) TOdeliver(m) m p1 m p2 m pn crash System model • Static set of processes Π = {p1 … pn} • Message passing over perfect channels (message exchanging between correct processes is reliable) • Asynchronous • Crash fault model for processes • We characterize the system in terms of its possible runs R R r

14. RP RP’ RS RS’ R RPn RP1 RS A few notation • Property P: predicate on the system, identifying a set of runs RP R • P  P’ iff RP RP’ • Specification S(P1,…,Pm): logical and of m properties, identifying a set of runs RS=RP1∩ … ∩ RPm R • S → S’ iff RS RS’ R R

15. TO specifications Total order specifications are usually composed by four properties, namely Validity, Integrity,Agreement, and Order. • A Validity property guarantees that messages sent by correct processes will eventually be delivered at least by correct processes; • An Integrity property guarantees that no spurious or duplicate messages are delivered; • An Agreement property ensures that (at least correct) processes deliver the same set of messages; • An Order property constrains (at least correct) processes delivering the same messages to deliver them in the same order.

16. TO(A,O) TO specifications • Total Order Broadcast = S(V,I,A,O) • V = Validity • I = Integrity • A = Agreement • O = Order • Distinct specifications arise from distinct formulations of each property • uniform vs non-uniform • A uniform property imposes restrictions on the behavior of (at least) correct processes on the basis of events occurred in some process NUV UI

17. TO Specifications • Crash failure + Perfect channels  • NUV. if a correct process TOCAST a message m then some correct process will eventually deliver m • UI. For any message m, every process p delivers m at most once and only if m was previously tocast by some (correct or not) process.

18. The Agreement property • (Uniform Agreement, UA) If a process (correct or not) todelivers a message m, then all correct processes will eventually todeliver m; • (Non-uniform Agreement, NUA) If a correct process todelivers a message m, then all correct processes will eventually todeliver m

19. m3 m1 m3 m1 m4 m4 m2 m2 p1 p1 m3 m4 m3 m4 m1 m1 m2 m2 NUA UA p2 p2 m4 m3 m4 m3 m1 m1 m5 p3 p3 The Agreement property • Constrains the set of delivered messages • Correct processes always deliver the same set of messages M • Each faulty process p delivers a set Mp • UA: Mp M • NUA: Mpcan be s.t. Mp- M ≠  m2

20. m3 m6 m1 m2 m3 m6 m1 m2 p1 p1 m3 m7 m1 m2 m3 m7 m1 p2 p2 m2 m4 m2 m4 m5 m5 m1 m1 p3 p3 The Order property • Constrains the order of message deliveries and possibly the set of delivered messages • SUTO: if p delivers m<m’, q delivers m’ only after m • same order • same prefix of the set of delivered messages • after an omission, disjoint sets of delivered messages • WUTO: if p,q deliver m,m’, they get the same order • no restrictions on the set of delivered messages WUTO SUTO

21. The Order property (2) • SUTO and WUTO are uniform • They both have a non-uniform counterparts: SNUTO and WNUTO • (Strong Non-uniform Total Order, SNUTO). If some correct process todelivers some message m before message m', then a correct process todelivers m‘ only after it has todelivered m. • (Weak Non-uniform Total Order, WNUTO) If correct processes p and q both todeliver messages m and m', then p todelivers m before m' if and only if q todelivers m before m‘

22. m3 m6 m1 m2 p1 m2 m3 m7 m1 p2 m1 m4 m5 m2 p3 m3 m6 m1 m2 p1 m3 m7 m1 p2 m1 m4 m5 m2 p3 The Order property (2) • SUTO  WUTO • SNUTO  WNUTO SNUTO m2 WNUTO

23. m3 m6 m1 m2 TO(UA,SUTO) (Strongest total order) p1 m2 m3 m6 m1 m3 m6 m1 m2 p2 p1 m2 m4 m2 m3 m5 m6 m1 m1 p2 p3 m2 m1 TO(NUA,SUTO) p3 TO specifications • TO(UA,SUTO) • The strongest TO spec. • TO(NUA,SUTO)

24. TO(UA,SUTO) (Strongest total order) m3 m4 m1 m2 TO(NUA,SUTO) p1 TO(UA,WUTO) m3 m4 m1 m2 p2 m3 m4 m1 p3 TO(NUA,WUTO) m6 m3 m4 m1 m2 p1 m6 m3 m4 m1 m2 p2 m3 m4 m5 m6 m1 p3 TO specifications (2) • TO(UA,WUTO) m2 TO(NUA,WUTO) m2

25. TO(UA,SUTO) (Strongest total order) m3 m4 m1 m2 TO(NUA,SUTO) p1 TO(UA,WUTO) m3 m4 m1 m2 p2 m4 m3 m1 p3 TO(UA,WNUTO) TO(NUA,WUTO) m6 m3 m4 m1 m2 p1 TO(NUA,WNUTO) m6 m3 m4 m1 m2 p2 m4 m3 m5 m6 m1 p3 TO specifications (3) • TO(UA,WNUTO) m2 TO(NUA,WNUTO) m2