CS4231 Parallel and Distributed Algorithms AY 2006/2007 Semester 2

1. CS4231Parallel and Distributed AlgorithmsAY 2006/2007 Semester 2 Lecture 6 Instructor: Haifeng YU

2. Review of Last Lecture • Chapter 9 “Global Snapshot” • What is our goal? • Formalizing consistent global snapshot • Protocol for capturing a consistent global snapshot CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

3. Today’s Roadmap • Chapter 12 “Message Ordering” • FIFO ordering • Already discussed last lecture • Causal ordering and its application • Total ordering and its application CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

4. A Internet Chat Room Example Alice (process1) Hi Bob, I just introduced you to Cary Introducing Bob to Cary Bob (process2) Hi Cary, this is Bob Cary (process3) ??? Sorry, I don’t know any Bob. The notion of “causal order”. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

5. Formalizing Causal Order • If a “send” event s1 caused a “send” event s2 • Causal order requires the corresponding receive event r1 be before r2 (r1 and r2 on the same process) • But how do we know whether s1 caused s2? • Based on “Happened-Before” relation • This is the only ordering observable to external users • If a “send” event s1 happens before a “send” event s2 • Then s1 may have caused s2 • Let’s be pessimistic and be safe -- Assume s1 indeed caused s2 • Causal order: If s1 happens before s2, and r1 and r2 are on the same process, then r1 must be before r2 CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

6. How to Ensure Causal Ordering – Intuition Message1 A to B Message1 A to C PS: Message1 A to C Alice (process1) Bob (process2) Message1 B to C PS: Message1 A to C Message1 A to B Cary (process3) CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

7. How to Ensure Causal Ordering – Protocol • Each process maintains n*n matrix M • M[i,j] denotes that number of messages sent from i to j, as known by the local process • If process i send a message to process j • On process i: M[i,j]++ • Piggyback M on the message • Upon process j receiving a message from process i with matrix M’ piggybacked • If M[k,j]  M’[k,j] for all k except k = i, AND M[i,j] == M’[i,j] -1, then deliver message and M = pairwise-max(M,M’) • Else delay message CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

8. Example Run for the Protocol (0,0,1) (0,0,0) (0,0,0) (0,1,1) (0,0,0) (0,0,0) Alice (process1) (0,1,1) (0,0,1) (0,0,0) can deliver Bob (process2) Message1 B to C (0,0,0) (0,0,0) (0,0,0) (0,1,1) (0,0,0) (0,0,0)  can’t deliver Cary (process3) (0,0,0) (0,0,0) (0,0,0) (0,1,1) (0,0,1) (0,0,0) CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

9. Correctness Proof of the Protocol • (The textbook does not have an explicit correctness proof.) • s1, s2, r1, r2, where s1 happened before s2 • We want to prove that r1 is never delivered after r2 • Consider the case where s1 and s2 are on different processes • W.l.o.g., assume s1 on process 1, s2 on process 2, r1 and r2 on process 3 • We focus on the first 3 elements of the 3rd row of the matrices CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

10. Correctness Proof of the Protocol x1 y1 0 x2 y2 0 x2  x1 y2  y1 Alice (process1) x4  x3  x2 x3 y3 0 x4 y4 0 y4 > y3  y2 Bob (process2) x1 y1 0 x4 y4 0 Cary (process3) CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

11. Correctness Proof of the Protocol (continued) • Prove by contradiction: Assume Red delivered before Blue • After delivering Red: 3rd column of the matrix on process 3 will be • The values in the matrix never decreases: Blue can never be delivered  x4  y4 0 x4  x1 x1 y1 0 x4 y4 0 y4 > y1 Cary (process3) CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

12. Need More Correctness Proof • What we proved so far: • If s1 happened before s2, then r1 is never delivered after r2 is delivered • We don’t know if r1 and r2 will be delivered at all! • We will now prove that • At any given time, there must be one message that can be delivered to the process – induction will take care of the other messages CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

13. More Correctness Proof • Consider a process and its corresponding column in the matrix • If there is any message (from process i) undelivered to the process, then there must be messages whose matrix has the column x_1 x_2 … x_n ? ? x_i + 1 ? call such message as successor messages CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

14. More Correctness Proof • Consider the non-empty set of all successor messages and their corresponding “send” events • There must be a send event s that does not have any other send events that happened before s • Call the corresponding message first successor message • Claim: Any first successor message can be delivered • W.l.o.g, assume the message is from process 1 x_1 x_2 … x_n x_1 + 1 y_2 … y_n need to show x_i  y_i for i between 2 and n W.l.o.g, assume i = 2 The matrix column on the receiver The matrix column in the message CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

15. More Correctness Proof • Prove by contradiction: Suppose y_2 > x_2 process1 this is not a first successor message then! process2 ? x_2 … ? x_1 + 1 y_2 … y_n The last delivered message from process 2 must be another un-delivered message Receiver process CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

16. Summary of Correctness Proof • All messages will eventually be delivered • The delivery order satisfies causal order CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

17. Causal Ordering of Broadcast Messages • Broadcast: Every message is sent to all people (including the sender itself) • Modeled as n point-to-point messages • Application: Internet chat room • For same reason as before, we may need to ensure causal order among messages • Each process uses the previous protocol CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

18. A Simpler Protocol for Causal Ordering Broadcast • Each process maintains a message log for all messages delivered • When sending a message, append the message to log and send the whole log • Upon receiving a log, scan the log sequentially, for any message not seen before: Append the message to local log and deliver it • Can be used for point-to-point as well • But inefficient.. CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

19. Total Ordering of Broadcast Messages • All messages delivered to all processes in exactly the same order • Also called atomic broadcast • Total ordering does not imply causal ordering • Causal ordering does not imply total ordering • Thus…there is something called “causal total order” CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

20. Application: Internet Chat Room • We want to assign numbers to people have said • The number have to be consistent across all users [0] Alice: Welcome! [1] Bob: Hello, Alice [2] Alice: Let’s try to prove P=NP [3] Bob: OK, how do we start? …. …. …. [1712] Bob: I am confused, what were we trying to prove? [1713] Alice: Please refer to message [2] CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

21. Application: Replicated State Machine • State machine can be an abstract of many things • E.g., A database • Typically deterministic • Replication state machine • Multiple copies of the same state machine for performance / availability • Each “instruction” is broadcast to all state machines • The state machines must execution the “instruction” in exactly the same order! CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

22. Using a Coordinator for Total Order Broadcast • A special process is assigned as the coordinator • To broadcast a message • Send a message to the coordinator • Coordinator assigns a sequence number to the message • Coordinator forward the message to all processes with the sequence number • Messages delivered according to sequence number order • Problem: • Coordinator is performance bottleneck • What if coordinator fails? CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

23. Skeen’s Algorithm for Total Order Broadcast pick the max clock value received as message number process 1 • Each process maintains • Logical clock and a message buffer for undelivered messages • A message in the buffer is delivered / removed if • All messages in the buffer have been assigned numbers • This message has the smallest number broadcast message acknowledge process 2 notify message number process 3 reply with current logical clock value put msg in buffer CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

24. Correctness Proof for Skeen’s Algorithm pick the max clock value received as message number process 1 • Claim: All messages will be assigned message numbers • Claim: All messages will be delivered • Claim: If message A has a number smaller than B, then B is delivered after A – Prove by contradiction (continue on next slide) broadcast message acknowledge process 2 notify message number process 3 reply with current logical clock value put msg in buffer CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

25. Correctness Proof for Skeen’s Algorithm -- Continued • Suppose A is delivered on process 3 after B. • A must be placed in buffer after B is delivered • A must have a number larger than B – Contradiction. process 3 B placed in buffer B’s number received and B delivered A must be placed in buffer after B delivered reply for A key: Process 3’s logical clock now must be larger than B’s number CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

26. Summary • Chapter 12 “Message Ordering” • FIFO ordering for point-to-point messages • Already discussed last lecture • Causal ordering for point-to-point messages • Applications • Protocol to ensure causal ordering • Causal ordering for broadcast messages • Protocol • Total ordering for broadcast messages • Application • Skeen’s algorithm CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2

27. Homework Assignment • Page 206, Problem 12.3. For each pair (x, y) of the three pairs (i.e., C1 & C2, C1 & C3, C2 & C3) • Prove that x=y, OR • Prove x implies y, and a counter example that satisfies y but not x, OR • Prove y implies x, and a counter example that satisfies x but not y, OR • A counter example that satisfies y but not x, and a second counter example that satisfies x but not y • Think about why the protocol on slide 18 will deliver all messages in causal order • For the protocol on slide 22, think about whether the resulting total order satisfy causal order? Why? • Homework due a week from today • Read Chapter 13 CS4231 Parallel and Distributed Algorithms AY2006/2007 Semester 2