UBI529 Distributed Algorithms

1. UBI529 Distributed Algorithms Global State of Distributed Systems

2. Motivation • Goal: Take a snapshot of the global computation • A snapshot of local states on n processes taken at exactly the same time • Two terms “global state” and “global snapshot” • Useful for debugging • Useful for backup/check-pointing • Useful for calculating global predicate • E.g., Exactly how much currency do we have in the country (notice that money flows among people constantly)? Deadlock Detection Rollback Recovery Termination Detection

3. The mystery of the missing dollars Picture taken at A - $400 A sends $100 to B Picture taken at B - $400 Total is $800 Send $100 B A $300 $400

4. Global Snapshot Problem • Determine the global system state (e.g. the total money ) • Each process records its own state • No shared clock/memory • Group of photographers taking snaps of different portions and trying to combine to get the overall picture.

5. Consistent cut • Given computation (E,!) and F µ E is a cut iff • F is a consistent cut (global snapshot) iff

6. Consistent and inconsistent cuts

7. (aconsistent cutC)  (b happened before a) b  C Consistent cut A cut is a set of events. b g c a d P1 e m f P2 P3 k h i j Cut 1 Cut 2 (Not consistent) (Consistent)

8. Consistent snapshot • The set of states immediately following a consistent cut forms a consistent snapshot of a distributed system. • A snapshot that is of practical interest is the most recent one. Let C1 and C2 be two consistent cuts and C1  C2. Then C2 is more recent than C1. • Analyze why certain cuts in the one-dollar bank are inconsistent.

9. Consistent snapshot • How to record a consistent snapshot? Note that • 1. The recording must be non-invasive • 2. Recording must be done on-the-fly. • You cannot stop the system.

10. Chandy Lamport Algorithm • Assumes • FIFO and • Unidirectional channels • A bidirectional channel is modelled as two unidirectional channels • Each process has an associated color. All processes are initially white. • A process records it local state just before turning red • On turning red the process sends out a marker on all outgoing channels • On receiving a marker a white process turns red

11. Works on a (1) strongly connected graph (2) each channel is FIFO. An initiator initiates the algorithm by sending out a marker ( ) Chandy-Lamport Algorithm

12. Initially every process is white. When a process receives a marker, it turns red if it has not already done so. Every action by a process, and every message sent by a process gets the color of that process. White and red processes

13. Step 1. In one atomic action, the initiator (a) Turns red (b) Records its own state (c) sends a marker along all outgoing channels Step 2. Every other process, upon receiving a marker for the first time (and before doing anything else) (a) Turns red (b) Records its own state (c) sends markers along all outgoing channels The algorithm terminates when (1) every process turns red, and (2) Every process has received a marker through each incoming channel. Two steps

14. Lemma 1. No red message is received in a white action. Why does it work?

15. Theorem. The global state recorded by Chandy-Lamport algorithm is equivalent to the ideal snapshot state SSS. Hint. A pair of actions (a, b) can be scheduled in any order, if there is no causal order between them, so (a; b) is equivalent to (b; a) Why does it work? All white All red SSS Easy conceptualization of the snapshot state

16. Why does it work? Let an observer observe the following actions: w[i] w[k] r[k] w[j] r[i] w[l] r[j] r[l] …  w[i] w[k] w[j] r[k] r[i] w[l] r[j] r[l] … [Lemma 1]  w[i] w[k] w[j] r[k] w[l] r[i] r[j] r[l] … [Lemma 1]  w[i] w[k] w[j] w[l] r[k] r[i] r[j] r[l] … [done!] Recorded state

17. Let us verify that Chandy-Lamport snapshot algorithm correctly counts the tokens circulating in the system Example 1. Count the tokens D C A B How to account for the channel states? Use sent and received variables for each process.

18. Chandy Lamport Algorithm

19. publicclass RecvCamera extends Process implements Camera { . . . public RecvCamera(Linker initComm, CamUser app) { . . . for (int i = 0; i < N; i++) if (isNeighbor(i)) { closed[i] = false; chan[i] = new LinkedList(); } else closed[i] = true; } publicsynchronizedvoid globalState() { myColor = red; app.localState(); // record local State; sendToNeighbors("marker", myId); // send Markers } publicsynchronizedvoid handleMsg(Msg m, int src, String tag) { if (tag.equals("marker")) { if (myColor == white) globalState(); closed[src] = true; if (isDone()){ ----- Display channel state (transit messages) chan[] ---- } } else { // application message if ((myColor == red) && (!closed[src])) chan[src].add(m); app.handleMsg(m, src, tag); // give it to app } } boolean isDone() { if (myColor == white) returnfalse; for (int i = 0; i < N; i++) if (!closed[i]) returnfalse; returntrue; } } Algorithm

20. Lai Yang Algorithm • LY1. The initiator records its own state. When it needs to send a message m to another process, it sends a message (m, red). • LY2. When a process receives a message (m, red), it records its state if it has not already done so, and then accepts the message m.

21. Another example of distributed snapshot: Communicating State Machines

22. Something unusual • Let machine i start Chandy-lamport snapshot before it has sent M along ch1. Also, let machine j receive the marker after it sends out M’ along ch2. Observe that the snapshot state is • down  up M’ • Doesn’t this appear strange? This state was never reached during the computation!

23. Understanding snapshot

24. Understanding snapshot The observed state is a feasible state that is reachable from the initial configuration. It may not actually be visited during a specific execution. The final state of the original computation is always reachable from the observed state.

25. Discussions • What good is a snapshot if that state has never been visited by the system? • - It is relevant for the detection of stable predicates. • - Useful for checkpointing.

26. Discussions • What if the channels are not FIFO? • Study how Lai-Yang algorithm works. It does not use any marker • LY1. The initiator records its own state. When it needs to send a message m to another process, it sends a message (m, red). • LY2. When a process receives a message (m, red), it records its state if it has not already done so, and then accepts the message m. • Question 1. Why will it work? • Question 1 Are there any limitations of this approach?

27. Global state collection • Some applications • - computing network topology • - termination detection • - deadlock detection • Chandy Lamport algorithm does a partial job. Each process collects a fragment of the global state, but these pieces have to be stitched together to form a global state.

28. Once the pieces of a consistent global state become available, consider collecting the global state via all-to-all broadcast At the end, each process will compute a set V, where V= {s(i): 0 ≤ i ≤ N-1 } A simple exercise s(i) s(j) i j s(k) s(l) k l

29. Program broadcast (for process i} define V.i, W.i : set of values; initially V.i={s(i)}, W.i =  andevery channel is empty do V.i ≠ W.i send (V.i \ W.i) to every outgoing channel; W.i := V.i  ¬ empty (k, i) receive X from channel(k, i); V.i := V.i  X od All-to-all broadcast Assume that the topology is strongly connected graph V.i W.i V.k W.k (i,k) Acts like a “pump”

30. Lemma. empty (i. k) W.i V.k. (Upon termination) i: V.i = W.i, and all channels are empty. So, V.i  V.k. On a cyclic path, V.i = V.k must be true. Since s(i)V.i, s(i)V.k Proof V.i W.i V.k W.k (i,k)

31. Acknowledgements • This part is heavily dependent on Dr. Sukumar Ghosh Iowa University Distributed Systems course 22C:166

33. Termination Detection and Deadlocks

34. Termination detection • During the progress of a distributed computation, • processes may periodically turn active or passive. • A distributed computation termination when: • (a) every process is passive, • (b) all channels are empty, and • (c) the global state satisfies the desired postcondition

35. Visualizing diffusing computation initiator active passive Notice how one process engages another process. Eventually all processes turn white, and no message is in transit -this signals termination. How to develop a signaling mechanism to detect termination?

36. An initiator initiates termination detection by sending signals (messages) down the edges via which it engages other nodes. At a “suitable time,” the recipient sends an ack back. When the initiator receives ack from every node that it engaged, it detects termination. Node j engages node k. Dijkstra-Scholten algorithm The basic scheme j k signal j k j k ack

37. Deficit (e) = # of signals on edge e - # of ack on edge e For any node, C = total deficit along incoming edges and D = total deficit along outgoing edges For the initiator, by definition, C = 0 Dijkstra-Scholten algorithm used the following two Invariants to develop their algorithm: Invariant 1. (C ≥ 0)  (D ≥ 0) Invariant 2. (C > 0)  (D = 0) Dijkstra-Scholten algorithm 0 1 2 3 4 5

38. The invariants must hold whenan interim node sends an ack. So, acks will be sent when (C-1 ≥ 0)  (C-1 > 0 D=0) {follows from INV1 and INV2} = (C > 1)  (C ≥1  D=0) = (C > 1) (C =1  D=0) Dijkstra-Scholten algorithm 0 1 2 3 4 5

39. program detect {for an internal node i} initially C=0, D=0, parent = i do - m = signal  (C=0)  C:=1; state:= active; parent := sender {this node can send out messages to engage other nodes, or turn passive} - m = ack  D:= D-1 - (C=1 D=0)  state = passive send ack to parent; C:= 0; parent := i - m = signal  (C=1) send ack to the sender; od Dijkstra-Scholten algorithm 0 1 2 3 4 5 Note that the engaged nodes induce a spanning tree

40. Distributed deadlock • Assume each process owns a few resources, and review how resources are allocated. • Why deadlocks occur? • - Exclusive (i.e not shared) resources • - Non-preemptive scheduling • - Circular waiting by all or a subset of processes

41. Distributed deadlock • Three aspects of deadlock • deadlock detection • deadlock prevention • deadlock recovery

42. Distributed deadlock • May occur due to bad designs/bad strategy • [Sometimes prevention is more expensive than detection and recovery. So designs may not care about deadlocks, particularly if it is rare.] • Caused by failures or perturbations in the system

43. Represents who waits for whom. No single process can see the WFG. Review how the WFG is formed. Wait-for Graph (WFG)

44. Resource deadlock [R1 AND R2 AND R3 …] also known as AND deadlock Communication deadlock [R1 OR R2 OR R3 …] also known as OR deadlock Another classification

45. Notations w(j) = true  (j is waiting) depend [j,i] = true j  succn(i) (n>0) P(i,s,k) is a probe (i=initiator, s= sender, r=receiver) Detection of resource deadlock 2 1 3 4 P(4,4,3) initiator

46. {Program for process k} do P(i,s,k) received  w[k]  (k ≠ i) ¬ depend[k, i]  send P(i,k,j) to each successor j; depend[k, i]:= true P(i,s, k) received w[k]  (k = i) process k is deadlocked od Detection of resource deadlock

47. To detect deadlock, the initiator must be in a cycle Message complexity = O(|E|) (edge-chasing algorithm) Observations E=set of edges Should the links be FIFO?

48. Communication deadlock This has a resource deadlock but no communication deadlock

49. A process ignores a probe, if it is not waiting for any process. Otherwise, first probe mark the sender as parent; forwards the probe to successors Not the first probe Send ack to that sender ack received from every successor send ack to the parent Communication deadlock is detected if the initiator receives ack. Detection of communication deadlock Has many similarities with Dijkstra-Scholten’s termination detection algorithm

50. Distributed deadlock • May occur due to faulty design or resource sharing problems • [Sometimes prevention is more expensive than detection and recovery. So certain designs deliberately do not care about deadlocks, particularly if it is rare.] • Sometimes failures failures or perturbations can modigy the system state and cause deadlock. Major issues detection prevention recovery