Distributed Systems: Atomicity, decision making, snapshots

1. Distributed Systems: Atomicity, decision making, snapshots Slides adapted from Ken's CS514 lectures

2. Announcements • Please complete course evaluations • http://www.engineering.cornell.edu/CourseEval/ • Prelim II coming up this week: • Thursday, April 26th, 7:30—9:00pm, 1½ hour exam • 101 Phillips • Closed book, no calculators/PDAs/… • Bring ID • Topics: • Since last Prelim, up to (and including) Monday, April 23rd • Lectures 19-34, chapters 10-18 (7th ed) • Review Session Tuesday, April 24th • during second half of 415 Section • Homework 6 (and solutions) available via CMS • Do it without looking at solutions. However, it will not be graded • Project 5 due after Prelim II, Monday, April 30th • Make sure to look at the lecture schedule to keep up with due dates

3. Review: What time is it? • In distributed system we need practical ways to deal with time • E.g. we may need to agree that update A occurred before update B • Or offer a “lease” on a resource that expires at time 10:10.0150 • Or guarantee that a time critical event will reach all interested parties within 100ms

4. Review: Event Ordering • Problem: distributed systems do not share a clock • Many coordination problems would be simplified if they did (“first one wins”) • Distributed systems do have some sense of time • Events in a single process happen in order • Messages between processes must be sent before they can be received • How helpful is this?

5. Review: Happens-before • Define a Happens-before relation (denoted by ). • 1) If A and B are events in the same process, and A was executed before B, then A B. • 2) If A is the event of sending a message by one process and B is the event of receiving that message by another process, then A  B. • 3) If A  B and B  C then A  C.

6. Review: Total ordering? • Happens-before gives a partial ordering of events • We still do not have a total ordering of events • We are not able to order events that happen concurrently • Concurrent if (not AB) and (notBA)

7. Review: Partial Ordering Pi ->Pi+1; Qi -> Qi+1; Ri -> Ri+1 R0->Q4; Q3->R4; Q1->P4; P1->Q2

8. Review: Total Ordering? P0, P1, Q0, Q1, Q2, P2, P3, P4, Q3, R0, Q4, R1, R2, R3, R4 P0, Q0, Q1, P1, Q2, P2, P3, P4, Q3, R0, Q4, R1, R2, R3, R4 P0, Q0, P1, Q1, Q2, P2, P3, P4, Q3, R0, Q4, R1, R2, R3, R4

9. Review: Timestamps • Assume each process has a local logical clock that ticks once per event and that the processes are numbered • Clocks tick once per event (including message send) • When send a message, send your clock value • When receive a message, set your clock to MAX( your clock, timestamp of message + 1) • Thus sending comes before receiving • Only visibility into actions at other nodes happens during communication, communicate synchronizes the clocks • If the timestamps of two events A and B are the same, then use the process identity numbers to break ties. • This gives a total ordering!

10. Review: Distributed Mutual Exclusion • Want mutual exclusion in distributed setting • The system consists of n processes; each process Piresides at a different processor • Each process has a critical section that requires mutual exclusion • Problem: Cannot use atomic testAndSet primitive since memory not shared and processes may be on physically separated nodes • Requirement • If Pi is executing in its critical section, then no other process Pj is executing in its critical section • Compare three solutions • Centralized Distributed Mutual Exclusion (CDME) • Fully Distributed Mutual Exclusion (DDME) • Token passing

11. Today • Atomicity and Distributed Decision Making • What time is it now? • Synchronized clocks • What does the entire system look like at this moment?

12. Atomicity • Recall: • Atomicity = either all the operations associated with a program unit are executed to completion, or none are performed. • In a distributed system may have multiple copies of the data • (e.g. replicas are good for reliability/availability) • PROBLEM: How do we atomically update all of the copies? • That is, either all replicas reflect a change or none

13. 11 am ok? Yes, 11 works So, 11 it is? Yeah, but what if you Don’t get this ack? General’s Paradox • General’s paradox: • Constraints of problem: • Two generals, on separate mountains • Can only communicate via messengers • Messengers can be captured • Problem: need to coordinate attack • If they attack at different times, they all die • If they attack at same time, they win • Named after Custer, who died at Little Bighorn because he arrived a couple of days too early! • Can messages over an unreliable network be used to guarantee two entities do something simultaneously? • Remarkably, “no”, even if all messages get through • No way to be sure last message gets through!

14. Replica Consistency Problem -Concurrent and conflicting updates • Imagine we have multiple bank servers and a client desiring to update their back account • How can we do this? • Allow a client to update any server then have server propagate update to other servers? • Simple and wrong! • Simultaneous and conflicting updates can occur at different servers? • Have client send update to all servers? • Same problem - race condition – which of the conflicting update will reach each server first

15. Two-phase commit • Since we can’t solve the General’s Paradox (i.e. simultaneous action), concurrent and conflicting updates may be sent by clients, let’s solve a related problem • Distributed transaction: Two machines agree to do something, or not do it, atomically • Algorithm for providing atomic updates in a distributed system • Give the servers (or replicas) a chance to say no and if any server says no, client aborts the operation

16. Framework • Goal: Update all replicas atomically • Either everyone commits or everyone aborts • No inconsistencies even in face of failures • Caveat: Assume only crash or fail-stop failures • Crash: servers stop when they fail – do not continue and generate bad data • Fail-stop: in addition to crash, fail-stop failure is detectable. • Definitions • Coordinator: Software entity that shepherds the process (in our example could be one of the servers) • Ready to commit: side effects of update safely stored on non-volatile storage • Even if crash, once I say I am ready to commit then a recover procedure will find evidence and continue with commit protocol

17. Two Phase Commit: Phase 1 • Coordinator send a PREPARE message to each replica • Coordinator waits for all replicas to reply with a vote • Each participant replies with a vote • Votes PREPARED if ready to commit and locks data items being updated • Votes NO if unable to get a lock or unable to ensure ready to commit

18. Two Phase Commit: Phase 2 • If coordinator receives PREPARED vote from all replicas then it may decide to commit or abort • Coordinator send its decision to all participants • If participant receives COMMIT decision then commit changes resulting from update • If participant received ABORT decision then discard changes resulting from update • Participant replies DONE • When Coordinator received DONE from all participants then can delete record of outcome

19. Performance • In absence of failure, 2PC (two-phase-commit) makes a total of 2 (1.5?) round trips of messages before decision is made • Prepare • Vote NO or PREPARE • Commit/abort • Done (but done just for bookkeeping, does not affect response time)

20. Failure Handling in 2PC – Replica Failure • The log contains a <commit T> record. • In this case, the site executes redo(T). • The log contains an <abort T> record. • In this case, the site executes undo(T). • The log contains a <ready T> record • In this case consult coordinator Ci. • If Ci is down, site sends query-statusT message to the other sites. • The log contains no control records concerning T. • In this case, the site executes undo(T).

21. Failure Handling in 2PC – Coordinator CiFailure • If an active site contains a <commit T> record in its log, then T must be committed. • If an active site contains an <abort T> record in its log, then T must be aborted. • If some active site does not contain the record <ready T> in its log then the failed coordinator Cicannot have decided to commit T. Rather than wait for Cito recover, it is preferable to abort T. • All active sites have a <ready T> record in their logs, but no additional control records. In this case we must wait for the coordinator to recover. • Blocking problem – T is blocked pending the recovery of site Si.

22. Failure Handling • Failure detected with timeouts • If participant times out before getting a PREPARE can abort • If coordinator times out waiting for a vote can abort • If a participant times out waiting for a decision it is blocked! • Wait for Coordinator to recover? • Punt to some other resolution protocol • If a coordinator times out waiting for done, keep record of outcome • other sites may have a replica.

23. Failures in distributed systems • We may want to avoid relying on a single server/coordinator/boss to make progress • Thus want the decision making to be distributed among the participants (“all nodes created equal”) => the “consensus problem” in distributed systems. • However depending on what we can assume about the network, it may be impossible to reach a decision in some cases!

24. Impossibility of Consensus • Network characteristics: • Synchronous - some upper bound on network/processing delay. • Asynchronous - no upper bound on network/processing delay. • Fischer Lynch and Paterson showed: • With even just one failure possible, you cannot guarantee consensus. • Cannot guarantee consensus process will terminate • Assumes asynchronous network • Essence of proof: Just before a decision is reached, we can delay a node slightly too long to reach a decision. • But we still want to do it.. Right?

25. Distributed Decision Making Discussion • Why is distributed decision making desirable? • Fault Tolerance! • A group of machines can come to a decision even if one or more of them fail during the process • Simple failure mode called “failstop” (different modes later) • After decision made, result recorded in multiple places • Undesirable feature of Two-Phase Commit: Blocking • One machine can be stalled until another site recovers: • Site B writes “prepared to commit” record to its log, sends a “yes” vote to the coordinator (site A) and crashes • Site A crashes • Site B wakes up, check its log, and realizes that it has voted “yes” on the update. It sends a message to site A asking what happened. At this point, B cannot decide to abort, because update may have committed • B is blocked until A comes back • A blocked site holds resources (locks on updated items, pages pinned in memory, etc) until learns fate of update • Alternative: There are alternatives such as “Three Phase Commit” which don’t have this blocking problem • What happens if one or more of the nodes is malicious? • Malicious: attempting to compromise the decision making • Known as Byzantine fault tolerance. More on this next time

26. Introducing “wall clock time” • Back to the notion of time… • Distributed systems sometimes needs more precise notion of time other than happens-before • There are several options • Instead of network/process identitity to break ties… • “Extend” a logical clock with the clock time and use it to break ties • Makes meaningful statements like “B and D were concurrent, although B occurred first” • But unless clocks are closely synchronized such statements could be erroneous! • We use a clock synchronization algorithm to reconcile differences between clocks on various computers in the network

27. Synchronizing clocks • Without help, clocks will often differ by many milliseconds • Problem is that when a machine downloads time from a network clock it can’t be sure what the delay was • This is because the “uplink” and “downlink” delays are often very different in a network • Outright failures of clocks are rare…

28. Synchronizing clocks • Suppose p synchronizes with time.windows.com and notes that 123 ms elapsed while the protocol was running… what time is it now? Delay: 123ms p What time is it? 09:23.02921 time.windows.com

29. Synchronizing clocks • Options? • p could guess that the delay was evenly split, but this is rarely the case in WAN settings (downlink speeds are higher) • p could ignore the delay • p could factor in only “certain” delay, e.g. if we know that the link takes at least 5ms in each direction. Works best with GPS time sources! • In general can’t do better than uncertainty in the link delay from the time source down to p

30. Consequences? • In a network of processes, we must assume that clocks are • Not perfectly synchronized. • We say that clocks are “inaccurate” • Even GPS has uncertainty, although small • And clocks can drift during periods between synchronizations • Relative drift between clocks is their “precision”

31. Temporal distortions • Things can be complicated because we can’t predict • Message delays (they vary constantly) • Execution speeds (often a process shares a machine with many other tasks) • Timing of external events • Lamport looked at this question too

32. Temporal distortions • What does “now” mean? p 0 a d b c e p 1 f p 2 p 3

33. Temporal distortions What does “now” mean? p 0 a d b c e p 1 f p 2 p 3

34. Temporal distortions Timelines can “stretch”… … caused by scheduling effects, message delays, message loss… p 0 a d b c e p 1 f p 2 p 3

35. Temporal distortions Timelines can “shrink” E.g. something lets a machine speed up p 0 a d b c e p 1 f p 2 p 3

36. Temporal distortions • Cuts represent instants of time. • But not every “cut” makes sense • Black cuts could occur but not gray ones. p 0 a d b c e p 1 f p 2 p 3

37. Consistent cuts and snapshots • Idea is to identify system states that “might” have occurred in real-life • Need to avoid capturing states in which a message is received but nobody is shown as having sent it • This the problem with the gray cuts

38. Temporal distortions Red messages cross gray cuts “backwards” p 0 a d b c e p 1 f p 2 p 3

39. Temporal distortions Red messages cross gray cuts “backwards” In a nutshell: the cut includes a message that “was never sent” p 0 a b c e p 1 p 2 p 3

40. Who cares? • Suppose, for example, that we want to do distributed deadlock detection • System lets processes “wait” for actions by other processes • A process can only do one thing at a time • A deadlock occurs if there is a circular wait

41. Deadlock detection “algorithm” • p worries: perhaps we have a deadlock • p is waiting for q, so sends “what’s your state?” • q, on receipt, is waiting for r, so sends the same question… and r for s…. And s is waiting on p.

42. Suppose we detect this state • We see a cycle… • … but is it a deadlock? p q Waiting for Waiting for Waiting for r s Waiting for

43. Phantom deadlocks! • Suppose system has a very high rate of locking. • Then perhaps a lock release message “passed” a query message • i.e. we see “q waiting for r” and “r waiting for s” but in fact, by the time we checked r, q was no longer waiting! • In effect: we checked for deadlock on a gray cut – an inconsistent cut.

44. Consistent cuts and snapshots • Goal is to draw a line across the system state such that • Every message “received” by a process is shown as having been sent by some other process • Some pending messages might still be in communication channels • A “cut” is the frontier of a “snapshot”

45. Chandy/Lamport Algorithm • Assume that if pi can talk to pj they do so using a lossless, FIFO connection • Now think about logical clocks • Suppose someone sets his clock way ahead and triggers a “flood” of messages • As these reach each process, it advances its own time… eventually all do so. • The point where time jumps forward is a consistent cut across the system

46. Using logical clocks to make cuts Message sets the time forward by a “lot” p 0 a d b c e p 1 f p 2 p 3 Algorithm requires FIFO channels: must delay e until b has been delivered!

47. Using logical clocks to make cuts “Cut” occurs at point where time advanced p 0 a d b c e p 1 f p 2 p 3

48. Turn idea into an algorithm • To start a new snapshot, pi … • Builds a message: “Pi is initiating snapshot k”. • The tuple (pi, k) uniquely identifies the snapshot • In general, on first learning about snapshot (pi, k), px • Writes down its state: px’s contribution to the snapshot • Starts “tape recorders” for all communication channels • Forwards the message on all outgoing channels • Stops “tape recorder” for a channel when a snapshot message for (pi, k) is received on it • Snapshot consists of all the local state contributions and all the tape-recordings for the channels

49. Chandy/Lamport • This algorithm, but implemented with an outgoing flood, followed by an incoming wave of snapshot contributions • Snapshot ends up accumulating at the initiator, pi • Algorithm doesn’t tolerate process failures or message failures.

50. Chandy/Lamport w t q r p s u y v x z A network