CS614 – Byzantine Agreement

1. CS614 – Byzantine Agreement Ken Birman

2. Outline of talk • Byzantine model and problem statement • Impossibility results • Solutions to the problem • With digital signatures • Using randomized techniques • Big issues? • Is this model realistic? In what ways is it “legitimate”? • Does it make sense to pursue lower bound results? • Byzantine agreement in distributed security settings

3. Distributed Computing Models • Why models? • To reason about networks and applications we need to be precise about the setting in which our protocols run • But “real world” networks are very complex • They can drop packets, or reorder them • Intruders might be able to intercept and modify data • Timing is totally unpredictable

4. Example: How to lose a packet • The sending operating system might be low on memory and could drop it at the time it is sent • The network interface can fumble the transmission action • Networks are glued together with routers and gateways, which drop packets when overloaded • Receiver may be low on buffering space • Message can be fragmented… on timeout a delayed fragment causes loss of whole message • With checksums might discover data corruption

5. Sources of delay? • Network routes are constantly adjusted • Routers fail, become overloaded • Links fail or become congested • The operating system might be busy doing other things (and this is bursty and unpredictable) • The application may be swapped out or waiting for something else to happen (most protocols live in the application, not the operating system) • The network itself experiences bursts of load

6. How to model a network • Asynchronous approach: • Network can arbitrarily delay a message • But we assume that messages are sequenced and retransmitted (arbitrary numbers of times), so they eventually get through. • “Free” to say: lossless, ordered • No value to assumptions about process speed • Failures in asynchronous model? • Usually, limited to process “crash” faults • If detectable, we call this “fail-stop” – but how to detect?

7. An asynchronous network Not causal!

8. How to model a network • Synchronous model: • Assume that every link has bounded delay • Assume that every process will run within bounded delay • Usually described as “synchronous rounds” • Failures? • Can talk about message “omission” failures: failure to send is the usual approach • But network assumed reliable (loss “charged” to sender) • Process crash failures, as in asynchronous setting • “Byzantine” failures: arbitrary misbehavior by processes

9. A synchronous network

10. How to model a network • Both models view network as an undirected graph • Nodes are processes • Edges are communication links • No broadcast links (Babaoglu & Drummond considered this case but the work was unpopular) • Messages assumed to be forwarded but this doesn’t count into the round delays • Is this realistic? • A strange mix of reality and “arbitrary rules” • A network that only a theoretician could invent…

11. Comparison? • Neither model is realistic! • Asynchronous model is too weak (real systems have clocks, “most” timing meets expectations… but heavy tails) • Byzantine model is too strong (real systems lack a way to implement synchronize rounds) • Network wierdness • Failures? • Asynchronous model is too weak (for example, a mundane bug could easily corrupt a packet) • Byzantine model is too strong (assumes an adversary of arbitrary speed who designs the “ultimate attack”)

12. Justification? • If we can do something in the asynchronous model, we can probably do it even better in a real network • Clocks, a-priori knowledge can only help… • If we can’t do something in the Byzantine model, we can’t do it in a real network • After all, synchronized rounds are a powerful, if unrealistic, capability to introduce

13. Paradigms • Fundamental problems, the solution of which yields general insight into a broad class of questions • In distributed systems: • Agreement (on value proposed by a leader) • Consensus (everyone proposes a value… pick one) • Electing a leader • Atomic broadcast/multicast (send a message, reliably, to everyone who isn’t faulty, such that concurrent messages are delivered in the same order everywhere) • Deadlock detection, clock or process synchronization, taking a snapshot (“picture”) of the system state….

14. Consensus problem • Models distributed agreement • Comes in various forms (with subtle differences in the associated results)! • With a leader: leader gives an order, like “attack”, and non-faulty participants either attack or do nothing, despite some limited number of failures: Byzantine Agreement • Without a leader: participants have an initial vote; protocol runs and eventually all non-faulty participants chose the same outcome, and it is one of the initial votes (typically, 0 or 1): Fault-tolerant Consensus

15. Curses! I’m surrounded! Attack! Byzantine Agreement • General commands soldiers • If all loyal soldiers attack victory is certain • If none attack, the Empire survives • If some attack, Empire is lost • Gong keeps time • … but they don’t need toall attack at once

16. Byzantine soldiers • The enemy works by corrupting the soldiers • Orders are distributed by exchange of messages, corrupt soldiers violate protocol at will • But corrupt soldiers can’t intercept and modify messages between loyal troops • The gong sounds slowly: there is ample time for loyal soldiers to exchange messages (all to all)

17. More formal • Agreement: • Every correct node chooses the same value • If all the correct nodes have the same input, that input must be the value chosen

18. Impossibility Results • Let t be the maximum number of faulty processes that our protocol is supposed to tolerate • Byzantine agreement is not possible with fewer than 3t+1 processes • Same result holds for fault-tolerant consensus in the Byzantine model

19. Impossibility Results • Example: 3 processes, 1 is faulty (A, B, C) • Non-faulty processes start with input 0 and 1, respectively • They exchange messages: each now has a set of inputs {0, 1, x}, where x comes from C • C sends 0 to A and 1 to B • A has {0, 1, 0} and wants to pick 0. B has {0, 1, 1} and wants to pick 1. What to do? • Can we prove that consensus is impossible with just 3 processes?

20. History • Lamport, Shostak, Pease: 1980 paper defined the problem (“Reaching agreement in the presence of faults”), 1982 paper gave it a nice name, solved the problem, offered early proofs of some bounds • But protocols were sub-optimal, proofs hairy, and topic lived on for many years • Many variations on basic problem…

21. “Easy Proofs” paper • Fisher, Lynch and Merritt think of a process as a machine processing a tape of inputs • Called an agreement device • They build a communications graph. The messages that pass over an edge from a source to a destination node are a “behavior” of the device on that edge • Behavior of the system is a set of node and edge behaviors • In their proofs, faulty devices often exhibit different and inconsistent behaviors with respect to different participants

22. Locality • An axiom of this model • Basically says that the way a node will behave is completely determined by the inputs it starts with and that it receives on incoming edges • Fault axiom: a faulty device can mimic behavior of any correct device in any run. At the receiving end of an edge from it, the receiver can’t distinguish the faulty device from the device it mimics.

23. How they prove the 3t+1 bound • Start by assuming that the consensus problem can be solved; for 3 processes the system looks like this: A B C

24. Now build a covering graph • Looks like the original graph G – each node is attached to two others by edges • Also assign initial input values as shown A0 C1 A B0 B1 C0 A1 B C

25. Now focus on a first scenario • Consider B0 and C0 in a run where A is faulty A0 F C1 B0 B1 C0 A1 B0 C0

26. By assumption they reach agreement • In particular, they reach agreement if F mimics what A0 would have done on the edge (A,B) and what A1 would have done on the edge (A,C) • By the validity requirement, B and C must chose 0 • So A0 and A1 both pick 0 too, with these inputs

27. Consider a second scenario • Consider A1 and C0 in a run where B is faulty A0 A1 C1 B0 B1 C0 A1 F C0

28. Causing trouble • Suppose that F mimics B0 when talking to C • This is indistinguishable to C from the initial scenario • So C will need to decide 0 • By agreement, A also decides 0

29. Consider a final scenario • Consider A1 and B1 in a run where C is faulty A0 A1 C1 B0 B1 C0 A1 B1 F

30. Force a contradiction • Now we have the original setup with inputs 1 • Validity requirements force a decision value of 1 • But the edge behaviors for A are actually identical in the second and third scenarios! • We’ve shown that a single device, presented with identical inputs, would pick different values • A contradiction…

31. Generalize to arbitrary number of nodes • They partition the nodes into three groups, A, B and C, with at least 1 and at most 1/3 of the nodes in each group • They treat all the nodes in group A the way that we treated device A in our 3-node case, and similarly for B and C • Same argument again leads to contradiction

32. Other Byzantine impossibility results • Connectivity: 2t+1 connectivity required to achieve Byzantine agreement • They extend this to “weak” agreement • Every correct node chooses the same value • If all the correct nodes nodes are correct and have the same input, that input must be the value chosen

33. More Byzantine results • Time: t+2 rounds are required to tolerate t failures • Byzantine firing squad: • Say that when a node decides, it enters “fire” state • If a correct node enters the FIRE state at time t, every correct node enters the FIRE state at time t • If all nodes are correct and the stimulus occurs at any node, they enter the FIRE state after some finite delay. If there is no stimulus and all nodes are correct, no node enters the FIRE state

34. Possibility results • Using digital signatures, we can solve the problem fairly easily • A digital signature is some form of proof that a message is (1) uncorrupted, and (2) that it originated with the “claimed” sender • Suppose F asserts to B that it received m from A • Lacking signatures, F could corrupt the message • With signatures, F says nothing, or tells truth

35. Using signatures to solve Byzantine Agreement • Initially, leader (who could be faulty) broadcasts a proposed value, signing it. • Now run t additional rounds (for a total of t+1): • Input to round j for process p is a set of messages broadcast in round j-1 • Discard any that lack j signatures • Sign remaining messages • Broadcast these • In round t+1 if the leader proposed two values pick 0, else pick the leader’s proposed value, within the set of messages that have t+1 signatures.

36. A run with 2 failures A B C D {0}a {0}a,b “decide” 0 {0}a,b,d “decide” 0

37. A run with 2 failures A B C D {0}a {0}a,b {1}a {1}a,d,c “decide” {1}a,d {0}a,b,d…. “decide”

38. Making the decision • We need to convince ourselves that the correct processes make the same decision • That is, that they base their decision on the same set of messages • So, if correct process p includes message m in the set of messages it considers valid, we need to know that all other correct processes q also have m and consider it valid • Notice that m need not have reached q from the same source. That is, the signatures are used to decide which messages to consider, but are not “part” of the message • In this sense the set only contains 1 or 2 messages

39. Two cases • Message m was broadcast by a correct process in the final round. • In this case, since the process was correct, all other correct processes will have received m • Keep in mind that in this model, message loss is counted as a “failure of the sender process”

40. Two cases • What if message m was sent to p by an incorrect process? • There is a chain of t+1 rounds of relaying m • At most t failures occurred • So one round was a broadcast of m by a correct process • In the next round m reached every correct process • And so every correct process still has m in its “decision set”

41. Observations • Notice that the number of ways for m to reach any particular correct process could vary • Some correct process might have received m just once, while another received it t+1 times • If no failures occur, each process receives n messages on each round: O(n2) total • This is a fairly expensive protocol, but all Byzantine protocols are costly and slow!

42. Relatively recent work • Rabin introduced idea of a randomized protocol • Each process has a form of coin available to it • Can flip coin in each round, but behavior of coin is not predictable before that round occurs • With randomness he gets very rapid agreement “with high probability” in very little time • But this use of randomness remains controversial

43. Streets of Byzantium • Babaoglu and Drummond looked at Byzantine agreement with broadcast devices, like Ethernets • On these, one message will reach all receivers with high (not perfect) reliability • He shows that if broadcast is reliable, or if there are multiple independent broadcast devices, Byzantine agreement can be solved quickly • But results were not well received – model disputed

44. Byzantine Security • Reiter asked if security systems shouldn’t use Byzantine model in some ways • He developed group communication tools hardened against Byzantine failures within the application • This was well received, but can question decision to draw the line where he does • Does it make sense to assume Byzantine failures in the application, but not in the operating system, or the windowing system, or the network itself?

45. Space shuttle

46. Triple modular redundancy Vote

47. Stratus Computer • Pair and a spare: pairs of components with a comparison to check for faults • Offline the pair if any disagreement is found • Spare takes over in this case (and calls home for service)

48. Is the real world Byzantine? • Space shuttle cargo hold should have 4 actuators, not 3… • TMR is a standard for radiation-hardened applications. Is voter a “fourth” participant? • Stratus “pair and a spare” architecture: comparison isn’t following Byzantine rules

49. Questions for discussion • Suppose that we are offered upper and lower bounds for the problem. Are these of equal interest? • Is the Byzantine model reasonable at all? Are failures really Byzantine? • Is it right to treat message loss as a process failure? • Often these protocols “use up” the failures, e.g. by always running t+2 rounds even if no failures are actually occurring. Is this reasonable? • Costs are very high. How high a cost is tolerable?