Fault Tolerant Distributed Computing system.

1. Fault Tolerant Distributed Computing system.

2. Fundamentals • What is fault? • A fault is a blemish, weakness, or shortcoming of a particular hardware or software component. • Fault, error and failures • Why fault tolerant? • Availability, reliability, dependability, … • How to provide fault tolerance ? • Replication • Checkpointing and message logging • Hybrid

3. Message Logging • Tolerate crash failures • Each process periodically records its local state and log messages received after • Once a crashed process recovers, its state must be consistent with the states of other processes • Orphan processes • surviving processes whose states are inconsistent with the recovered state of a crashed process • Message Logging protocols guarantee that upon recovery no processes are orphan processes

4. Message logging protocols • Pessimistic Message Logging • avoid creation of orphans during execution • no process p sends a message m until it knows that all messages delivered before sending m are logged; quick recovery • Can block a process for each message it receives - slows down throughput • allows processes to communicate only from recoverable states; synchronously log to stable storage any information that may be needed for recovery before allowing process to communicate

5. Message Logging • Optimistic Message Logging • take appropriate actions during recovery to eliminate all orphans • Better performance during failure-free runs • allows processes to communicate from non-recoverable states; failures may cause these states to be permanently unrecoverable, forcing rollback of any process that depends on such states

6. Causal Message Logging • Causal Message Logging • no orphans when failures happen and do not block processes when failures do not occur. • Weaken condition imposed by pessimistic protocols • Allow possibility that the state from which a process communicates is unrecoverable because of a failure, but only if it does not affect consistency. • Append to all communication information needed to recover state from which communication originates - this is replicated in memory of processes that causally depend on the originating state.

7. KAN – A Reliable Distributed Object System • Developed at UC Santa Barbara • Project Goal: • Language support for parallelism and distribution • Transparent location/migration/replication • Optimized method invocation • Fault-tolerance • Composition and proof reuse

8. System Description Kan source Kan Compiler Java bytecode + Kan run-time libraries JVM JVM JVM UNIX sockets

9. Fault Tolerance in Kan • Log-based forward recovery scheme: • Log of recovery information for a node is maintained externally on other nodes. • The failed nodes are recovered to their pre-failure states, and the correct nodes keep their states at the time of the failures. • Only consider node crash failures. • Processor stops taking steps and failures are eventually detected.

10. Logical Node y Logical Node x Failure handler Fault Detector Request handler Communication Layer Basic Architecture of the Fault Tolerance Scheme Physical Node i External Log IP Address Network

11. Logical Ring • Use logical ring to minimize the need for global synchronization and recovery. • The ring is only used for logging (remote method invocations). • Two parts: • Static part containing the active correct nodes. It has a leader and a sense of direction: upstream and downstream. • Dynamic part containing nodes that trying to join the ring • A logical node is logged at the next T physical nodes in the ring, where T is the maximum number of nodes failures to tolerate.

12. Logical Ring Maintenance • Each node participating in the protocol maintains a variables: • Failedi(j): true if i has detected the failure of j • Mapi(x): the physical node on which logical node x resides • Leaderi: i’s view of the leader of the ring • Viewi: i’s view of the logical ring (membership and order) • Pendingi: the set of physical nodes that i suspects of failing • Recovery_counti: the number of logical nodes that need to be recovered • Readyi: records whether I is active. • Initial set of ready nodes; new nodes become ready when they are linked into the ring.

13. Failure Handling • When node i is informed of failure of node j: • If every node upstream of i has failed, then I must become new leader. It remaps all logical nodes from the upstream physical nodes, and informs the other correct nodes by sending a remap message. It then recovers the logical nodes. • If the leader has failed but there is some upstream node k that will become the new leader, then just update the map and leader variables to reflect the new situation • If the failed node j is upstream of i, then just update map. If I is the next downstream node from j, also recover the logical nodes from j. • If j is downstream of i and there is some node k downstream of j, then just update map. • If j is downstream of I and there is no node downstream of j, then wait for the leader to update map. • If i is the leader and must recover j, then change map, send a remap message to change the correct nodes’ maps, and recover all logical nodes that are mapped locally

14. Physical Node and Leader Recovery • When a physical node comes back up: • It sends a join message to the leader. • The leader tries to link this node in the ring: • Acquire <-> Grant • Add, Ack_add • Release • When the leader fails, the next downstream node in the ring becomes the new leader.

15. AQuA • Adaptive Quality of Service Availability • Developed in UIUC and BBN. • Goal: • Allow distributed applications to request and obtain a desired level of availability. • Fault tolerance • replication • reliable messaging

16. Features of AQuA • Uses the QuO runtime to process and make availability requests. • Proteus dependability manager to configure the system in response to faults and availability requests. • Ensemble to provide group communication services. • Provide CORBA interface to application objects using the AQuA gateway.

17. Proteus functionality • How to provide fault tolerance for appl. • Style of replication (active, passive) • voting algorithm to use • degree of replication • type of faults to tolerate (crash, value or time) • location of replicas • How to implement chosen ft scheme • dynamic configuration modification • start/kill replicas, activate/deactivate monitors,voters

18. Group structure • For reliable mcast and pt-to-pt. Comm • Replication groups • Connection groups • Proteus Communication Service Group for replicated proteus manager • replicas and objects that communicate with the manager • e.g. notification of view change, new QuO request • ensure that all replica managers receive same info • Point-to-point groups • proteus manager to object factory

19. AQuA Architecture

20. Fault Model, detection and Handling • Object Fault Model: • Object crash failure - occurs when object stops sending out messages; internal state is lost • crash failure of an object is due to the crash of at lease one element composing the object • Value faults - message arrives in time with wrong content (caused by application or QuO runtime) • Detected by voter • Time faults • Detected by monitor • Leaders report fault to Proteus; Proteus will kill objects with fault if necessary, and generate new objects

21. AQuA Gateway Structure

22. Egida • Developed in UT, Austin • An object-oriented, extensible toolkit for low-overhead fault-tolerance • Provides a library of objects that can be used to composelog-basedrollback recovery protocols. • Specification language to express arbitrary rollback-recovery protocols

23. Log-based Rollback Recovery • Checkpointing • independent, coordinated, induced by specific patterns of communication • Message Logging • Pessimistic, optimistic, causal

24. Core Building Blocks • Almost all the log-based rollback recovery protocols share event-driven structures • The common events are: • Non-deterministic events • Orphans, determinant • Dependency-generating events • Output-commit events • Checkpointing events • Failure-detection events

25. A grammar for specifying rollback-recovery protocols Protocol := <non-det-event-stmt>* <output-commit-event-stmt>* <dep-gen-event-stmt> <ckpt-stmt>op t <recovery-stmt>op t <non-det-event-stmt> := <event> : determinant : <determinant-structure> <Log <event-info-list><how-to-log>on <stable-storage>>opt <output-commit-event-stmt> := <output-commit-proto> output commit on < event-list> <event> := send | receive | read | write <determinant-structure> := {source, sesn, dest, dest} <output-commit-proto> := independent | co-ordinated <how-to-log> := synchronously | asynchronously <stable-storage> := local disk | volatile memory of self

26. Egida Modules • EventHandler • Determinant • HowToOutputCommit • LogEventDeterminant • LogEventInfo • HowToLog • WhereToLog • StableStorage • VolatileStorage • Checkpointing • …