Fault Tolerance

1. Fault Tolerance

2. Basic Concepts • Availability • The system is ready to work immediately • Reliability • The system can run continuously • Safety • When the system fails, nothing catastrophic happens • Maintainability • A failed system can be easily repaired. • Fault types: transient, intermittent, permanent

3. Failure Models • Different types of failures.

4. Failure Masking by Redundancy • Information redundancy (extra bits) • Time redundancy (extra operations) • Physical redundancy (extra equipment or processes)

5. Failure Masking by Redundancy • Triple modular redundancy (TMR). • An electronic circuit example

6. Process failures • To tolerate a faulty process, identical processes organized into a group • When one process of the group fails, some other process in the group takes care of the work • Process groups may be dynamic • Mechanisms are needed for managing groups membership • Group server maintains information on membership (centralized) • Distributed management(less simple and time consuming)

7. Flat Groups versus Hierarchical Groups • Communication in a flat group (voting mechanism, slow decision) • Replicated write protocols • Communication in a simple hierarchical group (single point of failure) • Primary based protocols

8. Client-server communication failures • Using a reliable transport protocol (TCP) masks omission failures, but many failures are not masked. • Classes of failure • The client is unable to locate the server – exception is a solution, but we loose in transparency • The request message from the client to the server is lost – retransmission • The server crashes after receiving a request • The reply message from the server to the client is lost – retransmission, but… • The client crashes after sending a request – orphan is generated. (extermination, reincarnation with epoch #, gentle reincarnation, expiration…)

9. Server Crashes (1) • A server in client-server communication • Normal case • Crash after execution • Crash before execution • At least oncesemantic: after server reboot, to try until a request is obtained • At most oncesemantic: immediate failure report • Exactly oncesemantic: no way

10. Server Crashes (2) Example: a client send a message to a server for printing (P) it, having a completion message back (M). The server can crash (C) • Different combinations of client and server strategies in the presence of server crashes.

11. Group CommunicationBasic Reliable-Multicasting Schemes Important for messaging in process group • A simple solution to reliable multicasting when all receivers are known and are assumed not to fail • a) Message transmission b) Reporting feedback • Efficient only for little # of receivers ( only nack, timer etc..)

12. Nonhierarchical Feedback Control To scale, we need to reduce the number of messages, with feedback suppression • Several receivers have scheduled a request for retransmission, but the • first retransmission request leads to the suppression of others (Scalable Reliable Multicasting protocol). • It leads to timing problems, useless retransmissions or a complicated organization of the group membership

13. Hierarchical Feedback Control • The essence of hierarchical reliable multicasting. A tree of receivers partitions is formed • Each local coordinator forwards the message to its children. • A local coordinator handles retransmission requests. • Acknowledge between coordinators

14. Atomic Multicast In presence of process failures, the guarantee that a message is delivered to all or none of the receivers is needed. This lead to the atomic multicast problemAtomic multicasting ensures that group members maintain consistency • The logical organization of a distributed system to distinguish between message receipt and message delivery • In atomic multicasting a multicast message is uniquely associated to a list of receiving processes ( Group view ) • A view change takes place when a process joins or leaves the group

15. Virtual Synchrony We need an ordered reliable multicasting. Virtual Synchrony guarantees that a message sent to a group view is delivered to each non-faulty member of the group. If the sender crashes, the message may be either delivered to all the other processes or ignored by each of them. • The principle of virtual synchronous multicast (view change similar to synchronization variable)

16. Message Ordering • Four different type of ordering of multicasts: • Reliable, unordered multicast • no guarantees is given on the order in which messages are delivered • FIFO ordered multicast • messages from the same process are delivered in the order as they are sent • Causally ordered multicast • causality between messages is preserved • Totally-ordered multicast • messages are delivered in the same order to all members of the group

17. Message Ordering • Unordered multicast: • Three communicating processes in the same group. The ordering of events per process is shown along the vertical axis. • Four processes in the same group with two different senders, and a possible delivery order of messages under FIFO-ordered multicasting

18. Message Ordering Virtually synchronous reliable multicasting offering totally ordered delivery is called atomic multicasting • Six different versions of virtually synchronous reliable multicasting.

19. Distributed Commit Distributed commit means that an operation has to be performed by each member of a group or none at all One phase distributed commit is performed using a coordinator (if a participant cannot perform the operation, no means to advise the coordinator) • The finite state machine for the coordinator in two phase commit. • The finite state machine for a participant. • The first phase is the vote phase, the second is the decision phase • Timeout mechanisms are necessary, coordinator can crash

20. Two Phase Commit • The coordinator send a vote_request to all participants • A participant returns a vote-commit (it is ready to commit its part of transaction) or a vote-abort • The coordinator collects the votes and send a global_commit or a global_abort (if one of the participants has sent a vote_abort) • A participant receive a global_commit and locally commits the transaction, or receive a global_abort and locally aborts the transaction • 1 – voting phase • 2 – decision phase 1 2

21. Three-Phase CommitIt avoids blocking processes in case of coordinator crash • There is no state from which it is possible to make a transition directly to COMMIT or ABORT • There is no state in which it is not possible to make a final decision and from which a transition to a COMMIT can be made

22. Recovery • Backward recovery brings the system to the previous correct state. It is necessary to record the state (check-pointing) • Forward recovery attempt to bring the system in a correct new state to continue the execution.