Unfortunately

1. Unfortunately • Nothing works perfectly all the time. • What will happen if: • Your hard-drive crashes. • Your last transaction made a failure. • Power shutdowns while you’re saving changes on your database. Wiliwili…..

2. Recovery By: Sebbane Mehdi Supervised by:Dr. H.Haddouti Monday, April 14th 2003

3. Agenda • Intro… • Transactions Failures… • System Failures… • Media Failures… • Two-Phase Commit… • ARIES Recovery Algorithm… • Conclusion… • References…

4. Intro… • Data warehouse are mission-critical. • Downtime can lead to IMPORTANT loss in • Revenue • Productivity • Profitability & • Customers.

5. Intro… (more) • The most frequent causes for data warehouse downtime are storage related: • Component failure. • Lengthy load times. • Lengthy backups. • Others: • User Errors. • System failure.

6. Failures classification • Synchronous: • Trappable by the operating system. • No loss of data of any kind. • Possible causes: Program/logic errors: e.g. division by zero • Asynchronous: • System crash: • Assume loss of all data on volatile storage. • Possible causes: power failure, OS error. • Media crash: • Loss of data on online and volatile storage. • Possible causes: damage of storage media: human errors.

7. What is Recovery??? • Recovery = Redundancy…. • Simple example: • Periodically, copy or dump the database to an archive storage. • For every change, a log entry is made. • If failure: • A) Database damaged… • B) Content unreliable…

8. Why not Duplexing?? • Have two identical databases. • Applying changes simultaneously. • However, • Twice as much storage. • The 2 copies should be independent. To reduce the chance that a single failure affects both copies. Very hard to achieve.

9. Transaction Failures… • Intro… • Transactions Failures… • System Failures… • Media Failures… • Two-Phase Commit… • ARIES Recovery Algorithm… • Conclusion… • References…

10. Transaction Failures… • Intro… • Transactions Failures… • Transactions. • Message handling. • Transaction structure. • Transaction failures. • System Failures… • Media Failures… • Two-Phase Commit… • ARIES Recovery Algorithm… • Conclusion… • References…

11. Transactions… • The fundamental purpose of dbase system is to carry out transactions. • Transaction is: • the smallest unit of work… • Atomic… BEGIN TRANSACTION recoverable operations recoverable operations … … COMMIT or ROLLBACK

12. A recoverable operations are: All database updates (for which an entry has been loged) message I/O. Example: TRANSFERT $1000 3452332 TO 9087665 TRANSFERT: PROC; GET (FROM, TO, AMOUNT); FIND UNIQUE (ACCOUNT WHERE ACCOUNT = FROM); ASSIGN (BALANCE – AMOUNT) TO BALANCE; IF BALANCE < 0 THEN DO; ROLLBACK; PUT (‘INSUFFICIENT FUNDS’); ELSE DO; FIND UNIQUE (ACCOUNT WHERE ACCOUNT = TO); ASSIGN (BALANCE + AMOUNT) TO BALANCE; COMMIT; PUT (‘TRANSFER COMPLETE’); END; END /* TRANFERT */; Transactions… (more)

13. Transactions… (more) • To the user: “transfer x dollars from account A to account B” is a single operation. • Either succeed of fail. • Succeed: well & good • Fail: nothing should have changed in the database. What about messages??????

14. Message handling TRANSFERT: PROC; GET (FROM, TO, AMOUNT); FIND UNIQUE (ACCOUNT WHERE ACCOUNT = FROM); ASSIGN (BALANCE – AMOUNT) TO BALANCE; IF BALANCE < 0 THEN DO; ROLLBACK; PUT (‘INSUFFICIENT FUNDS’); ELSE DO; FIND UNIQUE (ACCOUNT WHERE ACCOUNT = TO); ASSIGN (BALANCE + AMOUNT) TO BALANCE; PUT (‘TRANSFER COMPLETE’); COMMIT; END; END /* TRANFERT */; • In the TRANSFER example: • the transaction not only updates the database, • it also sends messages to the end users: • INSUFFICIENT FUNDS. • TRANSFER COMPLETE. • Handling messages is done by the Data Communication Manager. • Note: output messages should not be transmitted until the planned end-of-transaction… WHY????? FAILURE

15. DC Manager • DC Manager that receives the original input message: (giving FROM, TO and AMOUNT) • writes a log record, and • places the message on the input queue • GET: retrieve a message from the input queue. • PUT: put a message in the output queue. • COMMIT & ROLLBACK affects also messages.

16. COMMIT & ROLLBACK • They cause the DC Manager to: • Write a log entry for the messages on the output queue. • Arrange for the actual transmission of those messages. • Remove messages from the input queue. Note: A transaction failure such as overflow causes the DC Manager to cancel the output messages

17. Transaction structure • As we can see from the TRANSFER examlpe: • Accept input message; • Perform database processing; • Send output message(s); • Too simple. • What about complex structures with multiple communications???

18. Complex structures • Two ways to deal with multiple communications: • Subdivide them into a sequence of simple transactions. • the database may be changed in the interval between two “conversations” • treated as one big transaction. • at any time the end-user must be prepared to get a message like “ignore all previous messages, a failure occurred”.

19. Transaction failures. • Transaction local failures that are detected by the application code itself (INSUFFISANT FUNDS). • Transaction local failures that are not explicitly handled by the application code (arithmetic overflow). • System-wide failures (CPU failure) that affect all transactions currently in progress but do not damage the database. • Media failures (disk head failure) that damage the database, or some portion of it, and affect all transactions currently using that portion.

20. Why??? • Conditions that may cause such terminations include: • arithmetic operation overflow, • division by zero, and • storage protection violation. • Transaction failure means that the program did not reach its planned termination.  ROLLBACK

21. How??? • undo all changes the transaction made on the database and cancel all output messages. • three basic types of changes: • updating an existing record, • deleting an existing record, or • inserting a new record. • For more convenience, the log file should be kept on a direct access device….. However….

22. What about if… • A failuer happens during rolling back… • UNDO must be idempotent : UNDO(UNDO(UNDO(. . . (x))))= UNDO(x) for all x. • As a transaction is a unit of work… It is also a unit of receovery.

23. System Failures… • Intro… • Transactions Failures… • System Failures… • Media Failures… • Two-Phase Commit… • ARIES Recovery Algorithm… • Conclusion… • References…

24. System Failures… • Intro… • Transactions Failures… • System Failures… • Checkpoints. • Types of transactions. • REDO. • Write-Ahead-Log. • System startup. • Media Failures… • Two-Phase Commit… • ARIES Recovery Algorithm… • Conclusion… • References…

25. System Failures… • we mean by this any event that cause the system to stop and thus require a system restart. • How the recovery manager knows at restart which transactions to rollback?? • checkpoints

26. Checkpoints • reduce the search time drastically. • very straightforward: Periodically, the system “takes a checkpoint”: • Force-writing any log records that are still in main storage out to the actual log; • Forcing a “checkpoint record” out to the log data set; • Force-writing any updates that are still in main storage out to the actual database; • Writing the address of the checkpoint record within the log data set into a “restart file”.

27. Checkpoints • Each checkpoint record contains: • A list of all transaction active at the time of the checkpoint; together with • The address within the log of each such transaction’s most recent log record. At restart time, the manager needs then to check which transactions need to be undone, and which should be redone.

28. T1 T2 T3 T4 T5 t1 t2 Checkpoint System crash Types of transactions • Consider the following: • A system failure has occurred at time t2. • The most recent checkpoint prior to time t2 was taken at time t1.

29. T1 T2 T3 T4 T5 t1 t2 Checkpoint System crash Types… • Transactions of type T1 were complete before time t1. • Transactions of type T2 started after time t1 and completed before time t2. • Transactions of type T3 started prior to time t1 and completed after t1 and before time t2.

30. T1 T2 T3 T4 T5 t1 t2 Checkpoint System crash Types… • Transactions of type T4 started prior to time t1 but did not complete by time t2. • Finally, transactions of type T5 started after time t1 but did not complete by time t2.

31. T1 T2 T3 T4 T5 t1 t2 Checkpoint System crash Types…. • What will happen at restart???? • Undo T4 & T5 • But also redo T2 & T3

32. REDO • Recovery manager able to track the log and invoke REDO for appropriate transactions. • Idempotent. • handling messages: • reschedule transactions of type T2 and T3. • force-write input messages log records.

33. Write-Ahead Log Protocol • Up to now, • Changing the database. • Writing the log record. • Two separate operations…. • What will happen after a failure occurring in the interval between the two. • Write-Ahead-Log…

34. Write-Ahead Log Protocol • for safety, the log record should always be written first. • A transaction is not allowed to write a record to the physical database until at least the undo portion of the corresponding log record has been written to the physical log. • A transaction is not allowed to complete COMMIT processing until both the redo and the undo portions of all log records for the transaction have been written to the physical log.

35. How does a system react to failures… 3 types: Emergency restart. the process that is invoked after a system failure has occurred. It involves the recovery procedures (UNDO or REDO). System startup

36. How does a system react to failures… 3 types: Emergency restart. warm start. the process of starting up the system after a controlled system shutdown. On a receipt of a SHUTDOWN command… System startup

37. How does a system react to failures… 3 types: Emergency restart. warm start. Cold start. Starting the system from scratch the process of starting the system after some disastrous failure that makes warm start impossible involves starting again from some archive version of the database. System startup

38. Media Failures… • Intro… • Transactions Failures… • System Failures… • Media Failures… • Two-Phase Commit… • ARIES Recovery Algorithm… • Conclusion… • References…

39. Media failures… • A media failure is a failure in which a portion of the secondary storage medium is damaged. • The recovery process consists of: • restoring the database from an archive dump, then • use the log to redo transactions run since that dump was taken.

40. Media failures… • A Media failure occurred…. • All current transactions will be abnormally terminated. • New device should be allocated to replace the one that failed. • A utility program is then run which: • load the database on to the new device from the most recent archive dump, and • use the log to redo all the transactions that completed since the dump was taken.

41. Two-Phase Commit… • Intro… • Transactions Failures… • System Failures… • Media Failure… • Two-Phase Commit… • ARIES Recovery Algorithm… • Conclusion… • References…

42. Two-Phase commit • Required whenever a transaction is able to invoke multiple independent resource managers. • No separate COMMITs. • Transaction issues a single “global” COMMIT to the coordinator. • Coordinator goes in the two phases:

43. Two-Phase commit • Phase I: • request all resource managers to get them selves into a valid state (commit or rollback) • If the resource manager succeeds reaching this state, it replies “OK” to the coordinator. • Phase II: • If all replies are “OK”, broadcasts the command “COMMIT” to all. • Otherwise, broadcasts the command “ROLLBACK”

44. ARIES Recovery Algorithm… • Intro… • Transactions Failures… • System Failures… • Media Failure… • Two-Phase Commit… • ARIES Recovery Algorithm… • Conclusion… • References…

45. reminder • Up to now, • Transaction failures. • System failures. • Media failures. • WAL,2PC, and checkpoint • All this is good, but how to combine it to have a better recovery……

46. ARIES Recovery Algorithm… • Intro… • Transactions Failures… • System Failures… • Media Failure… • Two-Phase Commit… • ARIES Recovery Algorithm… • The concept. • Data structure used in ARIES. • ARIES in details. • ARIES features. • Why ARIES. • ARIES optimizations. • Conclusion… • References…

47. ARIES Recovery Algorithm • The dominant crash recovery algorithm in commercial DBMSs. • Based on three concepts: • Write-Ahead logging; • Repeating history during REDO. • Retrace all actions of the DBMS prior the crash to reconstruct the database state when crash occurred. • Logging changes during UNDO. • Prevents ARIES from repeating complete undo operations when a failure occurred during recovery.

48. ARIES Recovery Algorithm • Step 1: analysis • Identify updated pages in the buffer. • Identify active transactions when the crash occurred. • Identify the point in the log where redo should start. • Step 2: REDO • Redo operations are applied until he end of the log. • Include writes from uncommitted transactions. • Only necessary redo operations are applied.

49. ARIES Recovery Algorithm • Step 3:UNDO • Log is scanned backward. • Updates from active transactions are undone.

50. Data Structures Used in ARIES