EXPLODE: A Lightweight, General Approach to Finding Serious Errors in Storage Systems

1. EXPLODE: A Lightweight, General Approach to Finding Serious Errors in Storage Systems Presented by Pramodh Pochu Junfeng Yang, Can Sar, and Dawso Engler

2. Agenda • Motivation • Introduction • Key Contributions • Checking a Storage System • Implementation • Exploring choices • Generating crash disks • Checking different storage systems • Conclusion • Evaluation

3. Motivation • Storage systems are important pieces of systems code • Storage systems are difficult to test • Verifying system recovery from a crash is difficult • Previous works (Traditional model checking and Implementation model checking) for finding data integrity bugs are heavy weight

4. Introduction • Traditional model checking takes as specification a model which it checks by starting from an initial state and repeatedly performing all possible actions to this state and successors • Traditional model checking requires rewriting the system using artificial modeling language Ex: Promela

5. Introduction • Implementation level model checking uses the code as its own model • So it eliminates need for writing a model • But it requires porting the entire OS to run on top of the model checker to run it as an user process • Verisoft and CMC are examples of these types

6. Key Contributions • Explode checkers are effective • Explode checkers can do thorough checks • A series of new file system specific checks for catching bugs in the data sync facilities used by applications • Simple checkers find bugs

7. Principles • Explore all choices : When program point can legally do one of N different actions, fork execution N times and do each • Exhaust states: Do every action before proceeding to next state • Touch nothing: modifying large checked systems may produce corner case mistakes. So those are not modified • Report true errors and deterministically

8. Checking Storage System • Clients use Explode for two things • Systematically exhaust all possibilities • Check that it correctly recovers from the crash • Clients provide three system specific components • Checker • Storage component • Checking stack

9. Choose • Explode provides “choose” fn. Given an program that has N possible actions clients insert a call “choose (N)”, Which will appear to fork execution N times returning 0,1,..N-1 in each child execution resp.

10. Choose • In low memory situations, kmalloc canreturn NULL when called without _GFP_NOFAIL flag. Void *kmalloc (size_t size, int flags) { If (flags & _GFP_NOFAIL)==0) If(choose(2)==0) return NULL; ……

11. Writing Checkers • Client provides a checker that Explode uses to drive and check a storage system • Checker implements five methods • Mutate: system specific operations and calls into Explode to explore choices and to do crash checking • Check: checking for storage errors after crash • get_sig: returns byte array signature representing the current state of the checked system

12. Writing Checkers • init and finish: setup and clear checkers internal states • File system checker checks if a file synchronously written to disk persists after a crash

15. Storage Components • For each storage subsystem involved in checking, clients provide a storage component that implements five methods • Init: for initialization • Mount: setup storage system • Unmount: tear down storage system • Recover: repair the storage system • Threads: return thread ids for the storage system’s kernel threads

16. Checking stack • Checking stack builds a checker by gluing storage system components together and then attaching single checker on top of them. • Currently stack can only have one checker • There can be fan-out of storage components, such as setting up multiple RAM disks to make a RAID array

17. Implementation • Create clean initial state and invoke client’s mutate on it • At every Choose (N) call, fork N children • On client request generate all crash disks and run the client check method on them • When mutate returns, re invoke it

19. Implementation

20. State Checkpoint and Restore • Explode checkpoints a state by recording sequence of choices the checked code took to reach S • It restores S by starting from a clean state and replaying these choices • Checkpoint records the sequence of n choices that produced S in an array

21. State Checkpoint and Restore • Unmounting clears in state memory, removes buffer cache entries and frees up kernel data structures. • To restore a state, the current disk is unmounted. Then a copy of the initial pristine disk is mounted and all previously made choices are replayed.

22. Re-executing the code deterministically • Doing the same choice: Explode discards any calls from an interrupt or calls from any other process whose id is not associated with checked system. • Controlling threads: Explode uses priorities to control when storage system threads run. • Requirements on checked system: Checked system must issue same choose calls across replay runs. The systems checked are partly isolated during checking and nothing besides checker and their kernel threads modifies RAM disks.

23. Generating crash disks • As the storage system executes, EKM logs operations that affect which blocks could be written to disk • Explode extracts this log using EKM ioctl • Then applies add/remove operations to initial write set • Whenever “write set shrinks”, it generates all possible crash disks.

24. Generating crash disks

25. Checking different storage systems • Explode checks different types of storage systems which include File systems, Version control systems , database etc., • Bugs were found in each of the storage system that is checked using Explode

26. Checking File System • Explode checks ten different types of Linux file systems • A common checkers is used initially • Three checkers are developed using the common checker which focused on different special cases

27. Generic checker core • Starts from empty file system and systematically generates FS topologies • Mutate applies eight system calls to each node (File, Link, Directory) in the current topology before exploring the next • For each operation invoked, mutate duplicates its effect on fake abstract file system.

28. Check : Failed systems calls have no user visible effect • It uses Explode to systematically fail calls to six kernel functions. • If a system call succeeds then it updates abstract file system but otherwise doesn’t . • It then checks abstract FS with real FS • Two bugs are found • Bug in ReisserFS ftruncate which can fail with its job half done if memory allocation fails.

29. Check: sync operations work • Applications use OS provided methods to force data to disk to prevent crashes from destroying it • SYNC • FSYNC • Synchronous mount • O_SYNC

30. Check: sync operations work

31. Check: a recovered FS is reasonable • After crash a file system recovers to a reasonable state • As Mutate issues operations, it builds two sets. • Stable set: contains operations it knows as contained on the disk • Volatile set: operations that may or may not be on the disk

32. Check: a recovered FS is reasonable • Check verifies that recovered file system can be constructed using some of the operations in volatile set legally combined with all the stable ones. • Two bugs (in JFS & Reiser4) are found. Crashed disks could not be recovered using FSCK. A bug was found in Reiser4, where mounting causes a kernel panic.

33. Checking Version Control • The checkers mutate method checks out a repository, does a local modification, commits the changes and simulates a crash on the block device. • It then calls check_crashes_now() • All three systems CVS, SubVersion and ExpENSiv made the same mistake. • To update File A, they update temp file B, which they then atomically rename it to A. However they forget to force B’s contents to disks before rename. So a crash can destroy it.

34. Checking Berkeley DB • The database checker checks that after a crash no committed transaction records are corrupted or disappear. • Mutate method is a simple loop that starts a transaction, adds several records to it an then commits this transaction. It records committed transactions • On ext2 creating a database inside the transaction may corrupt if system crashes.

35. Checking Berkely DB • Furthermore even committed transactions may disappear • On ext3 crash while adding a record can lead to unrecoverable state • On all three FS (ext2,ext3 and JFS) a record which is added but not committed can appear after crash

36. Checking RAID • Two checks are done • A file system’s behavior (crash & non crash) on top of RAID should be same as without it. • Losing any single sector in a RAID does not cause data loss • Explode found that Linux RAID does not reconstruct unreadable sector • If two sector read errors occurs then all maintenance operations fail. Disk writes also fail

37. Checking NFS • Setup NFS partition: Export local FS as an NFS partition over loopback interface • Tear : Unmount it • Recovery: FSCK for local file systems • Writing to a file, then reading the same file through a hard link in a different directory yields inconsistent data. • Linux NFS security feature called Subtree checking caused this error. • There are also additional bugs specific to individual file systems.

38. Conclusion • Lightweight approach can be used to find crash recovery errors • Explode runs on a slightly modified Linux kernel on raw hardware and • Explode is applied to a variety of storage systems and serious bugs are found.

39. Evaluation • Explode is more general, lightweight and easier to apply than FISC. • Efforts are being made to make Explode open source • Improvements have been made when compared to paper presented on Explode in 2004 at the workshop

40. Questions ?

41. Thank you

42. Related Work • File system testing tools • Software Model Checking • Generic bug finding

43. File System Checking Tools • Non deterministic • Less Comprehensive • But Lightweight and work ‘out of box’ • Complementary to explode

44. Software Model Checking • Model checkers are used to find errors in design and implementation of software systems • Verisoft doesn’t store states at checkpoints. It relies heavily on partial order reduction techniques. • Java pathfinder relies on Virtual machine, which extracts current state of a java program

45. Generic Bug finding • Static analysis is better at finding errors in surface properties visible in source code. • Model checking is more strenuous as it requires running code and moreover checks only the path executed • Because it executes code it can check properties implied by the code • Static analysis is complementary to Explode