720 likes | 875 Views
Automated Tools for Software Reliability. Suhabe Bugrara suhabe@stanford.edu Stanford University. Problem. 80% of development cost on identifying and correcting defects Software errors cost US economy $60 billion annually (0.6% of GDP). Manual Testing.
E N D
Automated Tools for Software Reliability Suhabe Bugrara suhabe@stanford.edu Stanford University
Problem • 80% of development cost on identifying and correcting defects • Software errors cost US economy $60 billion annually (0.6% of GDP)
Manual Testing • Traditional approach to quality assurance • Expensive • Time consuming • Not systematic • Difficult to quantify effectiveness of test suite • Cannot make any guarantees about reliability • Insufficient for safety critical systems
Automated Tools • Programs to find defects in programs • Automated • Systematic • Easy to quantify effectiveness • Provide guarantees about reliability • Sometimes expensive (for now…) • Sometimes time consuming (for now…)
Program Analyzers Complete Incomplete Undecidable Decidable Sound • Reports all errors • Reports no false alarms • Reports all errors • May report false alarms Decidable Decidable Unsound • May not report all errors • Reports no false alarms • May not report all errors • May report false alarms
Static Driver Verifier • Program analyzer for API usage rules • Developed by Microsoft Research • Applied to device drivers in Windows • Sound: reports all possible errors • Incomplete: may report false alarms
SDV: Overview • Write API usage rule specification • Instrument program with usage checks • Abstract program • Check abstraction for errors • If error found, see if error is false alarm • If false alarm, refine abstraction • If not false alarm, report error as bug
API Usage Rules • Ex. locks are alternatingly acquired and released
API Usage Rules • Ex. locks are alternatingly acquired and released • Expressed as finite state machine • States = {locked, unlocked,error} • Transitions = {acquire(), release()}
API Usage Rules • Ex. locks are alternatingly acquired and released • Expressed as finite state machine • States = {locked, unlocked,error} • Transitions = {acquire(), release()} acquire(); unlocked locked release(); release(); acquire(); error
state { enum { Unlocked=0; Locked=1} state = Unlocked; } KeAcquireSpinLock.return { if (state == Locked) error(); else state = Locked; } KeReleaseSpinLock.return { if (!(state == Locked)) error(); else state = Unlocked; }
enum {Unlocked=0, Locked=1} state = Unlocked voidKeAcquireSpinLock_return() { if (state == Locked) error(); else state = Locked; } voidKeReleaseSpinLock_return() { if (!(state == Locked)) error(); else state = Unlocked; }
1: voidexample() { 2: do { 3: KeAcquireSpinLock(); 4: 5: nPacketsOld = nPackets; 6: req = devExt->WLHV 7: if (req && req->status) { 8: devExt->WLHV = req->Next 9: KeReleaseSpinLock(); 10: 11: irp = req->irp; 12: if (req->status > 0) { 13: irp->IoS.Status = SUCCCESS; 14: irp->IoS.Info = req->Status; 15: } else { 16: irp->IoS.Status = FAIL; 17: irp->IoS.Info = req->Status; 18: } 19: SmartDevFreeBlock(req); 20: IoCompleteRequest(irp); 21: nPackets++; 22: } 23: } while (nPackets!=nPacketsOld); 24: KeReleaseSpinLock(); 25: 26: }
enum {Unlocked=0, Locked=1} state = Unlocked voidKeAcquireSpinLock_return() { if (state == Locked) error(); else state = Locked; } voidKeReleaseSpinLock_return() { if (!(state == Locked)) error(); else state = Unlocked; }
1: voidexample() { 2: do { 3: KeAcquireSpinLock(); 4: KeAcquireSpinLock_return(); 5: nPacketsOld = nPackets; 6: req = devExt->WLHV 7: if (req && req->status) { 8: devExt->WLHV = req->Next 9: KeReleaseSpinLock(); 10: KeReleaseSpinLock_return(); 11: irp = req->irp; 12: if (req->status > 0) { 13: irp->IoS.Status = SUCCCESS; 14: irp->IoS.Info = req->Status; 15: } else { 16: irp->IoS.Status = FAIL; 17: irp->IoS.Info = req->Status; 18: } 19: SmartDevFreeBlock(req); 20: IoCompleteRequest(irp); 21: nPackets++; 22: } 23: } while (nPackets!=nPacketsOld); 24: KeReleaseSpinLock(); 25: KeReleaseSpinLock_return(); 26: } Program A
SDV: Abstraction • Construct abstractionB of original program A • Over-approximates reachability • If error() is reachable in A, then it is also reachable in B • This characteristic makes SDV sound • If error() is reachable in B, then it may not be reachable in A • This characteristic makes SDV incomplete • Check abstraction B for any errors
Reachable States Abstraction B real bug! error Original A Sound: If A has error, then B has error
Reachable States false alarm! Abstraction B error Original A Incomplete: If B has error, then A may not have error
bool b1; b1 = false; Abstract state == Locked withb1 voidKeAcquireSpinLock_return() { if (b1) error(); else b1 = true; } voidKeReleaseSpinLock_return() { if (!(b1)) error(); else b1 = false; }
1: voidexample() { 2: do { 3: ; 4: KeAcquireSpinLock_return(); 5: ; 6: ; 7: if (SdvMakeChoice()) { 8: ; 9: ; 10: KeReleaseSpinLock_return(); 11: ; 12: if (SdvMakeChoice()) { 13: ; 14: ; 15: } else { 16: ; 17: ; 18: } 19: ; 20: ; 21: ; 22: } 23: } while (SdvMakeChoice()); 24: ; 25: KeReleaseSpinLock_return(); 26: } Program B
1: voidexample() { 2: do { 3: ; 4: KeAcquireSpinLock_return(); 5: ; 6: ; 7: if (SdvMakeChoice()) { 8: ; 9: ; 10: KeReleaseSpinLock_return(); 11: ; 12: if (SdvMakeChoice()) { 13: ; 14: ; 15: } else { 16: ; 17: ; 18: } 19: ; 20: ; 21: ; 22: } 23: } while (SdvMakeChoice()); 24: ; 25: KeReleaseSpinLock_return(); 26: } Error trace found!
1: voidexample() { 2: do { 3: KeAcquireSpinLock(); 4: KeAcquireSpinLock_return(); 5: nPacketsOld = nPackets; 6: req = devExt->WLHV 7: if (req && req->status) { 8: devExt->WLHV = req->Next 9: KeReleaseSpinLock(); 10: KeReleaseSpinLock_return(); 11: irp = req->irp; 12: if (req->status > 0) { 13: irp->IoS.Status = SUCCCESS; 14: irp->IoS.Info = req->Status; 15: } else { 16: irp->IoS.Status = FAIL; 17: irp->IoS.Info = req->Status; 18: } 19: SmartDevFreeBlock(req); 20: IoCompleteRequest(irp); 21: nPackets++; 22: } 23: } while (nPackets!=nPacketsOld); 24: KeReleaseSpinLock(); 25: KeReleaseSpinLock_return(); 26: } But, no bug in original program!
1: voidexample() { 2: do { 3: ; 4: KeAcquireSpinLock_return(); 5: b2 = false; 6: ; 7: if (SdvMakeChoice()) { 8: ; 9: ; 10: KeReleaseSpinLock_return(); 11: ; 12: if (SdvMakeChoice()) { 13: ; 14: ; 15: } else { 16: ; 17: ; 18: } 19: ; 20: ; 21: b2 = !b2 ? true : SdvMakeChoice(); 22: } 23: } while (b2); 24: ; 25: KeReleaseSpinLock_return(); 26: } ProgramC
Reachable States Abstraction B error Refined C Original A false alarm no longer reported!
SDV: Summary • Write API usage rule specification • Instrument program with usage checks • Abstract program • Check abstraction for errors • If error found, see if error is false alarm • If false alarm, refine abstraction • If not false alarm, report error as bug
Soundness • Assume memory safety • No buffer/integer overflows • Safe memory management • No null pointer dereferences • Oversimplified harness • Use stubs to model calls into OS procedures • Stubs may not represent all behavior
Research Challenges in Verification • Eliminate assumption of memory safety • Eliminate false alarms • Scale to the entire operating system • Verify more complicated properties • prove consistency of file system data structures
Program Analyzers Complete Incomplete Undecidable Decidable Sound • Reports all errors • Reports no false alarms • Reports all errors • May report false alarms Decidable Decidable Unsound • May not report all errors • Reports no false alarms • May not report all errors • May report false alarms
EXE • Automatically generate test cases that explore important program paths • Developed by Dawson Engler’s group • Bug finding tool • Unsound: may not report all errors • Complete: never reports false alarms
intbad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; }
intbad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; }
intbad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; } (x >= INT_MIN) && (x <= INT_MAX) && (x < 0) && (ret = -x) find a solution using an automatic constraint solver… x = -1
intbad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; } (x >= INT_MIN) && (x <= INT_MAX) && (x >= 0) && (x = 12345678) && (ret = -x) find a solution using an automatic constraint solver… x = 12345678
intbad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; } (x >= INT_MIN) && (x <= INT_MAX) && (x >= 0) && (x != 12345678) && (ret = x) find a solution using an automatic constraint solver… x = 4
intbad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; } EXE automatically generated test cases for each path… x = -1 x = 12345678 x = 4
intbad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; }
1: intsymbolic_bad_abs (int x) { 2: add_constraints(x >= INT_MIN, x <= INT_MAX); 3: ret = new symbol; 4: 5: if (fork() == child) { 6: add_constraints(x < 0, ret = -x); 7: return ret; 8://(x >= INT_MIN) && (x <= INT_MAX) && (x < 0) && (ret = -x) 9: } else 10: add_constraints(x >= 0); 11: 12: if (fork() == child) { 13: add_constraints(x = 12345678, ret = -x); 14: return ret; 15: //(x >= INT_MIN) && (x <= INT_MAX) && (x >= 0) && (x = 12345678) 16:// && (ret = -x) 17: } else 18: add_constraints(x != 12345678); 19: 20: add_constraints(ret = x); 21: return ret; 22: //(x >= INT_MIN) && (x <= INT_MAX) && (x >= 0) && (x != 12345678) 23:&& (ret = x) 24:}
1: intmain (void) { 2: unsigned i, t, a[4] = { 1, 3, 5, 2}; 3: make_symbolic(&i); 4: 5: if (i >= 4) 6: exit(0); 7: 8: char *p = (char *) a + i * 4; 9: *p = *p – 1; 10: 11: t = a[*p]; 12: 13: t = t / a[i]; 14: 15: if (t == 2) 16: assert(i == 1); 17: else 18: assert(i == 3); 19: }
Review • Why does SDV produce false alarms and EXE doesn’t? • Why use SDV, then?
Saturn • Large-scale program verification • Developed by Alex Aiken’s group • Sound: reports all errors • Incomplete: may report false alarms • Gives guarantees of reliability on systems as large as the Linux kernel with over 6.2 million lines of code
Program Analyzers Complete Incomplete Undecidable Decidable Sound • Reports all errors • Reports no false alarms • Reports all errors • May report false alarms Decidable Decidable Unsound • May not report all errors • Reports no false alarms • May not report all errors • May report false alarms
Unchecked User Pointer Dereferences • Security property of operating systems • Two types of pointers in operating systems • kernel pointer: pointer created by the operating system • user pointer: pointer created by a user application and passed to the operating system via an entry point such as a system call • Must check that a user pointer points into userspace before dereferencing it