RUGRAT: Runtime Test Case Generation using Dynamic Compilers

1. RUGRAT:Runtime Test Case Generation using Dynamic Compilers Ben Breech NASA Goddard Space Flight Center Lori Pollock John Cavazos University of Delaware

2. Motivating Example if ((sptr = malloc (size + 1)) == NULL) { findmem (); if ((sptr = malloc (size + 1)) == NULL) xlfail (“insufficient string space”); } How do I test this callsite? • Make the machine run out of memory? • Flip the conditional, recompile, flip back? • Pretend it doesn’t exist during testing?

3. Generalizing the Problem • Code to handle uncommon situations • Difficult to test • May need external environment event to trigger • Examples: • Error handling code • Testing program security mechanisms

4. Observation • Hard to reach code executes when program thinks something uncommon has occurred if ((sptr = malloc (size + 1)) == NULL) { findmem (); if ((sptr = malloc (size + 1)) == NULL) xlfail (“insufficient string space”); } • Could test findmem() by simulating error • e.g., could add instructions to program so program believes malloc failed

5. RUGRAT Approach Use Dynamic Compilers to generate test cases for hard to reach code. Automatically add instructions to program during execution to simulate uncommon situation.

6. Dynamic Compilers • Dynamic compilers perform compilation tasks during program execution Analysis transformation optimization Dynamic Compiler Basic block Mod. Basic block Create basic block Executeon CPU code translate

7. RUGRAT Architecture Dynatest Generator Test spec Analysis transformation optimization Dynamic Compiler Basic block Mod. Basic block Create basic block Executeon CPU Create basic block code translate Test Report Test Oracle

8. Test Spec • Details where/how for inserting tests • Current prototype limited (environment vars). Can express: • Function locations • “test all calls to function x” • “test only second call to x in function y” • Failure value (e.g., 0, -1, etc) • Some side effects

9. Dynatest Generator • Scans instructions for location to insert test (e.g., call to function X) • Allows function X to execute • Adds instructions to simulate error • Instructions added after function X • Program thinks error happened, reacts

10. Example call malloc (code for malloc) movl 0, <return val> movl ENOMEM, errno movl <return val> sptr cmpl sptr, 0 jnz L1 call findmem …. L1: … call malloc (code for malloc) movl <return val> sptr cmpl sptr, 0 jnz L1 call findmem …. L1: … if ((sptr = malloc (size + 1)) == NULL) { findmem (); if ((sptr = malloc (size + 1)) == NULL) xlfail (“insufficient string space”); } L1: Dynatest Generator

11. The Good, the Bad and the Ugly The Bad: • Not a perfect simulation The Good: • Adequate simulation • Can target system or appl calls • Saves quite a lot of tester effort The Ugly: • Still a prototype

12. Security Mechanism Testing:Encrypting Function Pointers • Protects progs against func pointer attacks • Difficult to test (need vulnerable program and attack) • RUGRAT can simulate attack by adding instructions • Very different from error handling code case RUGRAT can be used for variety of testing tasks.

13. Current Implementation Notes • Used DynamoRIO1 dynamic compiler • Some limitations (but new version is available) • Test spec from env. vars • Nothing fancy for oracle 1 Bruening, et al., CGO 2003

14. Experiments • Ran variety of programs with RUGRAT • space, SPEC, MiBENCH • Tested handling of errors in • malloc / fopen / write • Application calls

15. Experiments Summary Can RUGRAT generate tests to cover error handling code? YES! RUGRAT tested error handling code at 120+ callsites (missed one because DynamoRIO incurred a segfault)

16. Experiments Summary Can RUGRAT increase statement coverage for error handling code? YES! RUGRAT increased code coverage ~ 50% (on average) of error handling code • Not all statements executed b/c of different options • RUGRAT detected cases of omission errors

17. Fault Detection • Could RUGRAT help detect failures in error handling code? • Grad students seeded faults into error handling code for space program • Changed assignments, loops, conditionals,etc • Seeded total of 34 faults

18. Fault Detection Summary • RUGRAT detected 15 / 34 faults • Of 19 undetected faults: • 6 changed return values, but callers only checked certain vals (e.g., if (func () != 0)) • 2 allocated too little memory (malloc may allocate more memory than requested anyway) • 2 unknown • 1 caused space to quit • 8 instances were caller performed same code as callee (e.g., any fault in callee was undid by caller)

19. Some related work • Holodeck1, FIG2 • Require tester provide alternative “stub” functions to do testing • Miss application calls • Dynamic branch switching3 • Not originally intended for testing error code • Need to know which branch to change • Far less accurate simulation 1 Thompson et al., SAC 2002 2 Broadwell et al., SHAMAN 2002 3 Zhang et al., ICSE 2006

20. Conclusions and Summary • Presented RUGRAT architecture • Can test hard to reach (and seldom tested) code by using dynamic compilers • Saves tester effort • RUGRAT is a general tool

21. RUGRAT Architecture Dynatest Generator Test spec Dynamic Compiler Basic block Mod. Basic block Executeon CPU Create basic block code Test Report Test Oracle

22. Experiments Summary • Tested variety programs with RUGRAT • 120+ error code handling callsites covered • Both application and system calls • Increased error code coverage ~ 50% over regular test cases • Not all error code statements could be covered • Different options, etc • Reasonable time overhead