1 / 38

Relaxing the Testcase Assumptions in GenProg

Relaxing the Testcase Assumptions in GenProg. Jonathan Dorn. EVALUATE FITNESS. INPUT. DISCARD. ACCEPT. GenProg. OUTPUT. MUTATE. EVALUATE FITNESS. INPUT. DISCARD. ACCEPT. GenProg. OUTPUT. MUTATE. EVALUATE FITNESS. INPUT. DISCARD. ACCEPT. GenProg. OUTPUT. MUTATE.

wynona
Download Presentation

Relaxing the Testcase Assumptions in GenProg

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Relaxing the Testcase Assumptionsin GenProg Jonathan Dorn

  2. EVALUATE FITNESS INPUT DISCARD ACCEPT GenProg OUTPUT MUTATE

  3. EVALUATE FITNESS INPUT DISCARD ACCEPT GenProg OUTPUT MUTATE

  4. EVALUATE FITNESS INPUT DISCARD ACCEPT GenProg OUTPUT MUTATE

  5. Guide the search. • Validate the repair. EVALUATE FITNESS Testcases in GenProg

  6. Indicate which functionality to repair. Indicate which functionality to retain. Negative Testcases Positive Testcases Testcases in GenProg

  7. Testcasesexist. • They are correct. • They are comprehensive. Testcase Assumptions

  8. Human-written testcases are expensive. • Development and maintenance costs. • Legacy code or remote deployments. • “Complete coverage” approached but rarely achieved. Testcase Difficulties

  9. Machine-generatedtestcases! How Can We Help?

  10. Automatic Testcase Generation • Evaluation • Preliminary Results Agenda

  11. “Competent Programmer” assumption • Correct behavior already encoded in program. • Extract and re-encode as testcases. Automatic Testcases

  12. Input Output Oracle Comparator Pass / Fail The Oracle Comparator Model

  13. Test Setup • Network connections, opened files, etc. • Argument values • Run Function Under Test • Check Results • Return value • Side-effects Structure of a Testcase

  14. Test Setup • Network connections, opened files, etc. • Argument values • Run Function Under Test • Check Results • Return value • Side-effects Automatic Testcases

  15. Test Setup • Network connections, opened files, etc. • Argument values • Run Function Under Test • Check Results • Return value • Side-effects Automatic Testcases

  16. DART • CUTE • CREST • … • Symstra • Austin • Pex Long-established research area Test Input Generation

  17. Generate initial input. • Run test; record constraints at branches. • Negate one constraint. • Solve for new input. • Repeat. Concolic Execution

  18. int f(int x) { • if (x < 10) • return 1; • else • return 2; • } Input: x = 456123 pred = {} Concolic Execution

  19. int f(int x) { • if (x < 10) • return 1; • else • return 2; • } Input: x = 456123 pred = {} Concolic Execution

  20. int f(int x) { • if (x < 10) • return 1; • else • return 2; • } Input: x = 456123 pred = {x>10} Concolic Execution

  21. int f(int x) { • if (x < 10) • return 1; • else • return 2; • } Input: x = 9 pred = {} Concolic Execution

  22. Test Setup • Network connections, opened files, etc. • Argument values  • Run Function Under Test • Check Results • Return value • Side-effects Automatic Testcases

  23. Test Setup • Network connections, opened files, etc. • Argument values  • Run Function Under Test • Check Results • Return value • Side-effects Automatic Testcases

  24. Oracle: check for invariants. • What are the interesting invariants? • Checking that 1 = 1 is not useful. • Invariants are true for all runs of program but violated for some runs of not-quite-the-same program. μtest

  25. f = min(a,b) • Identify predicates that are true for all inputs. μtest

  26. Mutate the function. • Identify predicates that fail in the mutants. μtest

  27. Take intersection as oracle invariants. May miss invariants if mutants do not fail. μtest

  28. Test Setup • Network connections, opened files, etc. • Argument values  • Run Function Under Test • Check Results • Return value  • Side-effects Automatic Testcases

  29. Does augmenting the human-generated test suite enable more fixes? • Do automatic testcases miss desired functionality? Research Questions

  30. Only generated testcases. • Pretend human test cases do not exist. • Generated testcases + X% of original. • How much is human burden reduced? Evaluation

  31. Preliminary Results

  32. Preliminary Results

  33. Preliminary Results

  34. *Invalid repair. **No repair found. Preliminary Results

  35. *Invalid repair. **No repair found. Preliminary Results

  36. Testcasesexist.(concolic execution + μTEST) • They are correct.(compent programmer assumption) • They are comprehensive.(may need small number of human tests) Testcase Assumptions

  37. We can create testcasesautomatically to augment human-created tests. • Initial results suggest these tests permit repairs without comprehensive test suites. Conclusion

  38. Questions?

More Related