Effective Testing and Debugging Methods and Its Supporting System with Program Deltas

1. Effective Testing and Debugging Methods and Its Supporting System with Program Deltas Makoto Matsushita, Masayoshi Teraguchi, and Katsuro Inoue Osaka University

2. Software Maintenance Software maintenance procedures • Code modification to apply changes of specs • Regression testing and debugging if any bugs • Supporting tools for software maintenance • Version management system (VMS) • Regression testing tool ISFST2000 at Kanazawa Nov/02/2000

3. Debugging with Program Deltas It is hard to debug features which are not changed during maintenance Debugging with a code differences (deltas) • Using test results of each version • The deltas may contain some bugs ISFST2000 at Kanazawa Nov/02/2000

4. Ness and Ngo: Regression Containment • Using VMS and testing tool • Correct output (○) and wrong output (×) doesn’t appear by turns • Only a single “wrong” case • Only removes codes; no debugging ISFST2000 at Kanazawa Nov/02/2000

5. Zeller: Delta Debugging • Using test tool (no VMS is required) • Assumes “core” (△) output • Can’t consider development sequences • Only two “wrong” cases (△ and ×) • Only analysis; no debugging ISFST2000 at Kanazawa Nov/02/2000

6. Our Research Objectives An improved method with program deltas • Consider more “wrong” cases • Support actual debugging activities Debugging methods “DMET”, and its supporting environment “DSUS” ISFST2000 at Kanazawa Nov/02/2000

7. Today’s topic • Our debugging method “DMET” • “DSUS” supporting environment • DMET evaluation with DSUS prototype • Conclusion ISFST2000 at Kanazawa Nov/02/2000

8. An approach of DMET DMET is composed of three phases: • Detection: a program delta (may contains some bugs) to be detected • Indication: show a delta with latest code • Reflection: apply modification to old versions ISFST2000 at Kanazawa Nov/02/2000

9. DMET: Detection Phase • Using VMS and testing tools • Assumes wrong output (－) which is different from the one of latest version • Detect the last correct output and the oldest wrong output ISFST2000 at Kanazawa Nov/02/2000

10. DMET: Indication Phase • Program deltas are arranged as “removed” and “added” program codes • Detected deltas (maybe in older version) are mapped to the latest version ISFST2000 at Kanazawa Nov/02/2000

11. DMET: Reflection Phase Modification to the latest version are also applied to older versions • if application causes some errors, remove error version from test sequence • new sequence are used for further testing ISFST2000 at Kanazawa Nov/02/2000

12. Supporting System DSUS • A GUI-based SDE for debugging • Program editors • Compile, Execute a program • Cooperates other tools • RCS for version management system • DejaGnu for testing tools • Auto-execution support • check-in/check-out, regression testing • three DMET procedures ISFST2000 at Kanazawa Nov/02/2000

13. System Configuration • GUI: user interaction • DSUSmain: integration of other components • RCS: wrapper for RCS commands • DejaGnu: wrapper for DejaGnu commands ISFST2000 at Kanazawa Nov/02/2000

14. DSUS Screenshot ISFST2000 at Kanazawa Nov/02/2000

15. DSUS Evaluation • Two experimentations • Experimentation A: correcting programs which contain bugs in unchanged potion of program • Experimentation B: debug programs corrected by experimentation A • Preparation • Program specification and its program • Requirements for the new program • Test data ISFST2000 at Kanazawa Nov/02/2000

16. Experimentation Procedures • Experimentation A • Apply new requirement to a (prepared) program • Correct programs which has a bug in unchanged portion of the program • Experimentation B • Debug program collected in exp. A • Subjects are divided into two groups • debugging without DMET • debugging with DMET • Compare Elapsed time of debugging ISFST2000 at Kanazawa Nov/02/2000

17. Results of experiment A • Three programs are collected • DMET detects bugs in two programs; however, detected deltas of program Z doesn’t contain a bug ISFST2000 at Kanazawa Nov/02/2000

18. w/ DMET w/o DMET elapsed time (min) Program Results of experiment B Debugging time is reduced in X and Y, which bug is detected correctly. ISFST2000 at Kanazawa Nov/02/2000

19. Results of experiment B (cont.) According to Welch’s test (5 percents differences), a significant difference of X+Y between both groups is detected DMET support effective debugging if the detection is correct ISFST2000 at Kanazawa Nov/02/2000

20. Discussion In program Z, elapsed time of w/ DMET takes longer time than w/o DMET • A bug in program Z is the case that DMET doesn’t consider; it is caused since some modification should be needed • DMET generates wrong deltas • Testees are confused with deltas ISFST2000 at Kanazawa Nov/02/2000

21. Discussion (cont.) However, we may estimate that the deltas truly contains a bug since it’s doubtful that: • the distance of deltas is longer, compared with total numbers of version • the delta contains the “base version (1.1)” ISFST2000 at Kanazawa Nov/02/2000

22. Conclusion • DMET contributes to reduce debugging costs of the engineers, if the detected program delta contains bugs • DMET doesn’t consider all cases, it may generates wrong delta • If wrong delta, debugging takes longer time • However, it can be estimated ISFST2000 at Kanazawa Nov/02/2000

23. Further Research Topics • Some estimation methods for checking DMET deltas If the delta seems wrong, DSUS shows to the engineers that it maybe wrong. • DMET extension for new programs Current DMET assumes that “base version” exists and it is correct. New program have no “base version”. ISFST2000 at Kanazawa Nov/02/2000