1 / 22

Studying the test process in open source and industrial software systems

Studying the test process in open source and industrial software systems. Andy Zaidman, Bart Van Rompaey, Serge Demeyer, Arie van Deursen TU Delft, University of Antwerp Andy Zaidman, IPA Herfstdagen 24 th November 2008. How does test code * co-evolve?. WHY?. HOW?.

lowell
Download Presentation

Studying the test process in open source and industrial software systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Studying the test process in open source and industrial software systems Andy Zaidman, Bart Van Rompaey, Serge Demeyer, Arie van Deursen TU Delft, University of Antwerp Andy Zaidman, IPA Herfstdagen 24th November 2008

  2. How does test code* co-evolve? WHY? HOW? (*) Developer tests

  3. Well, why would you… … want to know the “quality” of the tests? • When maintaining the system…  can you trust the existing tests/test process? • When judging the overall quality of a software system…  is it good enough for our standards? • When monitoring the test process…  Comparing intended with actual process

  4. You might say … “What about test coverage?” Nice, but somewhat hollow measure • Easy to keep artificially high • What about boundary testing? • “Exercising versus testing…” • Test process?

  5. How?

  6. Repository mining • Software developers like to use VCS • e.g., CVS, SVN, Clearcase, MS Visual Sourcesafe, … • Backing up their work • Making it easy to work in groups • Easy to manage releases • On the other hand… • A wealth of information is stored in there • “It’s like a free museum” of past development activities and development decisions • Let’s use this!

  7. “Test health” • Determine the long-term quality of your test suite. • Don’t only look at the “testing product”, but also at the “testing process” • Tells something about the future of the testing (e.g., maintenance of the tests) “Test health” was introduced by Kent Beck

  8. “Test health” VCS views • Change History View • When are tests adapted? • Growth History View • Determine impact of changes to production and test code. • Test Writing Efficiency View • How much test code for coverage?

  9. 3 subject software systems • Checkstyle • Java source-code style checker (and improver) • ~ 6 years development, 2260 commits, 6 developers, 738 classes, 47 kSLOC • ArgoUML • UML drawing application • ~ 7 years development, 7477 commits, 42 developers, 1533 classes, 130 kSLOC • 1 industrial system • Software Improvement Group (SIG), Amsterdam, The Netherlands

  10. Example change history view

  11. Checkstyle Change History

  12. SIG Change History “CS”

  13. Checkstyle Growth History %

  14. ArgoUML growth history view

  15. SIG Growth History

  16. Growth History co-evolution scenarios

  17. Checkstyle test writing efficiency

  18. SIG test writing efficiency

  19. Test writing efficiency patterns

  20. Conclusion? • In our case studies: • Big difference between OS systems and SIG! • SIG • Continuous testing effort • eXtreme Programming + test-driven development • OS • Sometimes 6 months of no testing • No increase in testing before major release • But sometimes after! Buy 2.1 instead of 2.0!!! • ArgoUML project was started without testing process in place

  21. Next steps in this research • Try this on more (diverse) software systems • e.g., OS projects from Apache group • Try to correlate the testing effort to the number of bugs reported in Bugzilla • More bugs when testing effort is less good, and vice versa? • More information? • See our ICST 2008 paper (it’s included in your IPA Fall Days welcome pack!) • An STVR journal paper is currently in submission

  22. ?a.e.zaidman@tudelft.nlhttp://www.st.ewi.tudelft.nl/~zaidman

More Related