Empirical Evaluation of Reliability Improvement in an Evolving Software Product Line

1. Empirical Evaluation of Reliability Improvement in an Evolving Software Product Line Sandeep Krishnan, ISU Robyn R. Lutz, ISU & JPL Katerina Goseva-Popstojanova, WVU This research is supported by NSF grants 0916275 and 0916284 Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

2. Background • Product line – “A family of products designed to take advantage of their common aspects and predicted variabilities” [Weiss and Lai 1999] • e.g. Nokia cellphones, HP printers, etc. • Products - • Commonalities – Shared by all products. e.g. Platform • Variabilities – Differentiate the products • High-reuse variation – JDT, PDE, Mylyn, Webtools, etc. • Low-reuse variation – CDT, Datatools, Java EE tools. Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

3. Motivation Does reliability improve as a product line evolves? Reliability – Continuity of correct service; measured in terms of post-deployment failures [Avizienis, et al., 2001] Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

4. Research Questions • Failure trends: Do serious failures decrease over time as the common/high-reuse variation/ low-reuse variation components evolve over releases? • Change trends: Does the percentage of new files and/or modifications to the source code decrease across releases for common/ high-reuse variation/ low-reuse variation components? • Failure/Change relationships: Does the number of serious failures normalized for source-code changes decrease over time for the common/ high-reuse variation/ low-reuse variation components? Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

5. Findings • Failure trends: Do serious failures decrease over time as the common/high-reuse variation/ low-reuse variation components evolve over releases? Common High-reuse Low-reuse • Change trends: Does the percentage of new files and/or modifications to the source code decrease across releases for common/ high-reuse variation/ low-reuse variation components? Common High-reuse Low-reuse • Failure/Change relationships: Does the number of serious failures normalized for source-code changes decrease over time for the common/ high-reuse variation/ low-reuse variation components? Common High-reuse Low-reuse Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

6. Approach 2007 2008 2009 2010 2011 • Source of failure reports- • Failure cateogries – • Raw number of severe failures and percentage. • Source of change reports – CVS repository of Eclipse. • We look at change to existing files measured by KChangesand change via new files [Mockus et al., 2000]. Europa Ganymede Galileo Helios Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

7. Observations Number of severe failures decreases over time as expected Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

8. Observations Percentage of severe failures tends to stabilize and even increases gradually Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

9. Observations Number of severe failures show mixed behavior, contrary to our expectations Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

10. Observations Percentage of severe failures tends to stabilize and even increases gradually, contrary to our expectations Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

11. Observations Number of severe failures do not monotonically decrease; rather show mixed behavior Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

12. Observations Percentage of severe failures show mixed behavior and not a decreasing trend Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

13. Observations • Common components: Percentage of new files decreases. • High-reuse variation components: Percentage of new files tends to remain stable. • Low-reuse variation components: Percentage of new files shows decreasing trend. • All components show relatively high modification rate. Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

14. Observations • Failures/new-file • Common components: For two subcomponents, release 1 has lower failures/new-file than release 4. • High-reuse variation components: Release 1 always has higher failures/new-file than release 4. • Low-reuse variation components: No specific trend. Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

15. Observations • Failures/Kchanges • No stable behavior for all components; mixed trends are observed. Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

16. Conclusion • Common components have fewer severe failures over time and amount of change also decreases. • Variation components show no uniform pattern of decrease in failures. • Even high-reuse variation components continue to change fairly rapidly. • Study suggests there may be no simple answer as to whether reliability tends to improve as product line matures. • Further investigation is needed on the relationship between failure and change in software product lines. Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

17. Thank You! Dept. of Computer Science, Iowa State University, MSR, May 22, 2011