Mining Software Repositories to Assist Developers and Support Managers

1. Mining Software Repositoriesto Assist Developers and Support Managers Ahmed E. Hassan SoftWare Architecture Group (SWAG) University Of Waterloo ahmed@alumni.uwaterloo.ca

2. Software Repositories • Track the evolution of a software project: • source control systems store changes to the code • defect tracking systems follow the resolution of defects • archived personnel communications record rationale for decisions throughout the life of a project • Used primarily for historical record supporting activities such as: checking bug status, or retrieving old code

3. Usage of Software Repositories Beyond Record Keeping • Prior Research: • SEL - NASA [Basilli, et. al]: • Fault analysis (Fortran 84) • 5ESS - Lucent [Eick, Graves, Mockus, Perry, et. al]: • code decay, fault and effort predictions, visualizations, process • EMERALD - Nortel [Hudepohl, Khoshgoftaar, et. al] • Telephony - Nokia [Gall et. al]: • reveal hidden dependencies • Open Source [Chen et. al]: • index code for searching • Personal Industrial Experience • Open Source Phenomena: easy access to rich repositories for large projects

4. Survey on Usage of Source Control Repositories in Industry • Developers: • More likely to record the change rationale than list implementation alternatives or limitations • Use change msgs to understand legacy code and fix bugs • Senior developers use messages to: • Track the progress of the project • Monitor the quality of the code (specific developers) • Ensure code ownership boundaries • Change messages in open source projects are as descriptive as ones in industrial systems

5. Bugzilla CVS Mailings

6. Bugzilla CVS Mailings Software Understanding Change Propagation Resource Allocation Bug Prediction Historical Information Extraction

7. Bugzilla CVS Mailings Software Understanding Change Propagation Resource Allocation Bug Prediction Historical Information Extraction

8. Extracting Information from Source Control Repositories • CVS tracks changes at the line level, instead an evolutionary extractortracks: • Addition, removal, or modification of code entities(functions, variables & macros): • function_1 is added/removed/modified • Addition and removal of dependencies: • function_1 no longer calls function_2 • Attaches additional CVS change information: developer name, co-changed entities, change reason

9. Complexity of Evolutionary Extraction main() { int a; /*call help*/ helpInfo(); } helpInfo() { errorString! } main() { int a; /*call help*/ helpInfo(); } helpInfo() { int b; } main() { int a; /*call help*/ helpInfo(); } V1: Undefined func. (Link Error) V2: Syntax error V3: Valid code

10. Evolutionary Extractors: Challenges and Complexity • Unstable & Evolving Code • Scalability and Robustness • Large amount of data/snapshots • Legacy systems are hard to parse • Accuracy • Entities with similar names throughout time

11. Assisting Software Developers

12. Understanding the Software Architecture ? Why? Who? When? Where? Conceptual (proposed) Concrete (reality)

13. Dependency Investigation Questions (W4 Approach) • Which low-level code entity is responsible for a dependency? • Network (SendData) Scheduler (PrintToLog) • Whoadded/removed the dependency? • Junior vs. senior/experienced developer • When was the dependency modified? • Late night / Just before release • Whywas the dependency added/removed? • The rationale!

14. Source StickyNotes • Static dependencies give a static view of the system – not enough detail! • Need to extend static dependencies, but how? • Ask developers to fill StickyNotes for each change • Use software repositories to build these notes automatically

15. Case Study – NetBSD(VM Subsystem) ? Why? Who? When? Where? Conceptual (proposed) Concrete (reality)

16. Unexpected Dependencies • Eight unexpected dependencies • All except two dependencies existed since the first day: • Virtual Address Maintenance Pager • Pager  Hardware Translations

17. Understanding Software Using StickyNotes • Traditional dependency graphs and program understanding models usually do not use historical information • In many open source projects, CVS comments are used for: • Communicating new features • Explaining design decisions • Narrating the progress of a project

18. Supporting Managers

19. Code Development Process • Central part in making software – Code changes to: • Implement/Enhance features • Repair faults • Acts as an early warning: • Does a complex code development process produce complexsoftware? • Can we: • measure its complexity? • monitor the evolution of its complexity? • predict problems (delayed releases or faults)?

20. Development Process Chaos Domain A chaotic code development process negatively affects its outcome, the software system, such as the occurrence of faults Requirements Development Process Code/Design Team (size/structure) “Complexity is the business we are in and complexity is what limits us”.Fred Brooks, The Mythical Man-Month Schedule Pressure

21. Complexity of Development Process • “To implement feature A we modified one file” • “To implement feature B we modified too many files – I don’t even remember half of them!” • Too much information to recall Decay of grasp of what is going in a complex system

22. Information Theory • Measures the amount of information in a message • Shannon Entropyfor a random prob distrib. P • pk = 1: • If all pk = 1/n (equal prob.) maximal entropy • If exists pi = 1minimal entropy • Min. number of bits – Best compression Hn(P) = - pklog2(pk)

23. Info Theory in Dev Process

24. Evolution of Entropy

25. Adaptive System Sizing • System size changes over time • Standardized Shannon Entropy H: • Many files are rarely modified (eg. util or libs): • A large n reduces the entropy • Use an active working set instead of n = # of files H(P) = (log2 n)-1 * Hn(P) = - (log2 n)-1 * pklog2(pk)

26. Postgres – Development Chaos “This release marks a major step in the development team's mastery of the source code we inherited from Berkeley. You will see we are now easily adding major features, thanks to the increasing size and experience of our world-wide development team” 6.5 release notes

27. Case Study 2: KDE KDE 2.0 was also a very ambitious project, and the nature of the changes delayed the project for a whole year”, Miguel de Icaza, founder of GNOME and Ximian

28. Mathematical Validation Details • Studied repositories of six open source software systems (OpenBSD, FreeBSD, NetBSD, Postgres, KDE, KOffice) • Built statistical regression models using the 2nd and 3rd years: • Faults • Changes • Entropy metric data to predict faults in 4th and 5th years

29. Results of Mathematical Validation • Prior faults better fault predictor than prior changes (~50% improvement in prediction error) • Entropy metric better fault predictor than: • Changes (~13-40% improvement) • Faults (~15-38% improvement) • Results statistically significant at (95%) using a paired T-test

30. Development Chaos Summary • A new perspective on the complexity of software – focus on process instead of code • Entropy metrics used to study the evolution of complexity and predict faults • Results verified using 6 open source projects

31. Conclusion • Software repositories contain a wealth of valuable information about the evolution of projects • Presented approaches and techniques to: • Recover data from source control repositories • Assist developers in understanding code structure • Support managers in predicting the occurrence of faults • MSR likely to take a central role in supporting software development practices and research

33. Thesis Contributions • Evolutionary Software Extractors (C-REX) • For Developers: • Source StickyNotes • Development Replay Approach • For Managers: • Top Ten List • Software Development Chaos • All contributions were validated through: • case studies using open source systems, and • a survey of software practitioners

34. Research Overview • Develop approaches and techniques to : • Part I : Extract Information from Repositories • Part II : Assist Developers in • Understanding the current code structure • Changing code and propagating changes • Part III: Support Managers in • Allocating limited testing resources • Predicting faults

35. Part II: Assisting Developers • Source StickyNotes: • Assist in architecture understanding • Attach historical information to the dependency graph • Development Replay: • Reenacts historical changes using proposed tools and strategies to assess their benefits • Permits us to perform empirical based estimation of the benefits of tools and strategies, e.g. change propagation

36. Part III: Supporting Managers • Top Ten List: • Assists managers in allocating limited resources • Development Process Chaos: • Develop a complexity metric based on the process followed by the developers to produce the code instead of on the code or the requirements

38. Research Hypothesis “Software repositories contain a wealth of valuable information about the evolution of a software project. By mining such historical information, we can develop techniques and approaches to support software developers and managers in their endeavors to build and maintain complex software systems”