On software maintenance and evolution

1. On software maintenance and evolution Michael W. Godfrey Software Architecture Group (SWAG) University of Waterloo

2. Overview • Software aging • Software “maintenance” • Software “evolution” • A case study of growth and evolution Michael W. Godfrey

3. Software Aging Michael W. Godfrey

4. Parnas on “Software Aging” • “Bit rot” isn’t the problem! • Rather: • Our expectations change, new demands, new environments, emergent properties, … • e.g., MS-Word generates HTML, OLE, vmware, network drives, XML as a RDBMS, … Michael W. Godfrey

5. Parnas on “Software Aging” BUT • Large software systems are complex • Difficult and expensive to change! • “Legacy” systems are expensive assets! • “Ignorant surgery” leads to brittle, hard-to-change systems • Business pressures lead to “JIT maintenance” • Accidental complexity accrues over time Michael W. Godfrey

6. Parnas on “Software Aging” • Parnas’ advice: • Design for change! • Isolate likely-to-change hotspots and virtualize • Keep documentation up-to-date • Perform design reviews • Perform redesigns • But we know this already, right? Michael W. Godfrey

7. Software “maintenance” Michael W. Godfrey

8. A thought experiment … • Ignore software for a moment … • What does the term “maintenance” connote? • Keep it running, replace worn out bits, upgrades sometimes, … • Is this what software maintainers do? • Mostly not! Michael W. Godfrey

9. Kinds of software maintenance • What are the common categories of software maintenance? • Corrective • Adaptive • Perfective • Preventative • Alas, there is much confusion about what these terms (ought to) mean! Michael W. Godfrey

10. Swanson’s original view [ICSE 1976] • Corrective:[Fix program so that it meets its specs] • Processing failure • wrong answer or program aborts • Performance failure • doesn’t satisfy non-functional reqs, esp. real time performance • Impl. failure • programming standards ignored, impl. inconsistent/incomplete, reqs not implemented correctly • Adaptive: [Fix to adjust to new underlying env./formats] • Change in data format • Change in processing environment Michael W. Godfrey

11. Swanson’s original view [ICSE 1976] • Perfective: [It works fine, but let’s make it even better] • Processing inefficiency • Even in performance is within specified params, may still want to make it faster • Performance enhancement • Make the output more readable (NOT related to response time) • Maintainability • Better use of comments, documentation (NOT refactoring/restructuring/redesigning) Michael W. Godfrey

12. Swanson’s original view [ICSE 1976] • Corrective:[Fix program so that it meets its specs] • Adaptive: [Fix to adjust to new underlying env./formats] • Perfective: [It works fine, but let’s make it even better] BUT • Where does preventative maintenance fit in? • Where does “make it faster” fit in? Michael W. Godfrey

13. Swanson/Leintz [CACM 1978] Refined model, most commonly quoted: • Corrective: • fixing bugs • Adaptive: • adapting to new environment • but no outward semantic change, no new features • Perfective: • any other kind of change intended to make the system “better” • e.g, performance tuning, adding new features, “preventative maintenance” (redesign/refactoring) Michael W. Godfrey

14. IEEE Std on Software Engineering Terminology [1990] • Software maintenance • “The process of modifying a software system or component after delivery to correct faults, improve performance, or adapt to a changed environment” • Corrective maintenance • “Maintenance performed to correct faults in hardware or software.” • Adaptive maintenance • “Software maintenance performed to make a computer program usable in a changed environment.” • Perfective maintenance • “Software maintenance performed to improve the performance, maintainability, or other attributes of a computer program.” • Preventative maintenance • “Maintenance performed for the purpose of preventing problems before they occur.” Q: Adding new features? Michael W. Godfrey

15. IEEE Std on Software Maintenance Terminology [1998] • Software maintenance • “Modification of a software product after delivery to correct faults, improve performance or other attributes, or adapt to a modified environment” • Corrective maintenance • “Reactive modification of a software product performed after delivery to correct discovered faults.” • Adaptive maintenance • “Modification of a software product performed after delivery to keep a computer program usable in a changed or changing environment.” • Perfective maintenance • “Modification of a software product after delivery to improve performance or maintainability.” • Preventative maintenance … is MIA Q: Adding new features? • Part of the “environment”? • Not a maintenance activity per se? Michael W. Godfrey

16. The current view There’s no standard taxonomy … but here’s a reasonable view: • Corrective maintenance • Fix bugs • Adaptive maintenance • Add support for new deployment environments • May include “adding new features” • Perfective maintenance • Improve non-functional requirements (performance, security) apart from portability • May include “adding new features” • May include preventative maintenance • Preventative maintenance • Improve internal design w/o changing external semantics Michael W. Godfrey

17. Software maintenance vs. evolution Michael W. Godfrey

18. Software maintenance vs. evolution • So far, we’ve treated initial delivery as the major milestone in a product’s lifespan. • What about changes during “initial” development? • Is it reasonable to make such a sharp distinction between pre- and post-initial delivery? • Are there other important phases in a software systems? • What are the forces at play as a system evolves? • Is “evolution” a better term than “maintenance”? Michael W. Godfrey

19. Waterfall-like process model Requirements engineering Design and implementation Testing Michael W. Godfrey

20. Staged model of maintenance and evolution[Bennet/Rajlich 2000] • The staged model recognizes that there are fundamentally different life phases of a software system! • At each stage you are typically performing mini-waterfall development cycles … differently! • The problems, tensions, solutions, processes are very different! • Need to understand what is appropriate when. • Lots of research needs to be done here! • What is an evolvable architecture? • What are the likely effects of different kinds of change? … etc Michael W. Godfrey

21. Staged model of maintenance and evolution[Bennet/Rajlich 2000] • Initial development • First version developed, already possess the architecture and the fundamental system knowledge. • [Active] evolution • Takes place only when the initial development was successful, the goal is to adapt the application to the ever-changing user requirements. • Servicing • Once the architecture or the knowledge disappears, product is no longer evolvable (i.e., enters the service stage) • Phase out • no more servicing. Michael W. Godfrey

22. Staged model of maintenance and evolution[Bennet/Rajlich 2000] • Initial development • System architecture is devised • Often create a runnable infrastructure as soon as possible, gradually fill in missing/fudged bits [aka staged delivery] • Design for change! • Consider strategically where you expect changes to occur and design your architecture accordingly • Ideal: Cost of making a future change is proportional to size/complexity of change, not size/complexity of system • Change is the rule, not the exception. • Developers are aware of this. • QA artifacts developed • Knowledge about “the system” is fresh, accurate; experts can be found easily Michael W. Godfrey

23. Staged model of maintenance and evolution[Bennet/Rajlich 2000] • Active evolution • Knowledge about system is good; some personnel turnover since initial development. • Simple changes and adaptation to new requirements/environments are mostly straightforward • System architecture is flexible enough to allow for more wide-ranging changes to be planned and executed successfully • Architecture can be redesigned if necessary, and still be coherent and reasonably faithful to the original system Michael W. Godfrey

24. Staged model of maintenance and evolution[Bennet/Rajlich 2000] • Servicing • Simple changes and adaptation to new requirements/environments can still be done • Architectural change is harder and risky! • Knowledge about system is disappearing • JIT maintenance starts to occur • Effects of change (e.g., on other parts of system) become harder to predict • Start to treat system more like a caged animal than an intellectual creation • May perform preventative maintenance (program comprehension, reverse engineering) tasks to recover design, stave off death • In practice, it’s very hard to swim upstream back to “evolution” stage • Legacy wrapping, adapters, regression testing, impact analysis, … Michael W. Godfrey

25. Staged model of maintenance and evolution[Bennet/Rajlich 2000] • Phase out • System is getting too risky and expensive ($$$, performance, time/effort) to keep. • A new solution is sought • An exit strategy must be devised • Common approaches: • Create new adapter-style interface for clients/rest of system • Perform data recovery and migration • Deploy new solution to fit adapter interface, integrate with recovered data • Shut down old system Michael W. Godfrey

26. Maintenance vs. evolution Michael W. Godfrey

27. Software maintenance vs. evolution • Maintenance connotes • Fixing, rather than intellectually enhancing • Short-term, not long-term goals • Many prefer the term evolution: • Fundamental change and adaptation • Short- and long-term change • Planned and unplanned phenomena [my two cents] Michael W. Godfrey

28. What is evolution? • My answer: • Essentialchange in some identifiable thing (or family) over time. • Warning: Some very personal views follow. • They are far from widely accepted. Michael W. Godfrey

29. Evolution: Software vs. biology • Biology has the hard scientific theory and empirical results, but evolution is a generally occurring phenomenon! • Frogs, economies, ideas, music all evolve… and software does too! • Although the phenomenon of evolution is apparent widely, it is hard to be truly scientific in its study outside of biology. Michael W. Godfrey

30. Three fundamentals ideas of evolution • Mechanisms that increase variation / act as an agent for essential change. • Mechanisms that decrease variation / act as an inhibitor for essential change. • It’s all relative, man. • Change occurs within an environment, which also evolves. • That is, what succeeds today may not succeed tomorrow (but may succeed the day after). Michael W. Godfrey

31. Biological evolution in a nutshell “[Biological] evolution is change in a gene pool of a populationover time.” [talk.origins FAQ] • Genes/alleles are the essential encoding that is inherited by living creatures; they comprise the genotype of the individual. • A gene is the dedicated “slot” or memory location • Alleles are the set of abstract values at those genes • DNA are the bits that encode the allele messages (ACGT) • The phenotype is the totality of what an individual becomes. • The phenotype is determined by the genotype PLUS interaction with the environment. • Non-genotypic change is not inheritable! [Lamarck was wrong!] Michael W. Godfrey

32. Biological evolution in a nutshell • Consider: • Biston betularia, an English moth • 2% dark in 1848, 95% dark in 1898 [Manchester] • Hair colour, tattoos, height • Morphological change  evolution • Evolution  morphological change • Sickle cell anemia • Evolution is not progress! • Biological evolution is a greedy algorithm • Individuals are embedded in and alter their environments! • They consume resource, alter the competitive balances, and modify their environment to suit themselves as they are able Michael W. Godfrey

33. Biological evolution in a nutshell • Mechanisms that increase genetic variation • Mutation (new values) • Recombination (new combinations) • Gene flow • Genetic drift (chance increase in existing allele freq.) • Mechanisms that decrease genetic variation • Natural selection (!!) • Sexual selection • Genetic drift (chance decrease in existing allele freq.) Michael W. Godfrey

34. Biological vs. software evolution • How hard should we push the analogy of biology into software? • OK, genotype == source code, but what is a software phenotype? • What is a software individual/population? I won’t try to answer this … Michael W. Godfrey

35. Biological vs. software evolution • One important fact: • Unlike biology, the prime agent for new values/traits of software is not chance, it’s marketing! • Rate of change is much, much faster! • Software evolution is (arguably) Lamarckian! Michael W. Godfrey

36. Biological vs. software evolution • One important observation: • Not all of the effects of software change are planned or foreseen. • Software reacts with its environment (development, deployment, political) and forms a complex feedback system that influences the future evolution of the software [Lehman] • Software systems exhibit interesting emergent phenomena Michael W. Godfrey

37. Summary:Biological vs. software evolution • If you take away nothing else from this, at least remember this: • “We shape our tools, then our tools shape us.” [McLuhan] • We can plan and execute changes … • … but we cannot understand in advance all of the effects of the changes, both on the environment and feeding back into the development of the software system. • Understanding how software evolves requires studying both the planned and unplanned phenomena of software change. Michael W. Godfrey

38. Further reading • Evolutionary biology: • The talk.origins archives • http://www.talkorigins.org • The Selfish Gene, Richard Dawkins • “Other” evolution: • The Meme Machine, Susan Blackmore • The Nature of Economies, Jane Jacobs Michael W. Godfrey