Software Evolution and Agile Development

1. Software Evolution and Agile Development Václav Rajlich Department of Computer Science Wayne State University Detroit, Michigan 48202, U.S.A (C) 2005 Václav Rajlich

2. Confusing situation • New recommendations how to develop software • Rational Unified Process • Extreme Programming • SCRUM • … • They often contradict what has been considered good software engineering • how to make a sense of this? (C) 2005 Václav Rajlich

3. Paradigm • Thomas S. Kuhn • “The Structure of Scientific Revolutions” • “Coherent tradition of scientific research” • includes knowledge, techniques, research agenda, culture of the field… • (more profound meaning than the usage common in software engineering) (C) 2005 Václav Rajlich

4. Paradigm change • Discontinuity in the development of the discipline • Kuhn collected extensive historical data on paradigm change • old paradigm cannot explain compelling new facts • new paradigm is a genuine revolution • paradigm change is often contentious and traumatic • many examples • phlogiston - > oxygen in 1770’s (C) 2005 Václav Rajlich

5. Resistance to paradigm change • Knowledge that was accumulated up to that point may lose its significance • some knowledge may be completely lost • Advantages of the new paradigm in dispute • attempts are made to extend old paradigm to accommodate new facts • band-aid approaches try to fix old paradigm • Final victory of the new paradigm guaranteed by a generation change (C) 2005 Václav Rajlich

6. My claim • Software evolution research • Agile + iterative processes • Represent paradigm change! • Third paradigm change in 50 years of software engineering (C) 2005 Václav Rajlich

7. Paradigm change of 1950’s • Original programmers recruited from the ranks of hardware engineers and mathematicians • used ad-hoc techniques from their former fields • Software separated from the hardware in 1950’s • emerged as a distinct technology • became independent product • Programming became a new discipline • developed a specialized set of skills (C) 2005 Václav Rajlich

8. Paradigm change of 1970’s • Previous techniques did not scale up • Brooks: “Mythical Man-Month” • demands of the new operating system OS/360 taxed the limits of the programmers, project managers, and the resources of the IBM corporation • Paradigm change established discipline of software engineering • dealt with complexity of software • next step goes beyond programming • introduced the waterfall metaphor (C) 2005 Václav Rajlich

9. Waterfall metaphor • Used in construction and manufacturing • collect the requirements • create a design • follow the design during the entire construction • transfer finished product to the user • solve residual problems through maintenance • Intuitively appealing metaphor • good design avoids the expensive late rework • waterfall became the dominant paradigm (C) 2005 Václav Rajlich

10. Problems of waterfall • Requirements volatility (Cusumano and Selby) • 30% or more requirements may change during development • this is the direct result of the team’s learning process and software interoperability • Maintenance phase is not uniform • Lehner, Pigoski • frequency of the changes in long lived systems peaks, then declines (C) 2005 Václav Rajlich

11. Standish group report • In 1995 • 31% of all software projects were cancelled • 53% were “challenged” (completed only with great difficulty, with large cost or time overruns) • only 16% could be called successful. • Obviously, the waterfall metaphor did not solve the problems of software development (C) 2005 Václav Rajlich

12. Anticipation of changes • If changes can be anticipated at design time, they can be controlled by a parameterization, encapsulations, etc. • waterfall model still can be used • Experience confirms: • many changes are not anticipated by the original designers • inability to change software quickly and reliably means that business opportunities are lost • only a band-aid solution (C) 2005 Václav Rajlich

13. Prototyping • Another attempt to extend waterfall • Create a prototype to capture requirements • Problem: volatility continues after prototype has been completed • Another band-aid solution (C) 2005 Václav Rajlich

14. Paradigm change of 2000’s • New metaphor based on self-directed learning • Learner controls the process • Self-directed learning can be planned only to a limited degree • Needs a happy middle between rigid plan and chaos • One semester at a time • Iteration is the analogy of a semester • Ends with a clear feedback (grade) • Continuation is based on that feedback (C) 2005 Václav Rajlich

15. Instances of the new paradigm • New life-cycle models emphasize software evolution • Staged model of software lifecycle • Based on data from long-lived systems • Iterative development • Rational Unified Process • Agile development • SCRUM • Extreme programming (C) 2005 Václav Rajlich

16. Staged model Initial development first running version evolution changes Evolution code decay servicing patches Servicing servicing discontinued Phase-out switch-off Close-down (C) 2005 Václav Rajlich

17. New research agenda • Focus on the evolution • Modifications in the existing software • Incremental change • adds a new functionality or a new properties to the existing software • Refactoring • cleans up software structure without changing functionality • Redocumentation, reengineering, migration, … • Exciting and rich agenda (C) 2005 Václav Rajlich

18. Incremental change (IC) • IC was supposed to happen only rarely under the old paradigm • IC was largely ignored by the researchers and educators • elephant in the living room • The current programmers still must learn IC on their own • No presence in textbooks, curricula • Large backlog of research issues (C) 2005 Václav Rajlich

19. IC minicycle • Initiation • New feature request from users, programmers, managers, marketing, … • Problem report • Design • Implementation • Validation • Release (C) 2005 Václav Rajlich

20. IC design (C) 2005 Václav Rajlich

21. Concept location • Concept location finds where a change is to be made • Change requests are most often formulated in terms of domain concepts • Example: “Correct error that arises when trying to paste a text” • the programmer must find in the code the locations where concept “paste” is located • this is the start of the change (C) 2005 Václav Rajlich

22. Concept location assumptions • The programmer understands domain concepts better than the code • The knowledge of domain concepts is based on program use and is easier to acquire • All domain concepts map onto the fragments of the code • finding that fragment is concept location • “software engineer’s query” (C) 2005 Václav Rajlich

23. Concept location methodologies • Human knowledge • Traceability tools • Dynamic search (execution traces) • Static search • "grep" (pattern matching) • Search of dependency graph • Information retrieval techniques • … (C) 2005 Václav Rajlich

24. Dependency Graph Search • Programmer follows the dependencies among the modules. • Local functionality • All concepts that are actually implemented in the module and are not delegated to others. • Composite functionality • All concepts of the module combined with all its supporting modules. (C) 2005 Václav Rajlich

25. Search Process • The variant of the depth first search • Programmer investigates the module's local functionality. • If the concept is not present in the local functionality, but is present in the composite functionality, programmer continues search with one of the supporting modules. • In case of wrong choice programmer backtracks and begins the search with another supporting module. (C) 2005 Václav Rajlich

26. Search (C) 2005 Václav Rajlich

27. Information Retrieval Technique • Latent Semantic Indexing (LSI) • Preprocessing of the source code and documentation • Executing of queries formulated in natural language • Retrieving and analyzing the results • Results are returned as a list, ranked by relevance (similarity) to the search query (C) 2005 Václav Rajlich

28. Set of concepts is not fixed • New concepts emerge during evolution • set of concepts changes during the lifecycle • difficulty for traceability tools • Origin of the concepts • Domain concepts • familiar to an end user: "credit card payment" • Design concepts • "iterator pattern” • Coding CONCEPTS • "network error while validating credit card” (C) 2005 Václav Rajlich

29. Primitive and latent concepts • Primitive concepts have to be enriched • Point-of-Sale: introduce credit card payment • old code: payment represented as just one number • Latent concepts have to be implemented • Student Registration: introduce prerequisite check • old code assumption: prerequisites are satisfied (C) 2005 Václav Rajlich

30. Impact analysis • Assesses full extent of change • Impact set • Selects different implementation alternatives • Programmers often underestimate impact set (C) 2005 Václav Rajlich

31. Change implementation • Prepare software for the change • Opportunistic refactoring, prefactoring • Actualization • Implementing new functionality • Incorporation • Plug-in the new into the old • Change propagation • Improving software based on new knowledge • Postfactoring: Bad smells -> change (C) 2005 Václav Rajlich

32. Change propagation • Visit components one-by-one • If the visited component is modified, it may no longer fit • secondary changes must be made in interacting (“neighboring”) components • secondary changes may trigger additional changes (“ripple effect”) • Software is inconsistent during propagation (C) 2005 Václav Rajlich

33. Model of change propagation • Let C be a set of classes • For b,c  C, b  c, {b,c} is interaction <b,c> is mark b was visited (changed), c will be visited • Evolving interoperation graph (eig) is a set E of interoperations and marks <b,c>  E implies {b,c}  E. (C) 2005 Václav Rajlich

34. Neighborhood • All interactions, incoming, outgoing marks G(b) = { {b,c} |c {b,c}  E} interactions M(b) = { <c,b> |c <c,b>  E} incoming M(b) = { <b,c> |c <b,c>  E} outgoing E(b) = G(b)  M(b) M(b) (C) 2005 Václav Rajlich

35. Visit • Couple of eigs <E,E'> E' = (E - E(b))  E'(b') E'(b') = G'(b')  M'(b') M'(b') • Marks point away from b’ M'(b') Í { <b',c> | c {b',c} Î E'(b') } (C) 2005 Václav Rajlich

36. Model of propagation sequence • Sequence of eigs E1, E2, … En • Starts and ends with unmarked graphs marked graphs in the middle • <Ei, Ei+1> is either • a visit • or a backtrack: for some k, 0 < k < i, Ei+1 = Ek (C) 2005 Václav Rajlich

37. Finality • Non-final propagation • after visiting b and c, there may be an immediate need to revisit b again: M'(b') = {<b',c>| c {b',c} Î E'(b')} • Final propagation M'(b') Í {<b',c>| c {b',c} Î E'(b')} - {<b',d>| d <d,b> ÎM(b)} (C) 2005 Václav Rajlich

38. Case study • Subject: application framework Drawlets • adds graphical display to a host application • Drawing canvas • lines, free-hand lines, rectangles, rounded rectangles, triangles, pentagons, polygons, ellipses, text boxes, images (C) 2005 Václav Rajlich

39. Drawlets • More than 100 classes, 35 interfaces and 40,000 lines of code • Originally implemented in Smalltalk by Kent Beck, Ward Cunningham • later ported into Java • “perfect API” • Applet SimpleApplet runs in a browser (C) 2005 Václav Rajlich

40. SimpleApplet window (C) 2005 Václav Rajlich

41. Top classes LocatorConnectionHandle AbstractFigure Locator Figure StylePalette SequenceOfFigures SimpleApplet DrawingCanvas CanvasTool SimpleDrawingCanvas ToolBar SelectionTool ConstructionTool ToolPalette LabelTool PrototypeConstructionTool ShapeTool RectangleTool EllipseTool RectangularCreationTool PG_RectImageTool (C) 2005 Václav Rajlich

42. Change Request • Implement an "owner" for each figure • owner put that figure onto the canvas • only owner is allowed to move that figure • each session declares a session owner • this session owner will own all new figures created • no other owner will be allowed to manipulate them • This change will make SimpleAppletmore versatile and useful • support for cooperative work (C) 2005 Václav Rajlich

43. Concept • The concepts relevant to the change • figure owner and session owner • Both concepts are latent • old code assumes that there is just one owner who owns both sessions and figures • look for “similar” concepts in the code (C) 2005 Václav Rajlich

44. Location LocatorConnectionHandle AbstractFigure Locator Figure StylePalette SequenceOfFigures SimpleApplet DrawingCanvas CanvasTool SimpleDrawingCanvas ToolBar SelectionTool ConstructionTool ToolPalette LabelTool PrototypeConstructionTool ShapeTool RectangleTool EllipseTool RectangularCreationTool PG_RectImageTool (C) 2005 Václav Rajlich

45. Actualization LocatorConnectionHandle AbstractFigure Locator Figure StylePalette SequenceOfFigures OwnerIdentity SimpleApplet DrawingCanvas CanvasTool SimpleDrawingCanvas ToolBar SimpleListener SelectionTool ConstructionTool ToolPalette LabelTool PrototypeConstructionTool ShapeTool RectangleTool EllipseTool RectangularCreationTool PG_RectImageTool (C) 2005 Václav Rajlich

46. Incorporation LocatorConnectionHandle AbstractFigure Locator Figure StylePalette SequenceOfFigures OwnerIdentity SimpleApplet DrawingCanvas CanvasTool SimpleDrawingCanvas ToolBar SimpleListener SelectionTool ConstructionTool ToolPalette LabelTool PrototypeConstructionTool ShapeTool RectangleTool EllipseTool RectangularCreationTool PG_RectImageTool (C) 2005 Václav Rajlich

47. Propagation LocatorConnectionHandle AbstractFigure Locator Figure StylePalette SequenceOfFigures OwnerIdentity SimpleApplet DrawingCanvas CanvasTool SimpleDrawingCanvas ToolBar SimpleListener SelectionTool ConstructionTool ToolPalette LabelTool PrototypeConstructionTool ShapeTool RectangleTool EllipseTool RectangularCreationTool PG_RectImageTool (C) 2005 Václav Rajlich

48. Propagation LocatorConnectionHandle AbstractFigure Locator Figure StylePalette SequenceOfFigures OwnerIdentity SimpleApplet DrawingCanvas CanvasTool SimpleDrawingCanvas ToolBar SimpleListener SelectionTool ConstructionTool ToolPalette LabelTool PrototypeConstructionTool ShapeTool RectangleTool EllipseTool RectangularCreationTool PG_RectImageTool (C) 2005 Václav Rajlich

49. Propagation LocatorConnectionHandle AbstractFigure Locator Figure StylePalette SequenceOfFigures OwnerIdentity SimpleApplet DrawingCanvas CanvasTool SimpleDrawingCanvas ToolBar SimpleListener SelectionTool ConstructionTool ToolPalette LabelTool PrototypeConstructionTool ShapeTool RectangleTool EllipseTool RectangularCreationTool PG_RectImageTool (C) 2005 Václav Rajlich

50. Propagation LocatorConnectionHandle AbstractFigure Locator Figure StylePalette SequenceOfFigures OwnerIdentity SimpleApplet DrawingCanvas CanvasTool SimpleDrawingCanvas ToolBar SimpleListener SelectionTool ConstructionTool ToolPalette LabelTool PrototypeConstructionTool ShapeTool RectangleTool EllipseTool RectangularCreationTool PG_RectImageTool (C) 2005 Václav Rajlich