agile software engineering

1. Agile Software Engineering A Presentation by: Austin Riddle

2. Overview Agile Methodology Concepts and Ideology Origins Applicable Domains Case Studies Process Example Adoption Detractors Agile vs. Plan Driven Processes Future Headings

3. What is Agile Software Development? Easily moved, light, nimble, active software processes Fitting the process to the project Avoidance of things that waste time Agile SE -> Agile SD How many have heard of Agile Software Development? What does the word Agile mean? Anyone? Latin word agilis, which means “easily moved, light, nimble, active”. 3. Agile Software development – defined as easily moved, light, nimble, and active software processes. (Agile processes are not unique to software development. They appeared in mainstream business literature in 1991 in the form of Agile manufacturing.) 4. Fitting the process… Not project to the process. Battlefield commanders plan extensively when going to war -> but realize that their plans are just the beginning. Creating and responding to change are very important. Success is defined by accomplishing the mission (defeating the enemy) not conforming to the plan. Can you image a commander saying, “We lost the battle but by golly we were successful because we followed our plan to the letter.” Or “If we just plan this battle long and hard enough, and put repeatable processes in place, we can eliminate change early and not have to deal with it later.” Pretty unreasonable. Agile SE -> Agile SD How many have heard of Agile Software Development? What does the word Agile mean? Anyone? Latin word agilis, which means “easily moved, light, nimble, active”. 3. Agile Software development – defined as easily moved, light, nimble, and active software processes. (Agile processes are not unique to software development. They appeared in mainstream business literature in 1991 in the form of Agile manufacturing.) 4. Fitting the process… Not project to the process. Battlefield commanders plan extensively when going to war -> but realize that their plans are just the beginning. Creating and responding to change are very important. Success is defined by accomplishing the mission (defeating the enemy) not conforming to the plan. Can you image a commander saying, “We lost the battle but by golly we were successful because we followed our plan to the letter.” Or “If we just plan this battle long and hard enough, and put repeatable processes in place, we can eliminate change early and not have to deal with it later.” Pretty unreasonable.

4. Agile Software Development Ecosystem “Chaordic” Perspective Collaborative Values and Principles Streamlined Methodology Chaordic Perspective – due to the unpredictability of the environment. Goals are achievable, but project details are often unpredictable, the goal of repeatable processes is unattainable. Collaborative Values - capitalization on each individual's and each team's unique strengths. The process is adapted to the people, who indeed are part of the project. Process -> adapts -> project. Project understanding comes more from face-to-face interaction than from documentation. Agilists do not believe that a reliance on heavy processes makes up for lack of skill, talent, and knowledge. Streamlined Methodology – a balance of flexibility and structure in the process. Chaordic Perspective – due to the unpredictability of the environment. Goals are achievable, but project details are often unpredictable, the goal of repeatable processes is unattainable. Collaborative Values - capitalization on each individual's and each team's unique strengths. The process is adapted to the people, who indeed are part of the project. Process -> adapts -> project. Project understanding comes more from face-to-face interaction than from documentation. Agilists do not believe that a reliance on heavy processes makes up for lack of skill, talent, and knowledge. Streamlined Methodology – a balance of flexibility and structure in the process.

5. Agile Software Development Ideals Individuals and interactions over process and tools Working Software over comprehensive documentation Rework vs. Reuse Customer collaboration over contract negotiation Solutions vs. Products Responding to change over following a plan Idealisms Over instead of versus is because they are all good ideals Rework vs. Reuse – everything should be production quality, but reworkable for flexibility. Don’t be afraid of partial solutions. Military Adoption of the 80% solution. Responding to Change – in line with Lucy Suchman’s Plans and Situated Actions, with the trukese and european navigators.Idealisms Over instead of versus is because they are all good ideals Rework vs. Reuse – everything should be production quality, but reworkable for flexibility. Don’t be afraid of partial solutions. Military Adoption of the 80% solution. Responding to Change – in line with Lucy Suchman’s Plans and Situated Actions, with the trukese and european navigators.

6. Substantive Definition “the continual readiness of an entity to rapidly or inherently, proactively or reactively, embrace change, through high quality, simplistic, economical components and relationships with its environment”.

7. Origins and Foundations How did Agile development come about? Welcome to the Jungle. Diagram -> Started in the 80s and evolved from a combination of practices such as Crystal Methods, XP, ASD, Scrum, and PP. Crystal Methods – involves the use of customizable development methods that are color coded based on the heaviness of a project with regard to size and complexity. XP – Extreme Programming – is a collection of well-known software practices that are applied on an individual bases. (short iterations, small releases, rapid feedback. Adaptive Software Development – hallmarked by adaptive processes including incremental and iterative development with constant prototyping. Scrum – management of the software process in a volatile environment. It has an Empirical approach that leaves room for developers to choose the techniques and methods for implementation. Requires frequent management oversight. PP – Pragmatic programming – the use of a set of 70 programming best practices for incremental and iterative development, rigorous testing, and user-centered design. Test-Driven methods of development are building blocks of agile processes. 4. As shown by the diagram, the evolution of Agile Software Development is complex. And as we’ll talk about later, because of its roots, it has some issues. 5. In any case, what’s the motive behind the maddness? Developers sick of late deliveries and lost opportunities. Sometimes making the customer follow your software development processes is like banging your head into a brick wall. Is essence it is all about flexibility and leanness. -> for the Development side and the Customer side. How did Agile development come about? Welcome to the Jungle. Diagram -> Started in the 80s and evolved from a combination of practices such as Crystal Methods, XP, ASD, Scrum, and PP. Crystal Methods – involves the use of customizable development methods that are color coded based on the heaviness of a project with regard to size and complexity. XP – Extreme Programming – is a collection of well-known software practices that are applied on an individual bases. (short iterations, small releases, rapid feedback. Adaptive Software Development – hallmarked by adaptive processes including incremental and iterative development with constant prototyping. Scrum – management of the software process in a volatile environment. It has an Empirical approach that leaves room for developers to choose the techniques and methods for implementation. Requires frequent management oversight. PP – Pragmatic programming – the use of a set of 70 programming best practices for incremental and iterative development, rigorous testing, and user-centered design. Test-Driven methods of development are building blocks of agile processes. 4. As shown by the diagram, the evolution of Agile Software Development is complex. And as we’ll talk about later, because of its roots, it has some issues. 5. In any case, what’s the motive behind the maddness? Developers sick of late deliveries and lost opportunities. Sometimes making the customer follow your software development processes is like banging your head into a brick wall. Is essence it is all about flexibility and leanness. -> for the Development side and the Customer side.

8. Applicable Domains Multi-Sized Corporations Multi-team environments using overlapping cross-team communities Government Contracts Most effective for extreme projects Projects that do not work well in rigorous plan-driven processes Extreme Projects – Not extreme programming projects but rather projects that use leading or bleeding edge tech, Involve erratic requirements changes, and deliver quickly. Multi-Sized Corps – Nokia story. Government Contracts – Earned Value Management Even Intel has used Agile Methodology for Embedded Development.Extreme Projects – Not extreme programming projects but rather projects that use leading or bleeding edge tech, Involve erratic requirements changes, and deliver quickly. Multi-Sized Corps – Nokia story. Government Contracts – Earned Value Management Even Intel has used Agile Methodology for Embedded Development.

9. Case Studies Daimler-Chrysler Embedded Software CaribouLake.com Database Nuclear Control Systems Manufacturer

10. Daimler-Chrysler Case Study Reasons Rising workload Frequent late changes Difficult time and quality constraints Late delivery Embedded Software for buses and coaches. Time Constraints – hours or days, not weeks or months 50% rejection rate - Documentation flaws and incomprehensible function code 8% rejection rate - Functional unit fails test cases and integration errors. Current SE practices Inadequate. Embedded Software for buses and coaches. Time Constraints – hours or days, not weeks or months 50% rejection rate - Documentation flaws and incomprehensible function code 8% rejection rate - Functional unit fails test cases and integration errors. Current SE practices Inadequate.

11. Daimler-Chrysler Case Study Results Overall Solution: Combination of classical and agile practices Test First practices Unit Testing Challenges No external support Test cases were difficult to scope Developer transitions implement->document->test test->implement->document Effects Substantial increase in flexibility, quality and timeliness Greater acceptance of agile practices Solution: Classical - systematic documentation and measurement. Agile – Test First Process Test First Practices gave way to more process improvements such as: quicker simulation setups faster module tests Quality assurance effort spread out Test case scope – too detailed, not enough pruning of inputs. Concerns over business risk of a whole adoption. Solution: Classical - systematic documentation and measurement. Agile – Test First Process Test First Practices gave way to more process improvements such as: quicker simulation setups faster module tests Quality assurance effort spread out Test case scope – too detailed, not enough pruning of inputs. Concerns over business risk of a whole adoption.

12. CaribouLake.com Case Study Problem: Database design during short development iterations Reasons Traditional up-front database development impractical Late requirements changes Database changes more costly than application changes Again, agile practices treat change the same anywhere in the development process.Again, agile practices treat change the same anywhere in the development process.

13. CaribouLake.com Case Study Results Solution: Collaborative schema evolution Formalized Refactoring Test Suites Database coding standards Effects Distributed knowledge of database design Faster development Leaner database schema Collaborative schema – Developers made changes to database to make application work, then changes were approved by the team and modified/abstracted to fit other developers needs. Rework vs. Reuse Leaner database schema – obsolete areas of schema were easily phased out. No lingering question “What are these fields for?”Collaborative schema – Developers made changes to database to make application work, then changes were approved by the team and modified/abstracted to fit other developers needs. Rework vs. Reuse Leaner database schema – obsolete areas of schema were easily phased out. No lingering question “What are these fields for?”

14. Nuclear Control Systems Manufacturer Case Study Problem: Difficulty replacing old control software Heavyweight waterfall process too inflexible Solution: Independent research and wholesale adoption of agile processes No upfront training of team Absence of common working area, workstation configurations and an integration and build environment Results Five development iterations produced < 20% projected business value Agile process approach “canned” Team was demoralized Moral of the story is that, just like anything, Agile processes are not just a hack job that require no preparation.Moral of the story is that, just like anything, Agile processes are not just a hack job that require no preparation.

15. Example Process Comparison

16. Adoption Detractors Inconsistent and diverse definitions Lack of theoretical grounding Different way of thinking Role changes Situational customization Solid people skills required Short iterations inhibit long-term perspective Risks Harder to manage feature creep and customer expectations Difficult to quantify cost, time, quality. Why not more adoption (in SE? in corporate? In universities?) Inconsistent and diverse defs – need more quality methodology not just quantity. Lack of Theoretical Grounding – current concept of agility based on experience, not on underlying concepts like flexibility and leanness. Agile methods like SCRUM and XP are derived from subjective practical experience and not reliable systematic research. But there is hope… Conboy et. Al. and their framework of agility for software development (based on underlying concepts); Role changes – management role less prominent and active, more like a coach Situational customization – no out of the box operation. Takes time to customize a process and get good at it. Difficult to quantify – Agile practices are more philosophy based not activity based. Risks – Making the customer understand the tradeoffs for following them down a rabbit hole can be difficult. How do you budget a project if you can’t get the all/most requirements upfront?Why not more adoption (in SE? in corporate? In universities?) Inconsistent and diverse defs – need more quality methodology not just quantity. Lack of Theoretical Grounding – current concept of agility based on experience, not on underlying concepts like flexibility and leanness. Agile methods like SCRUM and XP are derived from subjective practical experience and not reliable systematic research. But there is hope… Conboy et. Al. and their framework of agility for software development (based on underlying concepts); Role changes – management role less prominent and active, more like a coach Situational customization – no out of the box operation. Takes time to customize a process and get good at it. Difficult to quantify – Agile practices are more philosophy based not activity based. Risks – Making the customer understand the tradeoffs for following them down a rabbit hole can be difficult. How do you budget a project if you can’t get the all/most requirements upfront?

17. Agile vs. Plan Driven Processes Small products and teams; scalability limited Untested on safety-critical products Good for dynamic, but expensive for stable environments. Require experienced Agile personnel throughout Personnel thrive on freedom and chaos Large products and teams; hard to scale down Handles highly critical products; hard to scale down Good for stable, but expensive for dynamic environments Require experienced personnel only at start if stable environment Personnel thrive on structure and order Good for dynamic.. - Simple design and refactoring principles Boehm and Turner propose these risk factors based on observation, not from experience. Good for dynamic.. - Simple design and refactoring principles Boehm and Turner propose these risk factors based on observation, not from experience.

18. Use of Agile and Plan Driven Processes Each have appropriate roles in software development Most use Agile-Plan Driven Hybrid When should each be used? Boehm and Turner Risk assessment model (5 factors) Boehm and Turner Risk assessment model (5 factors)

19. My Agile Synopsis No such thing as Agile hybrid. Agility is about flexibility and leanness. Agility != Lack of Structure Change of process control from Top-Down to Bottom-Up May be somewhat conflicting with Boehm and Turner. Agile processes at worst can be plan driven processes if your project demands such a process. If you can’t revert back to plan driven processes you are not using Agile processes. Again Agile Software Processes are simply about flexibility and leanness. It is not about a lack of structure, it is about adaptability in the presence and kind of structure for a process. Developers have been pretty Agile for a while, it is just that now they have more influence over the extent of their Agile practices. House metaphor Architect Builder All use agile processes.May be somewhat conflicting with Boehm and Turner. Agile processes at worst can be plan driven processes if your project demands such a process. If you can’t revert back to plan driven processes you are not using Agile processes. Again Agile Software Processes are simply about flexibility and leanness. It is not about a lack of structure, it is about adaptability in the presence and kind of structure for a process. Developers have been pretty Agile for a while, it is just that now they have more influence over the extent of their Agile practices. House metaphor Architect Builder All use agile processes.

20. An Eye on the Future Volatility and uncertainty in project environments Increased demand for more adaptive processes More theoretical research in Agile Methods Combined methods emerging (Agile RUP) Expanded fields of use (anywhere design is present) As a kid learning to draw, I would often indulgently render small parts of a drawing in rich detail, and pay less attention to how these parts related to each other. Lacking an overall spatial plan, I often ended up with gross inaccuracies that ruined the drawing. My instructional books recommended starting with a light sketch to plan the composition, and then to gradually commit to areas of light and shade over the entire surface to evenly develop the composition. Maintaining such a balanced overview of the drawing would reveal planning flaws sooner and allow their correction. Initially I resisted drawing these lines that would later need to be erased, but the advice eventually worked; results became more predictable and the eraser became my friend. As a kid learning to draw, I would often indulgently render small parts of a drawing in rich detail, and pay less attention to how these parts related to each other. Lacking an overall spatial plan, I often ended up with gross inaccuracies that ruined the drawing. My instructional books recommended starting with a light sketch to plan the composition, and then to gradually commit to areas of light and shade over the entire surface to evenly develop the composition. Maintaining such a balanced overview of the drawing would reveal planning flaws sooner and allow their correction. Initially I resisted drawing these lines that would later need to be erased, but the advice eventually worked; results became more predictable and the eraser became my friend.

21. References Abrahamsson, P. et al. New Directions on Agile Methods: A Comparative Analysis. In Proceedings of the 25th International Conference on Software Engineering. IEEE 244-256, Portland, Oregon, May 2003. May be found at: http://csdl.computer.org/comp/proceedings/icse/2003/1877/00/18770244abs.htm Alleman, G. B. and Henderson, M. Making Agile Development Work in a Government Contracting Environment. In Proceedings of the Agile Development Conference (ADC’03). IEEE 114-120, Salt Lake City, Utah, June 2003. May be found at: http://csdl.computer.org/comp/proceedings/adc/2003/2013/00/20130114abs.htm Armitage, J. Are Agile Methods Good for Design? Interactions. ACM 14-23. 11,1 January 2004. May be found at: http://portal.acm.org/citation.cfm?id=962342.962352 Beck, K. et al. Manifesto for Agile Software Development. Last Access: 02-7-2005. May be found at: http://www.agilemanifesto.org/ Boehm, B. and Turner, R. Using Risk to Balance Agile and Plan-Driven Methods. IEEE Computer. IEEE 57-66, 36,6, June 2003. May be found at: http://csdl.computer.org/comp/mags/co/2003/06/r6057abs.htm Conboy, K. and Fitzgerald, B. Toward a Conceptual Framework of Agile Methods: A Study of Agility in Different Disciplines. In Proceedings of the 2004 ACM Workshop on Interdisciplinary Software Engineering Research. ACM 37-44, Newport Beach, CA. November 2004. May be found at: http://portal.acm.org/citation.cfm?id=1029997.1030005 Derbier, G. Agile Development in the Old Economy. In Proceedings of the Agile Development Conference (ADC’03). IEEE 125-132, Salt Lake City, tah, June 2003. May be found at: http://csdl.computer.org/comp/proceedings/adc/2003/2013/00/20130125abs.htm Green, B. Agile Methods Applied to Embedded Firmware Development. In Proceedings of the Agile Development Conference (ADC’04). IEEE 71-77, Salt Lake City, Utah, June 2004. May be found at: http://csdl.computer.org/comp/proceedings/adc/2004/2248/00/22480071abs.htm Harriman, A., Hodgetts, P. and Leo, M. Emergent Database Design: Liberating Database Development with Agile Practices. In Proceedings of the Agile Development Conference (ADC’04). IEEE 100-105, Salt Lake City, Utah, June 2004. May be found at: http://csdl.computer.org/comp/proceedings/adc/2004/2248/00/22480100abs.htm

22. References (Cont.) Highsmith, J. What is Agile Software Development? CrossTalk: The Journal of Defense Software Engineering. Oct. 2002. May be found at: http://www.stsc.hill.af.mil/crosstalk/2002/10/highsmith.html Hirsch, M. Making RUP Agile. In Practitioners Reports of the Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA 2002). ACM 1-28. Seattle, Washington, November 2002. May be found at: http://portal.acm.org/citation.cfm?id=604251.604254 Hodgetts, P. Refactoring the Development Process: Experiences with the Incremental Adoption of Agile Practices. In Proceedings of the Agile Development Conference (ADC’04). IEEE 106-113, Salt Lake City, Utah, June 2004. May be found at: http://csdl.computer.org/comp/proceedings/adc/2004/2248/00/22480106abs.htm Huo, M. et. al. Software Quality and Agile Methods. In Proceedings of the 28th Annual International Computer Software and Applications Conference (COMPSAC'04). IEEE 520-525, Hong Kong. September 2004. May be found at: http://csdl.computer.org/comp/proceedings/compsac/2004/2209/01/220910520abs.htm Kähkönen, T. Agile Methods for Large Organizations – Building Communities of Practice. In Proceedings of the Agile Development Conference (ADC’04). IEEE 2-11, Salt Lake City, Utah, June 2004. May be found at: http://csdl.computer.org/comp/proceedings/adc/2004/2248/00/22480002abs.htm Manhart, P. and Schneider, K. Breaking the Ice for Agile Development of Embedded Software: An Industry Experience Report. In Proceedings of the 26th International Conference on Software Engineering (ICSE’04). IEEE 378-386, Edinburgh, Scotland, UK. May 2004. May be found at: http://csdl.computer.org/comp/proceedings/icse/2004/2163/00/21630378abs.htm Schneider, J. and Johnston, L. eXtreme Programming at Universities – An Educational Perspective. In Proceedings of the 25th International Conference on Software Engineering. IEEE 594-599, Portland, Oregon, May 2003. May be found at: http://csdl.computer.org/comp/proceedings/icse/2003/1877/00/18770594abs.htm

23. Questions?