Working in Chaos & Complexity for Success May 11-14, 2009

1. Agile Development Methodsin Current Environments Working in Chaos & Complexity for SuccessMay 11-14, 2009 Richard M. Wallace

2. Outline • Agile does work • Proven in commercial industry, very late to government procurement • Software is not manufactured • Government sees software development as a manufacturing process • Procurement treats software as a "mature technology" item • Chaos is always present – get used to it • In all development chaos and complexities arise preventing wanted functionality • Success is possible with simple steps • Agile development, a framework for traversing organizational social complexity, and efficient processes equals delivering wanted functionality

3. Agile Does Work Proven in commercial industry,very late to government procurement

4. Agile Accepted in Commercial Industry The 3rd Annual "State of Agile Development" survey was conducted and sponsored by VersionOne. This global survey ran in June and July of 2008 and received 2,319 completed surveys with 3,061 total respondents from 80 countries.

5. Percentage of Successful Agile Projects1 1: 3rd Annual "State of Agile Development" survey

6. Agile is Fully Adopted in Commercial Industry Agile has crossed the chasm and is now part of mainstream software development. Mary Poppendieck1 used Geoffrey Moore’s classic product adoption diagram to illustrate where agile practices are in the general software development adoption curve. 1: http://www.poppendieck.com/ Crossing the Chasm, Geoffrey A. Moore, Collins Business; Revised edition (August 20, 2002) ISBN-10: 0060517123 The chasm is between the early adopters of the product (the technology enthusiasts and visionaries) and the early majority (the pragmatists).

7. Accepted Agile Methodologies • Multiple development methodologies: • Agile Unified Process – AUP (simplified RUP) • eXtreme Programming – XP • Scrum – Very shorttime frame • Methodologies not accepted as “Agile” • Waterfall • Incremental • Spiral • Evolutionary (Deming) [1] What Northrop Grumman normally uses

8. Software Is Not Manufactured Government sees software development as a manufacturing process Treats software as a "mature technology"

9. DoD Views Software As A Manufactured Item What Engineering Has in Common With Manufacturing and Why It Matters -- Dr. Alistair Cockburn (4/2007) Production Planning for a Software Product Line -- Gary J. Chastek, Ph.D., SEI Linda M. Northrop, SEI John D. McGregor, Ph.D., Clemson (1/2009) Advancing Producibility for Software-Intensive Systems -- Grady H. Campbell, Jr., SEI (12/2008) DoD Instruction DODI 5000.2 views software as a "part" like hardware

10. 5 Reasons Why the ManufacturingMetaphor Doesn’t Work • Software development is much less predictable than manufacturing a product • Software is not mass-produced on the same scale as a physical product • Not all software faults result in failures • Software doesn’t wear out • Software is not bound by physics "The creation of software is an intellectual human endeavor. Creating good software relies on the personalities and the intellects of the members of the teams that create it. When applied to a different team of developers a process that delivers great software for one team of developers may fail to deliver anything at all for another team." -- The Practical Guide to S/W Arch.

11. What Software is Not "Software development is not a manufacturing process ... it is largely an intellectual and sociological activity" -- Martyn Ould (10/1999) Very Significant Paradigm Shift "These manufacturing focused models tell us nothing about the typical back-and-forth activities that lead to creating a final product. In particular, creation usually involves trying a little of this-or-that, developing and evaluating prototypes, assessing the feasibility of requirements, contrasting several design, learning from failure, and eventually settling on a satisfactory solution to the problem at hand." -- Shari Lawrence Pfleeger, Joanne M. Atlee (7/2005)

12. Chaos is Always Present – Get Used To It In all development chaos and complexities arise preventing wanted functionality

13. Map of Complexity Science http://en.wikipedia.org/wiki/Sociology_and_complexity_science

14. Development is Chaotic • Development, by its very nature, has historically been chaotic • System Engineering treats this chaotic phase as if it were a production phase which leads to overruns as shown by the historical record • This is a root problem with “agile” methods • This has been noted as Fr-”agile” in the literature • Agile Software Development is done using a defined process framework • Some Agile Methodologies, like Scrum, have been very prescriptive and lay down rigid practices as “rules of the game”

15. Synchronization • Physical and animate • Phase transitions Complexity Interrelationships1 • Small Worlds • Small avg # steps • High local clustering • Hubs • Resistance to failure • Security weak points • Tipping points Scale-Free Networks form via growth and preferential attachment Indicate Complex Adaptive Systems Lead to Power Law Distributions Autopoiesis (self-organized criticality, self-persistence) Complexity Edge of Chaos = Life Chaos (“Butterfly effect”) Fractals Strange Attractors Small changes Big Effects Movement from Order through Oscillation to chaos Entropy as a measure

16. Chaos & Complexities Regardless of method, chaos and complexities arise in many forms: • Changing Staff • Changing Budget • Requirements creep • Product morphing (outside of budget) • Intended – Eventual end user directs or wants the change • Unintended – Difference in vision between development and end user

17. Agile Manifesto – Some Chaos Applied Chaos Factor #1 EVMS • Customer satisfaction by rapid, continuous delivery of useful software • Working software is delivered frequently (weeks rather than months) • Working software is the principal measure of progress • Even late changes in requirements are welcomed • Close, daily cooperation between business people and developers • Face-to-face conversation is the best form of communication • Projects are built around motivated individuals, who should be trusted • Continuous attention to technical excellence and good design • Simplicity • Self-organizing teams • Regular adaptation to changing circumstances Chaos Factor #2 Slow Contract Changes Chaos Factor #3 Culture Gulf & “Arms Length” Relationships [1] [1]Short-term benefits of concurrent engineering

18. Agile Limitations… or not [1] • Large scale development efforts (>20 developers), though scaling strategies [2] and evidence to the contrary [3] have been described • Distributed development efforts (non-co-located teams). Strategies have been described in Bridging the Distance[4] and Using an Agile Software Process with Offshore Development[5] • Mission- and life-critical efforts • Chain of Command company cultures force Agile teams to fail • Development teams need extremely high levels of creativity, innovation, commitment and discipline and cannot bloom in a strictly defined process environment due to the high degree of interaction, informal face to face communication with stakeholders, ability to change directions midway, and learning along the way what is needed to build the best possible solution

19. Cynefin + Agile • Cynefin (Cun-ev-in): Welsh • Has no direct equivalent in English. It is translated as familiar habitat • Thus, in general, if a community is not physically, temporally and spiritually rooted, it is alienated from its environment and will focus on survival rather than creativity and collaboration • The use of methods such as eXtreme Programming and Scrum has helped many software projects to success, although it’s not always clear why and how it is accomplished

20. Using the Cynefin framework to move between domains requires a shift to a different model of understanding and interpretation as well as a different leadership style. Understanding differences among the different movements in the framework increases the sophistication of the response of a decision-making group to rapid change. This understanding allows Agile development teams to anticipate the pattern of social complexity, creating change or chaos [1] Cynefin Framework Use Four named domains and a fifth in grey that is “disorder”

21. Chaos Theory and Cynefin • A system is classified as chaotic if it has the following properties:[1] • It must be sensitive to initial conditions, • It must be topologically mixing, and • its periodic orbits must be dense The Cynefin[2]framework: This framework originated in the practice of knowledge management as a means of distinguishing between formal and informal communities, and as a means of talking about the interaction of both with structured processes and uncertain conditions.

22. Success is Possible with Simple Steps   • Agile development • A framework for traversingorganizational social complexity • Efficient processes Equals delivering wanted functionality

23. Agile + Chaos • An understanding of the roots of agility, though, is a requirement for adopting and adapting agile processes in organizations • New research in social complexity theory provides an explanation for when and how agile methods work • Using techniques from this relatively new field, developers and managers can more easily adapt agile methods to the idiosyncrasies of specific organizations and projects, are more aware of weak signals that can lead to problems, and can design interventions to correct them [1]

24. The Human Element in Agile • The Cynefin framework is based on three ontological states juxtapositioned • Order (Known & Knowable) • Complexity • Chaos • Success is not getting lost on your way from CONOPS to product instantiation • Relaxing the assumption of order and follow the pattern • Relaxing the assumption of rational choice from rigid formalism and use context and perspective • Relaxing the assumption of intentional actions allowing for accident of interaction

25. Movement at the known-chaos boundary (1)Asymmetric collapse – disastrous movement from known to chaotic (2) Imposition – forceful movement from chaotic to known (Draconian) Movement at the known-knowable boundary (3) Incremental improvement – engine of tech growth, but can be pathological Movement at the knowable-complex boundary (4) Exploration – selective movement to complexity with trust (5) Just-in-time transfer – choice of stable patterns in complex space Movement at the complex-chaotic boundary (6) Swarming – mass movement to an attractor, some may not survive (7) Divergence-convergence – Able to handle disruption Navigation on the Cynefin Framework

26. Visiting chaos (8) Entrainment breaking – Create and validate new sources “breaking out of the groove” (9) Liberation – casting idea “seeds” for growth and quick exploitation of “seeds” that grow (10) Immunization – Inure people to the devastation of chaos to be better prepared for it in the future Navigation on the Cynefin Framework Cont.

27. Assuring What Is Wanted By The Customer Is Delivered • Know that software development is highly subject to • Chaos and Complexity • Changing requirements or "desire-ments" • Human social structure changes • Assure delivering wanted functionality by use of Agile development as • Agile development is the natural form of human grouping • Agile has demonstrated world-wide acceptance and proof-of-performance • Scrap the "manufacturing" mentality • Understanding that the "manufacturing" mentality in procuring software is a failure and that software is an intellectual and sociological activity allows better delivery of software product • Understanding that what appears to be a process out of control is, in fact, a process in control if complexity and chaos patterns are known • Embrace the fact that social structures are dynamic and using the Cynefin framework allows informed navigation between styles for successful creation of software products

28. Schools of Thought for Chaos/Complexity According to De Wolf and Holvoet (2005), there are actually four central schools of research that influence the way complex behavior of complex systems is studied: • Complex adaptive systems theory (Santa Fe Institute (SFI) 2006): Complex systems are seen as having similarities that can be studied and exploited in order to ease finding underlying principles of a unified complexity theory (Holland, 1996). The SFI often call complex systems complex adaptive systems (CAS) • Nonlinear dynamical systems theory and chaos theory: Promulgates the central concept of attractors • Synergetics: Study of emergence in complex systems with the idea of an order parameter that influences which macro-level coherent phenomena a system exhibits (Haken, 1981) • Far-from-equilibrium thermodynamics: Based on nonlinear dynamics and far-from-equilibrium thermodynamics. Disciplines include catastrophe theory, chaos theory, and complexity theory. These theories push beyond some of the limitations of classical physics, and explore classes of phenomena outside the traditional linear realm

29. Hope exists Truth and Confidence: Some of the Realities of Software Project Estimation -- Phillip G. Armour (4/2008)