SEPG Conference Amsterdam 2006 Developing Metrics for agile projects compatible with CMMI

1. SEPG Conference Amsterdam 2006 Developing Metrics for agile projects compatible with CMMI Graham Collins, UCL graham.collins@ucl.ac.uk Research supported by Deutsche Bank

2. Introduction • UCL’s research and projects • The problems and requests • EV (Earned Value) • Agile practices • Combining EV and agile is it possible? • Developing suitable metrics compatible with CMMI • What works, and what reduces the overheads • Is this approach likely to lead to CMMI Level 5? • Key changes to projects

3. Background - Requests • We would like to use earned value • ‘We would like to predict the outcome of project end date, cost and value’ • As a project manager I cannot be there all the time • What is the best way to measure progress with earned value? • ‘As developers we would like to experiment with some agile approaches’ • Your team will be working on other projects as well • We are hoping to achieve metrics suitable for CMMI level 3 and higher…

4. The Death March Project Style Quadrant The Death March Project Style Quadrant high Mission Impossible Kamikaze Happiness Suicide Ugly low low high Chance of success Edward Yourdon, Death March:The complete Software Developer’s guide to surviving ‘Mission Impossible’ projects, 1997 Prentice Hall

5. Earned Value - Definition ‘The value of the useful work done at any given point in a project. The value of completed work expressed in terms of the budget assigned to that work. A measure of project progress. Note: The budget may be expressed in cost or labour hours’ APM (Association of Project Management) BoK 2006

6. Earned Value – Graphical Representation Planned Actual Cost or value Earned Value Time Planned (BCWS = Budgeted Cost of Work Scheduled) Actual (ACWP = Actual Cost of Work Performed) Earned Value (BCWP = Budgeted Cost of Work Performed) CPI = Cost Performance Index = BCWP/ACWP (or Earned/Actual) SPI = Schedule Performance Index = BCWP/BCWS (or Earned/Planned) note this is SPIc i.e. cost.

7. Use of Variance spic is used in this situation

8. Earned Value compared to Agile Process Planning Earned Value Agile Development

9. CMMI- Process Areas ‘Project planning is a necessity for success, Yet it still ranks on most surveys within the top three or four problem areas leading to failure.’ Tony Ciorra, Planner’s Progress, Project, APM May 2006

10. CMMI Comparative Advantages Continuous Representation Staged Representation

11. Agile Manifesto Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more Several agile projects have achieved CMMI level 3, example David Anderson, Stretching Agile to fit CMMI Level 3, Agile Conference 2005

12. The Agile Principles www.agilealliance.com

13. Iterative Development (Bittner-Spence) • Agree with the team the objectives for the iteration, including evaluation criteria, timescales, and constraints • Agree on a plan for how the team will achieve the objectives • Execute the plan • Assess the achievements of the team against the initial set of objectives and evaluation criteria • Assess the impact of the iteration’s results on the project as a whole • Start the next iteration. What is iterative development? Part 3: The management perspective 15 May 2005 www-128.ibm.com/developerworks/rational/library/may05/bittner-spence/index.html

14. Fundamental shift in measurement 100% Progress ( % complete measured in scenarios 0% Iteration 1 2 3 4 coded tested Tested & Passed

15. Developer Perspective • Developers are less interested in the business value, benefits realization and return on investment • They work on a small number of requirements or change requests from their list of outstanding work • They anticipate a decreasing number of requirements and change requests as the product is developed • Outstanding requirements and change requests is termed the product backlog • The developer will therefore be aware of progress via work completed, product backlog and new work allocated.

16. User Satisfaction driving Development Release User satisfaction Release planning Iteration User satisfaction Iteration planning Development Increment Iteration Planning (Goal identification, story selection, task estimation, team commitment)

17. Project teams need to adopt some attributes What is the purpose? What holds it together? To develop members’ capabilities; to build and exchange knowledge Passion, commitment, and identification with the group’s expertise Community of practice To accomplish a specified task The project’s milestones and goals Project team Adapted from: Communities of Practice: The organizational Frontier, Etienne C. Wenger and William M. Snyder, Harvard Business Review p139-145 Jan-Feb 2000

18. Rate of work - velocity Velocity, gives an indication of the average rate of work and also a comparison of planned against delivered, each iteration

19. Individuals and Moving Range (XmR) Charts

20. Control Limits for XmR Charts k sequential measurements provide k-1 =r (two-point) moving range values ith moving range = mRi = │Xi+1 – Xi │where integer i is 1 ≤ i ≤ k – 1 __ i=r Individuals average moving range =mR = 1∑ mRi r i=1 _ __ _ __ Upper Natural Process Limit =UNPLx= X + 3mR = X + 2.660mR d2 _ i=k Centerline = CLx = X = 1∑ Xi (average of individual values) k i=1 _ __ _ __ Lower Natural Process Limit =LNPLx= X - 3mR = X - 2.660mR d2 ___ Centerline or average moving range =UCLR = mR __ __ Upper Control Limit for moving range =UCLR= D4mR = 3.268mR __ Sigma for individual values = sigmax (σ) = mR d2 When n=2 d2 =1.128and D4 =3.268 (from Dispersion and Bias factor tables)

21. ‘Under Control’ Velocity measures of work rate are useful in that estimates of the next iteration can be planned in a rolling process The use of σvariation is supportive in this aim Automated colour coding (Red Amber Green) can be used to show condition requirements

22. ‘Weighted average’

23. ‘Burn-down’

24. With the appropriate metrics we can improve

25. Use of Multipliers Multipliers for estimating velocity based on number of iterations completed from Cohn 2006

26. Charts and Metrics • Velocity and Burn-down • Cumulative acceptance tests Inventory Failing Passing • Cumulative Issue Charts Backlog - Active issues (which show inventory line) Resolved issues Closed issues • Earned Value EV progress charts Performance via cpi and spi Cpi and spi combined with control charts

27. Earned Value • EV can be applied to estimates of agile projects - this is complex if more stories are added as the work progresses • EV may need to be shown to senior managers - who are used to EV figures, or comparison to other projects where EV figures have been tracked • EV estimates can be accurate - story points tend to remain static in an iteration when the process is understood by managers and developers. When additional stories are added, stories with lower business priority level may be dropped to compensate and keep the work load (story points) similar.

28. Business Value More importantly business value (or contribution) should be considered and evaluated Often units of measure such as story points can be valued as 0.5 or 1.0 units The key issue in agile project management is to continually assess with the client the most important work that should be done.

29. Conclusions • EV can be combined with agile reporting • The most useful approach to achieve process improvement is via understanding of process control, the basis for CMMI • The use of acceptance tests is the most useful approach for both methods and is the basis for metrics outlined • EV remains a viable approach where story points work load is maintained • EV is useful for reporting to senior managers and at a programme and project level, but even with major scope changes, re-planning can give some useful insights • The process used in agile project management is critical, that this should be adhered to in the team. It needs to be explained to managers that the team are trying to achieve maximum business value per iteration, and that the final goal and plans evolve during the process • Tracking suitable metrics and understanding variation is the key to better estimation and process improvement – CMMI assessments can help achieve this goal.

30. Key Project Changes • Pair reporting - valuing individuals and team, moving towards self-determining teams • Acceptance testing - working software • Ensuring Business Value – continual prioritisation, estimation and understanding there is a cost to development • Measuring progress over shorter time periods -meeting the needs of process improvement CMMI, velocity tracking in agile methods, and better EV planning Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan