Highlights

1. BSPIN Agile Practices Benchmarking Report October 2012 Highlights

2. Introduction • Agile Practice Benchmarking: BSPIN conducted a study during 2011-2012 to benchmark the Agile practices to provide insights on the effective ways of adopting Agile in small to large organizations. • 12 Organizations presented their Agile experience across 3 Panel discussions • Agile Practice benchmarking survey was conducted in July 2012 • Designing of Questionnaire, Consolidation, Analysis of feedback was done in coordination with Agile coach and Senior Consultant from Improvalue Consulting, Bangalore (www.improvalue.com). • Objectives : • Explore new areas like: • Domains where Agile has been adopted. Agile Practices in use. • Benefits and Barriers encountered by organizations on their Agile journey • Metrics Reported (Project and Product) • Coexistence/Compatibility with other standards (CMMi) and practices (Lean) • Best Practices • Focus is on IMPLEMENTATION aspects of AGILE • Outcome: • Learn from industry experience • Enable informed decisions

3. Agenda • Demographics - Agile Spread • Usage of Agile Practices • Estimation Techniques • Automation, Tools • Agile Metrics • Agile – Coexistence with other Methodologies/Models • Benefits and Barriers • Challenges Faced, Best Practices • Conclusions/Recommendations

4. Demographics • 94 responses from 75 Organizations • More participation from Small and Medium Organizations (compared to other surveys) • Product and Service Organizations covered equally • 98% of Large Organizations and 77% of Small an d Medium Organizations have external quality certifications • More than 50% of the respondents have been using Agile for more than 2 years • Panel discussions – 12 Organizations shared their Agile Experience ( 6 Large and 6 Small-Medium)

5. Support to Agile How Agile adoption is supported? Training Methods used: • This section was answered by 72% of the organizations • As the organization gains more experience on Agile, Agile becomes a part of QMS • Only 6% of the organization indicated “Self Management by teams” with no other support. • This was answered by 90% of respondents • About 40% of the organization had more than one method of training • About 20% of the organization indicated only “In-house” with no other support.

6. Agile Spread • SCRUM is the mostly adopted methodology • Increase in usage of KANBAN - • Many Organizations evolved from manufacturing background where LEAN is adopted • Agile experience of the organization has no major effect on the methodology chosen • Nearly 70% of the organizations are using 1 or 2 methods • Of the 7% using more than 4 methodologies, 3% are small organizations

7. Agile Spread • Others Domains Reported: • OS, Middleware, Kernel • Platform development • Control Systems • Product Integration • Media and Publication, Education • Travel, Energy and Storage • Agile has been used in almost all domains including safety critical domains like Pharma, Avionics and other Embedded Systems. • Telecom domain which is mostly hardware dependent has reported highest percentage of Agile implementation. • More than 75% of the respondents have reported the implementation of ‘Distributed Agile’ • Agile has been used in most of the project types – including Testing and Systems integration ABREVIATIONS (1) BFS – Banking and Financial Services (2) IA – Industrial Automation (3) A,A,A – Avionics, Aviation, Aerospace (4) SI – Systems Integration

8. Usage of Agile Practices 82% have release cycles <= 4 weeks • Core Agile (Scrum) Practices have been well adopted. • Engineering Practices usage (Refactoring, Test Driven Development) needs to be strengthened. • 48% of the respondents use 2 to 3 weeks release cycles.

9. Agile Practices Vs Agile Experience • Core Agile practices are embraced right from the beginning • There is a significant increase in the usage of Open Work Areas (compared to other surveys) • Some of Scrum practices take more time to stabilize • Organizations can have increased rigor early in the Agile adoption for better benefits. • The Engineering practices strengthen over time • Refactoring and TDD need more focus for effective Agile adoption

10. Agile Estimation Techniques • About 53% of the respondents use "User Stories" estimation at proposal/bid stage • About 80% of the respondents are using one or more of the following estimation techniques: • User Stories estimation • WBS • Guesstimation • Organizations are taking more than 2(two) years to move away from Guesstimation to WBS which needs attention. Usage of traditional estimation techniques at proposal/bid stage and User Story Estimation/ WBS at iteration stage seems to be in vogue

11. Extent of Automation • Only 76% of the respondents answered this section • Automation of unit testing is low, inline with low adoption of TDD • Automation of code reviews is low, as evident from low adoption rate of Refactoring • Automation done in later stages of testing and quality compromised- missing out the essence of Agile • The automation level increases with Agile experience of the organization • The size of the organization had no impact on the level of automation Organizations have shared multiple Tools : For Agile Project Management and Engineering.

12. Tools Used Requirements Code Reviews Testing Defect Tracking Project Management • Caliber RM • DOORS • Greenhopper • JIRA • RALLY • Rational Requisite Pro • Red Mine • ScrumWorks • Sharepoint • TFS • VersionOne • CheckStyle • Eclipse plugins • FxCop • IBM AppScan • Rational Software Analyzer • SONAR • StyleCop • TFS • Toad • xTest • xUnit • HP Quality Center • HP Quick Test Professional • JIRA • LoadRunner • MS VS • Selenium • Silk • VSTS • Winrunner • xTest • xUnit • Bugzilla • TeamForge • Clearquest • HP Quality Center • JIRA • Rational ClearQuest • TFS • CollabNet • Green Hopper • HP-Project and Portfolio Management • JIRA • Mingle • Rally • Scrum Works • TFS • Version One • XPlanner Not a complete list. Widely used tools are reported here.

13. Traditional Practices with Agile • 56% of the respondents reported: • Upfront collection and finalization of requirements • Upfront architecture • This practice was predominantly observed in Product companies when compared to Service Providers

14. Management Metrics Other Metrics Reported: Schedule: Schedule Variance, Sprint Stretch Factor Effort: Effort Variance, Scope: Commitment Accuracy, (De)Scoping Index, Requirements Stability, Scope burn down Quality: Product Metrics, Review Efficiency, Defect Turnaround Time, Defect Resolution Productivity, Defect Removal effectiveness Testing: Test Yield, Test case pass percentage • 90% of the respondents have answered this question • The usage of metrics increases with Agile experience to measure effectiveness • Recommend BSPIN to conduct ‘Agile data • Benchmarking’ to obtain industry benchmark

15. Engineering Metrics • Monitoring Product Metrics is fairly good except for the Code related activities • Measuring Code complexity is key for refactoring – needs more focus • Measuring Code coverage is key for quality – needs more attention • There seems to be a steady improvement as the organization gains more experience on Agile • However, Code Complexity and Code Coverage metrics needs more focus

16. Agile - CMMi Coexitence What does the Survey Report? Reasons cited for “Yes”:: Agile does not throw away the processes. It is a mature way of looking how efficient the process CMMi prescriptions can be applied to any methodology. Reasons cited for “No” and “not Sure” : Different focus (People centric Vs Process centric) Agile is project management, engineering framework and CMMi is a process improvement framework It is compatible till L3 except IPM, OPD • This question was answered by 72% of the respondents • 68% of the respondents think that Agile and CMMi can coexist. Of these 60% are from organizations that have ISO or CMMi certification. • 72% of the respondents reported that their Agile projects were also being audited (as per the corresponding certification process)

17. CMMi on Agile What does CMMi v1.3 say? To help those who use Agile methods to interpret CMMI practices in their environments, notes have been added to selected process areas. These notes are added, usually in the introductory notes, to the following process areas in CMMI-DEV: CM, PI, PMC, PP, PPQA, RD, REQM, RSKM, TS, and VER. . . . Even if you are not using an Agile approach, you might still find value in these notes. . . . Some general background and motivation for the guidance given on Agile development approaches are found in the SEI technical note CMMI or Agile: Why Not Embrace Both! [Glazer 2008]. Reference: Section-5: Interpreting CMMI When Using Agile Approaches – CMMI-DEV, V1.3 by SEI Possible reason for higher maturity organizations to agree that Agile and CMMi are compatible Practitioner Groups of CMMi and Agile can integrate the practices to adopt the best of both

18. Coexistence with other models Integrating LEAN with Agile • 33% of the respondents reported using LEAN techniques with Agile • Most organizations (26%) have adopted KANBAN Integrating Agile with other Life Cycle Models • 46% of the respondents reported using Agile best practices in other LCMs • Agile practices predominantly used:

19. Benefits of Agile adoption • Key Benefits: • Better management of Requirement and Priority changes, Risks • Improved time-to-market and customer satisfaction

20. Benefits Perceived – Role wise • Ability to handle risks and improved customer satisfaction were rated high by Agile team and Agile Coach • Managing changing priorities and better project visibility were rated high by senior management and process professionals • Benefits perceived by organizations increase with Agile experience, inline with practices adopted and metrics collected/analyzed

21. Barriers to Agile adoption These observations are in line with the world wide survey reports by other organizations Agile Coaches/Consultants and Senior Management to focus on methodical transition to AGILE

22. Barriers Perceived – Role wise • Barriers reported by Senior Management and Process Professional are much higher than Agile Coach/Consultant and Agile Team • The size and Agile experience of the organization had no significant impact on the barriers

23. Benefits and Barriers - Impact • Time-to-market • IT and Business Collaboration • Handle risks earlier/ better • Manage changing priorities • Better project visibility • Customer/End-user satisfaction Customer Management • Agile cannot handle Maintenance • Lack of skilled resources • Agile produces Insufficient documentation • Lack of coordinated transition to agile • Lack of focus among PM/PMO Groups • Resistance to change • Manage changing requirements • Better Team-morale • Better collaboration Team • Project technical complexity / size • Lack of onsite - offshore collaboration • Lack of tools/ Automation for short iterations

24. Challenges Expressed Challenges Faced Possible Solution • Fixed Price Model is a challenge, T&M have worked well. • Collection of Traditional Metrics in Agile Projects. • Moving away from Command & Control; Self Organizing Teams • Scope creep and sign-offs. • Compensation / Appraisal related issues due to no individual Heroes • Communication/ Collaboration challenges between the distributed sub-teams. • Infrastructure needs for Continuous integration. Teams Not exposed to usage of tools and automation. • Frequent release cycles in sprints thus not enough time for testing. Very less focus on Non functional Test. Some legacy application testing cannot be fully automated. • No predefined performance benchmarks. • User story articulation by different stakeholders. • Focus on Maintainability and code quality. • Not all teams will be empowered or trained for ‘Test Driven Development’. Need External Support. • Can think of Hybrid model based on the scope clarity. • Redefine/Customize the traditional metrics. • Handholding by Agile coach. Mandatory Agile skill training. • Stakeholders to come together and decide. • Policies need to be flexible to have Team benefits • Address the dependency issues in Daily stand-up meetings. Effective Scrum of Scrums • Senior management to focus on infrastructure needs. • Teams to look into non-functional aspects during planning meetings. Detail planning of testing effort for manual testing • Benchmark within the organization . Participate in external Forums. • Breakdown stories for ease of development. Help customers articulate User Stories and Product Backlog. • Usage of Tools • Enhance People Agile skills; especially on Engineering side.

25. Best Practices Reported • Record the retrospection and use some Agile tools for tracking them. • Stake holder involvement in defining test scenarios. • More encouragement from management to adopt Test Driven and Behaviour Driven Development. • Invest on tools and infrastructure to run automated regression suite on every check in • Representative at onsite for each offshore team: Overlap of the key people from offshore with onsite team. • Slightly heavy onsite team for better coordination. • Good build plan. Report invocation from Build. • Teams based on functionality rather than skills • Buddy Testing from developer machine to save the deployment time. • All the team members who are part of particular sprint should be part of the scope and estimation. • Measurement of code complexity for quantification. • Use of Agile training from time to time, participation in Agile conferences and Forums.

26. Conclusions and Recommendations Agile as a methodology is well established - more than 50% of respondents have used it for past 2 years and included it in QMS. Scrum is the most widely used methodology followed by Kanban Agile usage is spread across multiple domains and project types; Over 75% of the respondents have confirmed using Distributed Agile While Management practices (predominantly from SCRUM) have been well adopted, the adoption of engineering practices (predominantly from XP) needs more rigor Usage of metrics and automation level increased with Agile experience Coexistence of Agile with other industry standards (like ISO and CMMi) has been agreed upon NEXT STEPS Data Benchmarking of Agile projects Practitioner Groups of CMMi and Agile can integrate the practices to adopt the best of both • Bibliography • CMMi for Development, Version 1.3 by SEI • “2011 State of Agile Development Survey Results” by VersionOne • “Agile adoption in India Survey Report – 2011” by ThoughtWorks • “Agile India 2012 Survey Results” by Valtech

27. Discussions THANK YOU