Automation Culture: Essential to Agile Success Agile Austin

1. Automation Culture:Essential to Agile SuccessAgile Austin Geoff Meyer, geoffrey_meyer@dell.com October 2013 Last updated: September 29, 2013

2. Session Objectives • Challenges to expect when adopting test automation during your Agile transition • Practical solutions to what & when to automate • Real-world examples − from a large organization perspective • Challenges automating in a HW-dependent environment • For organizational leaders driving test strategy • IS • A Tools Discussion • Focused on how to develop Test Automation • IS • NOT

3. Introductions

4. Geoff Meyer • Dell Inc, 1998 – present • Test Architect • Agile Test & Automation Strategy • Agile Steering Committee Co-Chair • Global Projects • 15 Scrum teams (2 regions) • 11 Scrum teams (4 regions) • 9 scrum teams (2 regions) • 7 Scrum teams (4 regions) • NCR Corp. 1984 – 1998 • SW developer, Project Lead, SW Manager • B.S. Computer Science, San Diego State University • Masters Engineering Management - NTU

5. Agenda • Why Automation is essential to Agile • The Dell Context and Agile @ Dell • Challenges and Common Pitfalls • The Automation Landscape • Foundations of an Automation Culture • Care and Feeding of the Automation Culture

6. Ø The Need,The Challenge,The Pitfalls

7. Why is Automation So Important in Agile? • Long-term • Maximizes velocity of Scrum team (increased value creation) • Creates capacity for Exploratory and ad-hoc Testing • Enables activities that can’t be done cost-effectively by humans And if you don’t… Quality is at risk from an unmanageable regression suite • Near-term • Ensures that you don’t break what you just built • Provides safety net for developers & rapid feedback to new changes • Continuous Integration and use of Build Verification Test (BVT)

8. Differences with Automation in Waterfall? In Waterfall… • Automated tests are derived from the backlog of completed features • In Agile, Automation can be incorporated in the requirement • Testing and automation is performed after Development is “complete” • Focus is on first-time discovery of defects and optimizing your test coverage vs. Agile… where automation development is ongoing and provides immediate feedback throughout development

9. The Context @ Dell “Apply Test Automation in the context of your organizational automation needs” --- Bob Galen, Agile Coach http://amazngwallpapers.blogspot.com

10. Dell Enterprise Solutions GroupGlobal Design and Development using Agile Scrum Silicon Valley Design Center Noida Design Center Austin Design Center Bangalore Design Center

11. Dell Enterprise SolutionsSoftware Products • Server Systems Management • Converged Infrastructure Systems Management • Private Cloud Systems Management • Console Plug-ins (i.e. for SCCM, vCenter…) • Common Product Characteristics: • Large hardware support test matrix • Software is installed in the Data Center • Enterprise update cycles ~6 months • Products must function even as underlying HW, FW, BIOS, and Drivers are upgraded - SUSTAINING

12. Agile @ Dell Roles/Responsibilities Product Owner • Small teams • Co-located • Distributed Projects • Teams formed from functional silos Product Owner Proxy ~1 per Scrum Scrum Master 1 per Scrum Development 4-5 per Scrum Test 3:1 Tech Pubs 1 per 3 scrum teams UI 1 per Scrum Scrum teams

13. Agile @Dell with Adaptations Feature Complete Release Exit Release Plan Code Freeze Sprints Hardening Stability 1 2 3 N-1 N … 1 2 3 N-1 N … 1 2 3 N-1 N Extended Sprint Test Software System Test OLP Define Plan Develop Launch

14. Pitfalls encountered at • Black Box Test Mentality • Architecture(s) not optimized for Automatability • Automation was interpreted by many as “Automate the UI” • Insufficient SW engineering background across Test teams • Development didn’t historically automate unit tests • Build teams were staffed with non-Build practitioners • Minimal guidance to Test beyond “Go forth and automate”

15. Which Resulted In… 1 – “A nasty condition where people start valuing ticking off features more than tracking the real outcome of the project.”--- Martin Fowler • Over-emphasis on UI automation • Test Automation not keeping up within the sprint • Feature Devotion1 • Multiple automation tools & licenses • Test scripts not designed for re-use

16. The Automation Landscape http://amazngwallpapers.blogspot.com

17. Opportunities for Automation • It’s not just about automating test cases • Can also provide efficiencies to: • Test Preparation, Setup and Configuration • Non-Functional Testing (“ility”) • Longevity, Scale and Performance Characterization • Compatibility Testing (Solution and Device)

18. Test Preparation • Bare-metal Deployment • Setup & Configuration • OS Provisioning and Configuration • Test case staging • Environment Cleanup/Baseline • Virtualization-based test environments

19. Test Content DevelopmentWhole team ownership of QA & Automation • In-Sprint, Automated Acceptance Tests • Web-services (or CLI) Test Automation for Functional Testing • Automated CI (UT, Build, BVT) • Unit Test Automation 2 CAUTION: Application Architecture can be an enabler or inhibitor UI automation on Customer Usage workflows 2 - Agile Testing: A Practical Guide for Testers and Agile teams

20. Application Architecture Matters • Does it facilitate testing at the API/Services level? • Does the Business/Error Logic reside below the services level? • Does the UI architecture support test automation other than record-playback? For Large organizations: • Standardize UI architectures across the product portfolio • HTML5, Flex, Silverlight… • Standardize Service architectures across the product portfolio • SOAP, REST, CLI, API

21. Prioritize and Identify what NOT to Automate Prioritize High ROI • BVT Candidate Acceptance Test • Test of core functionality that executes in a short duration • Core Functionality to be run in Nightly Regression • Sustaining Test Candidate Test • Functional test that verifies the application can withstand subsystem changes What NOT to Automate • Everything • GUI – Except for High-value customer usage flows after the UI is stable • Tests that would only find low severity bugs • Where the product is unchanging

22. Non-Functional Testing 3 Often times analysis of these areas are simply too cost-prohibitive to be done manually 3- http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ Performance Characterization Longevity Stress Scale Concurrency

23. Foundations of an Automation Culture Images.yahoo.com

24. Keys To A Culture Transition • Development vs. Test • A different “School” of Test4 • Evolve from Functional Responsibility …to Whole Product Ownership 4 – Scott Barber “Approaches to Software Testing: An Introduction“

25. Establishing an Automation Culture • Identify • Near and long-term automation focus areas • Inventory the culture and skillset of organization • Establish • Tooling and Infrastructure Standardization • Your ‘community’ • Develop • Workforce transition plan • Organize • Embedded vs. Specialized Automation team • Re-align project staffing • Operationalize

26. IdentifyFocus Areas for Automation 4 4 - http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/

27. Establish • Multiple automation Tools/FW’s may be needed to automate at layers of testing: • Unit level • Middle tier • UI-driven • Tooling and Infrastructure • Tooling Standards and Frameworks • Team to develop/manage home-grown tools • Lead tool evaluations to meet new architectures & technologies • Community: Internal and External

28. Develop & Organize • Adjust Hiring Practices • Re-align Project staffing • Embedded Automation vs. Specialized Team • Training • New-hire • Ongoing Confidential

29. Operationalize • Whole team commitment in words and actions • Include Test Automation in Acceptance Criteria • Unit • Acceptance • Continuous Integration and BVT • Establish Metrics, Governance & Rewards

30. Automation @ PG Enterprise Solutions Group How did we go about it? Process/Project Internal Training Programs Non-Functional Scale, Longevity Automated BVT In-Sprint Automation UI Automation IVT Device Automation Day Service-level automation (CLI & Web-services) IVT Networking & Servers Setup & Config 2014 2012 2013 2010 2011 Automation Content Libraries TDD & Automated UT Automation Architects Maximize Utilization Established ALT Accelerate Automation Skillset “Automate-First” Culture/Organization

31. Care and Feeding of the Automation Culture

32. Maintaining the Automation Culture • Operationalize Automation • Include Test Automation in the Acceptance Criteria • Encourage Community Practice and Participation • External - Industry User Groups & Conferences • Internal - Brown-bag sessions, Showcases, Mini-conference • For Large organizations: • Overcome skillset deficiencies by adjusting staffing strategy • Automation Leadership Team, Automation Architect(s) • Continually monitor the alignment of Architecture, Dev and Test • Develop reward systems

33. Recognizing a Successful Automation Culture • Automation is a shared responsibility • Automatability is a key architectural consideration • Continuous Integration & BVT • Teams are staffed to include an automation skillset • Automation is operationalized

34. Resources • Agile Manifesto - http://agilemanifesto.org/ • Articles/Blogs: • http://support.smartbear.com/articles/testcomplete/automated-testing-agile-environment/ - SmartBear • http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ - Lisa Crispin • http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=17793&tth=DYN&tt=siteemail&iDyn=2 – Rajini Padmanaban • Beyond Agile Testing: http://www.utest.com/int-v1/a/beyond-agile-testing - uTest • Testing and Checking Refined: http://www.satisfice.com/blog/archives/856 - James Bach and Michael Bolton • Books: • Scaling Software Agility: Best Practices for Large Enterprises – Dean Leffingwell • How Google Tests Software – James Whittaker, Jason Arbon, Jeff Carollo • Agile Testing: A Practical Guide for Testers and Agile teams – Lisa Crispin, Janet Gregory • http://www.dorothygraham.co.uk/automationExperiences/index.html - Dorothy Graham • Presentations: • Approaches to Software Testing: An Introduction – Scott Barber • Agile Testing: Challenges Beyond the Easy Contexts – Bob Galen

35. Questions?