1 / 34

The difference that makes the difference

The difference that makes the difference. Make your test automation project a success story. Magnus Nilsson Lemontree. Agenda. The Test automation concept Success factors Components Test cases Framework Walkthrough of a test automation project Analysis phase and the business case

chadwick-mills
Download Presentation

The difference that makes the difference

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The difference that makes the difference

  2. Make your test automation project a success story Magnus Nilsson Lemontree

  3. Agenda • The Test automation concept • Success factors • Components • Test cases • Framework • Walkthrough of a test automation project • Analysisphase and the business case • Maintenance of an automated solution

  4. Test automation - success factors • Develop functional components • Decouple in- and output data from the script • Central data storage • Development project • strategy and • center of excellence • Combine components to test cases by actions as input data Functional components Test automation Central data storage Test Automation In-/outdata Project Concept

  5. Test automation - buildingup the components • Functionalityavailablein test • Developmentof the functionality as it comesavailable Func. 3 Phase 1 Phase 2 Func. 4 Test Automation Requirement Function 1 Function 2 Eachfunctionality is part of a component (or a componentitself) Function 1 Design and development Function 2 Concept Function 1 Test Func. 2

  6. Test automation - buildingup the components • Function 1: Createcustomerusing CRM CRM System CRMCustomer

  7. Test automation - buildingup the components • Function 1: CreatecustomerusingWebshop http://www.webshop.se WebCustomer

  8. Test automation - buildingup the components • Function 1: Createcustomerusing mobileapplication AppCustomer

  9. Test automation – test cases Credit check Order Customer Test case 1: [create] [new] Login Account [new] Test Automation Customer [open] Billing Test case 2: Inventory Account [new] Login Subscription Concept

  10. Test automation – test cases Credit check Order Customer Test case 1: ver 2.0 ver 1.0 [create] [new] Account [new] Login Account [new] Test Automation Customer [open] Billing Test case 2: Inventory Account [new] Account [new] Account [new] Login ver 1.0 Subscription Concept ver 2.0 ver 2.0

  11. Test automation - developmentof a framework • 1st generation script • Using automation tool for capture and replay only • Hardcoded data • 2nd generation script • Code refactoring for maintainable scripts • Uses generic functions for iterations and error handling • 3rd generation script • Data driven test (DDT) where dynamic data is taken from data table rows • Keyword driven test (KDT) where action words controls the actions • Use of driver scripts for executing multiple test cases • SME for test designs and requirement • More complex scripting by developers Framework Developmentof a

  12. Test automation - developmentof a framework ALM Dashboard Test resources • HP QuickTest Professional/Functional tester • Framework itself with • test management connection ALM, • keyword driven action scripts with • data table, • function libraries, • Repositories, • BPT, • recovery scenario • Generates code • Library definitions, template of standard functions • Structured data handling, input and output data Test plan Test lab Functionlibrary A Entity linked from previous component O LinkEntity_In In Application GUI layer Recovery scenario M Functionlibrary B FirstName In First name of customer List object Objectrecognitionengine M In Family name Framework FamilyName Text object M In Private or Business Table object OfficialName Test action script Business ID will be validated if provided BusinessID O In Out CustomerID_Out N/A The created CRM customer ID Repository Developmentof a Out Scriptstate_Out N/A Status of execution Data table objects

  13. Test automation - developmentof a framework • HP QuickTest Professional/Functional tester • Library definitions, template of standard functions • Applicationspecificlibraries • Standard actions in applicationsshould be in applicationlibrary • Component librariesenablingConfiguration Management • Branched Release Management for ST, AT and dev • Multiplecheckout and merge • Only driver scripts • Frameworklibraries handling framework and best practices • Structured data handling, input and output data Framework Developmentof a

  14. Test automation - developmentof a framework ALM Dashboard Test resources • HP QuickTest Professional/Functional tester • Library definitions, template of standard functions • Structured data handling, input and output data • Central data source • Decouple structure and data • Central structured data source Test plan Test lab Entity linked from previous component Entity linked from previous component Entity linked from previous component O O O LinkEntity_In LinkEntity_In LinkEntity_In In In In Test action script M M M FirstName FirstName FirstName In In In First name of customer First name of customer First name of customer Functionlibrary A Data & Structure M M M In In In Family name Family name Family name Framework FamilyName FamilyName FamilyName M M M Master DB In In In Private or Business Private or Business Private or Business OfficialName OfficialName OfficialName Data table Business ID will be validated if provided Business ID will be validated if provided Business ID will be validated if provided BusinessID BusinessID BusinessID O O O In In In Data & Structure Data Out Out Out CustomerID_Out CustomerID_Out CustomerID_Out N/A N/A N/A The created CRM customer ID The created CRM customer ID The created CRM customer ID Developmentof a Out Out Out Scriptstate_Out Scriptstate_Out Scriptstate_Out N/A N/A N/A Status of execution Status of execution Status of execution Component A sheet Component B sheet CreateCustomersheet

  15. The test automation project The Business Case Phase 0 Phase 1 Analysis Detailed design Requirement Development System test Walkthrough Acceptance test Plan nextphase Phase 2 Documentation & Training Week Detailed design Development Project …

  16. Test automation project - business case • Why do we need a business case? • Test automation cost time and money • The automated solution must at least ensure the time spent in development, maintenance and execution is less than a manual solution would cost to accomplish the same goal. • What is a test automation business case? • List of all costs and savings associated with test that possibly can be covered by test automation. • Not all drivers are possible to calculate directly as quality. But quality can be expressed as an estimation of the number of testing hours to ensure the desired level of quality. • Additional test automation drivers as repeated executions is calculated indirectly as increased quality. • Howcan a business case be used? • Business case can be used to predict effectiveness as automation deliveries should follow the calculated schedule. • ROI should be calculated at regular audits based on the business case. Walkthrough Project

  17. Test automation project - business case Total test caseexecutionsper cycle 1000 Cost for manual regression tests Numberof test cycles 4 Executiontime per test, h 0.5 Executioncost per hour 100 Annual cost for execution of regression test 200000 Cost for error handling AT Production Error source Number of errors found per year 1000 500 Handling time per error (report, reproduce, administer, test data, close defect) 4 2 2 Number of FTE's for error handling per year 2 Error handling cost per hour 100 100 200000 Annualerror handling costs 200000 Annual error correction costs for errors 0 100000 External and business costs for errors Cost multiplier for Production Errors 2 Business costs for productions errors1) 400000 Grand total 1100000 1)Goodwill, compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

  18. Test automation project - Business case Reduction ratio Annual reduction/cost Description Acc reduction Savings - Manual Test Labour Year 1 60% 120000 120000 260000 Year 2 70% 140000 420000 160000 Year 3 80% 90000 90000 Year 1 10% Savings - Error Reduction 270000 Year 2 20% 180000 Year 3 30% 540000 270000 Walkthrough Phase 1 100000 Implementation Cost (Year 1) Phase 2 70000 Phase 3 70000 Year 1 proj support, SME 50000 Operational costs (running) 60000 Year 2 TAC OP Project 60000 Year 3 TAC OP Acc savings Year 1 -80 000 -80 000 Total savings Year 2 260000 180000 Year 3 370000 550000

  19. Test automation project - Business case • Implementation cost calculations • Initial estimates • Design estimates Analyis&Design Selection Development Value Component/Functions CRM Customer Technology Mixed 2 Maintenance Create 20 10 Yes 1 Walkthrough Reusability Terminate 10 5% 10 1 Simple TC 45 1.5 WS Customer Complex TC 61 2 Create Framework New 20 3 15 Test data Create App Customer 1 Project Integrations 10 5 Create 2 20

  20. Business case – followup short term Phase 1 Analysis Walkthrough Week Detailed design Development System test Acceptance test Test cases Plan nextphase Project Planned Delivered Phase 2 Detailed design Week Development

  21. Business case – followuplong term”Test automation strategy” • Description of the standard approach on how to use the tools • Company specific set of rules on how automation should be done • Generic solution that is not dependent on individuals • Maintenance focused • Encapsulated by test automation centre • Enforced by framework • Ensured by the automation project But we need to know what we are testing! Walkthrough We should not test manually what we execute automatically! Project

  22. Analysis • Analysis • Analysis based on existing documentation, test cases, use cases and requirements (static test review) • Provides the prioritization to an analysis structure template • Disposition • Decision making: if we go live what does and does not work? • What business processes are working and not working or not tested? • Often details on number of test cases passed and failed on integration and system level is not vital • Test and defect status itself often only aids prioritizing of resources/focus • Structure • Common and clear reporting on a level that enables correct business decisions • Structured test design and coverage analysis • Will control the test automation deliveries • Used in test management to create common priorities for both manual and automated tests • Implementation • After the analysis and approved structure design, the structure may be implemented using available tools in HP QC/ALM Walkthrough Project

  23. Analysisstructure- disposition • what is the system used for? • how is it done? • which are the requirements on process level? • which data conditionsare covered by use cases? Usecase Process Requirement Example Walkthrough Data condition 1 Customer Data condition 2 Web Self Care New Project Private Business

  24. Analysisstructure- disposition Usecases/product Requirements Customer Account Subscription Folder area Process Description Prio Create Terminate New Modify Cancel New 1 Walkthrough Sales Web shop 1 1 3 1 10 Web selfcare Customer operations 3 Sales CRM 1 3 1 1 3 1 15 1 Sales Device App 2 1 3 Sales Device Online 2 1 1 1 Product configurations Product Inventory 1 Project

  25. Requirement and componentcoverage Components CRM Requirements Web Shop Customer Account Customer Folder area Requirement Process Data condition Test case Test case Prio Create Terminate New Modify Cancel Create Web selfcare R1.1 Create Customer TC23 Walkthrough Sales 1 X Customer op R2.3.4 Create Customer TC12 Sales 1 X Customer op R2.4.1 Terminatecustomer TC1 Sales 2 X Create bank account Customer op R4.1.1 TC45 Sales 2 X X Customer op R4.1.1 Createinvoiceaccount TC6 1 Sales X Project

  26. Coverage analysis and status report; BC follow up Status Not covered No run Not completed Failed Passed Customer 3 2 7 1 Customer operations 1 2 1 1 New 2 1 Not completed Terminate Not covered 1 Web selfcare 1 3 3 1 New 3 1 Passed Mobile device 2 0 3 2 New 0 2 3 Failed

  27. Coverageanalysis and status report, HP ALM • Coverage in ALM • Test either • manual, • automated or • business component Walkthrough Project

  28. Maintenance • Test cases created • Test data prepared • Result reporting arranged • Press the button Walkthrough Functional tester Then what? X Project Automation Engineer

  29. Project organisation - Userolestoenforcecompetence Test Automation Center Automation Project Lead Project C Project B Project A Automation stakeholder Automation stakeholder Automation stakeholder Automation stakeholder Subject Matter Expert Subject Matter Expert Subject Matter Expert Automation Solution Architect Automation Analyst Subject Matter Expert Walkthrough Automation Engineer Project

  30. Project organisation - Test automation center Business Objectives Strategy Management Organization Template Routines Processes Development Walkthrough Execution Maintenence Project

  31. Test automation center Management Test management Release management Project planning Defect and Issue Management Change request Communicating Reporting Plannedactivities Resolution Supplier TAC Service Management Operations Project A Project B Project C Request Maintenance ST SIT AT Result TAC Maintenance TAC Development TAC Execution

  32. Short about Lemontree • Swedish company established 1999 • Awarded Gasell and Superföretag 2008-2012 • Turnover +98 MSEK 2011 • AAA-rated • About 100 employees, growing intensive • Owners are actively working in the company, no external capital • Offices in Stockholm and Oslo • QM trainee program since 2012 • Focus areas • Quality Management • eGovernment • Enterprise Solutions

  33. Quality Management AQUA solutions

  34. Thanks! Magnus Nilsson Lemontree Solution Manager magnus.nilsson@lemontree.se

More Related