1 / 76

The difference that makes the difference

The difference that makes the difference. Make your test automation project a success story. Magnus Nilsson Lemontree. Agenda. The Test automation concept Walkthrough of a test automation project Execution and maintenance of an automated solution Test strategy

gasha
Download Presentation

The difference that makes the difference

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The difference that makes the difference

  2. Make your test automation project a success story Magnus Nilsson Lemontree

  3. Agenda • The Test automation concept • Walkthrough of a test automation project • Execution and maintenance of an automated solution • Test strategy • Development of a Framework • Organizing your code

  4. The test automation concept • Fundamentals behind test; a short review of • Lifecycle activities • Test levels • Test methods • Development models • Test automation and test objectives • Test automation success factors Test Automation Concept

  5. Test as part of the lifecycleactivities Business process Architecture Project Requirement Test Maintenance Monitoring Development Test Automation Test Defect Management Design Release Concept

  6. Connecting the activitiesTest levels • Unit test • System test • Integration test • Acceptance test Integration test Test Development Test Automation System test Test types • Functional • Non functional Acceptance test Unit test Concept

  7. Connecting the activitiesTest methods • Black box • Testing delivered functionality • GUI test • White box • Testing internal structures • Code • Testing modules, classes, libraries, functions on code level Black box Test Code Test Automation White box Concept

  8. Connecting the activitiesDevelopmentmodels • Waterfall • The lifecycle approach • Iterative • Planned repeated cycles • Agile • Iterative and development adaptive to changes • Practicing continuous Integration, CI • Scrum • Sprint-based development based on both on agile and iterative development methods Waterfall Development Scrum Test Automation Agile Iterative Concept

  9. The challenge of test Testingmethods Test levels Integration Black box Code Acceptancetest Development Developmentmodels System test Agile test Test Automation Waterfall White box Test Unit test Concept

  10. Test automation is all about unifying test. • Development • Increase code quality by implementing unit tests • Developmentmethodology TDD • Automatedbuilds, CI • Environment • Test environment creation • Test data management, TDM • Harness • Stubbing interfaces • Test • Scripted tests • Test interfaces • GUI • Performance Environment Automation Development Test Automation Test Harness Concept

  11. The challenge of test, connecting… Testingmethods Test levels TDM Integration Black box Code Acceptancetest Development TDD Development models System test Agile CI GUI Test Harness Test Automation test Automation Waterfall White box Test automation Test Unit test Concept

  12. Example 1 - Test harness • Enables test before interfaces exists or are available, agile development but also project delivery regardless development model System integration test System test Acceptance test Test Harness Test Automation Project 1 X X X Project 2 X Project 3 X Start SIT Concept Start AT Time Test harness Release Stableenvironment

  13. Example 1 - Test harness • Each flow have a known expected result • Data driven solution • Includes test of functions, web services and interfaces Test Harness Calling system Test harness Simulated system Test Automation 1. Call service 2. Evaluate simulator rules 3.1. Fetch simulator response 3.2.1 Call live test system 3.2.2 Returnresponsetoharness 4.Return responsetocalling system Concept Calling system Test harness Simulated system

  14. Test automation and test objectives • Test objectives depends on the testing level. But finding bugs is one of the main objectives. • A bug is • Wrong or missing requirement • Bad design or architecture • Coding error • Poor implementation • The objectives of test is to make sure none of the above takes place. Test Automation Concept

  15. Test automation and test objectives • The automated solution must fulfill the test objectives • Input is functional requirements, design use cases and test cases. • Output is functional verifications interpreted as check points. Usecases Test objectives Test cases Test Automation Requirement Concept

  16. Test automation success factors • Develop functional components • Decouple in- and output data from the script • Central data storage • Development project • strategy and • center of excellence • Combine components to test cases by actions as input data Functional components Test automation Central data storage Test Automation In-/outdata Project Concept

  17. Test automation GUI – functionalcomponents External systems (with or without test harness) Calling system Credit check Billing Product Customer Configuration Account Login Test Automation Order Subscription Product Task Concept Inventory

  18. Test automation GUI – input data • Eachcomponenthave a defineddegreeoffreedom • Population of mandatory data • Trigger ofvalidations Start customer form BusinessID Private customer? Populate business customer form no Test Automation CustomerType Business Name yes Customer Name Populate private customer form Validate Type Address Info Concept Completecustomer form and save Validate customer Next process

  19. Test automation GUI – output data • Resultsofvalidations • Identification data Start customer form BusinessID Private customer? Populate business customer form no Test Automation CustomerType Business Name yes Customer Name Populate private customer form Validate Type Address Info Customer ID Concept Completecustomer form and save Customer Status Validate customer Next process

  20. Test automation GUI – output data • Components arelinked by useofidentification data. • Output data from onecomponent is input data for anothercomponent Customer [create] Account [new] CustomerID Test Automation Concept

  21. Test automation GUI – test cases Credit check Order Customer Test case 1: [create] [new] Login Account [new] Test Automation Customer [open] Billing Test case 2: Inventory Account [new] Login Subscription Concept

  22. Test automation GUI – changes Credit check Order Customer Test case 1: ver 2.0 ver 1.0 [create] [new] Account [new] Login Account [new] Test Automation Customer [open] Billing Test case 2: Inventory Account [new] Account [new] Account [new] Login ver 1.0 Subscription Concept ver 2.0 ver 2.0

  23. Buildingup the components • Functionalityavailablein system test • Developmentof the functionality as it comesavailable Func. 3 Phase 1 Phase 2 Func. 4 Test Automation Requirement Function 1 Function 2 Eachfunctionality is part of a component (or a componentitself) Function 1 Design and development Function 2 Concept Function 1 System test Func. 2

  24. Buildingup the components • Systems ready • Testing relations offunctionalityRegression, E2E, Acceptance Function 1 Func. 2 Func. 3 Customer Phase 3 Phase 4 Func. 4 Test Automation Func. 3 Function 5 System test Func. 5 Func. 6 Func. 4 Regression test Function 1 Func. 2 Function 3 Account Func. 2 Func. 4 Concept Function 1 Function 1 Function 5 Func. 6 Function 5 Func. 6

  25. Example 2 - Agile Automation Development Release Iteration 3-4 weeks Agile Test Automation Requirement Requirement Test Design Concept D. Sprint release A. Product backlog B. Sprint backlog C. Sprint with automation

  26. Example 2 - Agile Automation sprint schema Sprint 2 Sprint 0 Sprint 1 Requirement Function 1 Function 3 Function 2 Function 4 Function 5 Function 1 Function 3 Design and development Function 2 Function 4 Test Automation Function 1 Function 3 System test Func. 2 Func. 4 Regression test Function 1 Concept Function 2

  27. The test automation concept • Lifecycle activities • Test levels • Test methods • Development models • Test automation and test objectives • Test automation success factors COVERED Test Automation Concept

  28. Agenda • The Test automation concept • Walkthrough of a test automation project • Execution and maintenance of an automated solution • Test strategy • Development of a Framework • Organizing your code

  29. Walkthrough of a test automation project • Project activities • Project plan - phased implementation • Project organisation • Business case and followup Walkthrough Project

  30. Test automation project - activities • Analysis • Design • Requirement • Development • Test • Documentation • Training • Execution Customer Customer Customer Order Order ? Customer ? ? Product Product Product Product ? Walkthrough ver 1.0 ver 1.0 ver 1.0 ? Order Order ? Project ver 2.0 ver 2.0 ver 2.0

  31. Project plan – phased implementation Phase 1 Analysis Detailed design Development System test Walkthrough Acceptance test Plan nextphase Phase 2 Week Detailed design Development Project

  32. Project organisation - Userolestoenforcecompetence Test Automation Center Automation Project Lead Project C Project B Project A Automation stakeholder Automation stakeholder Automation stakeholder Automation stakeholder Subject Matter Expert Subject Matter Expert Subject Matter Expert Automation Solution Architect Automation Analyst Subject Matter Expert Walkthrough Automation Engineer Project

  33. Project organisation - Test automation center Business Objectives Strategy Management Organization Template Routines Processes Development Walkthrough Execution Maintenence Project

  34. Test automation center Management Test management Release management Project planning Defect and Issue Management Change request Communicating Reporting Plannedactivities Resolution Supplier TAC Service Management Operations Project A Project B Project C Request Maintenance ST SIT AT Result TAC Maintenance TAC Development TAC Execution

  35. Business case • Why do we need a business case? • Test automation cost time and money • The automated solution must at least ensure the time spent in development, maintenance and execution is less than a manual solution would cost to accomplish the same goal. • What is a test automation business case? • List of all costs and savings associated with test that possibly can be covered by test automation. • Not all drivers are possible to calculate directly as quality. But quality can be expressed as an estimation of the number of testing hours to ensure the desired level of quality. • Additional test automation drivers as repeated executions is calculated indirectly as increased quality. • Howcan a business case be used? • Business case can be used to predict effectiveness as automation deliveries should follow the calculated schedule. • ROI should be calculated at regular audits based on the business case. Walkthrough Project

  36. Business case Total test caseexecutionsper cycle 1000 Cost for manual regression tests Numberof test cycles 4 Executiontime per test, h 0.5 Executioncost per hour 100 Annual cost for execution of regression test 200000 Cost for error handling AT Production Error source Number of errors found per year 1000 500 Handling time per error (report, reproduce, administer, test data, close defect) 4 2 1 Number of FTE's for error handling per year 2 Error handling cost per hour 100 100 200000 Annualerror handling costs 200000 Annual error correction costs for errors 0 100000 External and business costs for errors Cost multiplier for Production Errors 2 Business costs for productions errors1) 400000 Grand total 1100000 1)Goodwill, compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

  37. Business case Reduction ratio Annual reduction/cost Description Acc reduction Savings - Manual Test Labour Year 1 60% 120000 120000 260000 Year 2 70% 140000 420000 160000 Year 3 80% 90000 90000 Year 1 10% Savings - Error Reduction 270000 Year 2 20% 180000 Year 3 30% 540000 270000 Walkthrough Phase 1 100000 Implementation Cost (Year 1) Phase 2 70000 Phase 3 70000 Year 1 proj support, SME 50000 Operational costs (running) 60000 Year 2 TAC OP Project 60000 Year 3 TAC OP Acc savings Year 1 -80 000 -80 000 Total savings Year 2 260000 180000 Year 3 370000 550000

  38. Business case – followup Phase 1 Analysis Walkthrough Week Detailed design Development System test Acceptance test Test cases Plan nextphase Project Planned Delivered Phase 2 Detailed design Week Development

  39. Test automation strategy • Description of the standard approach on how to use the tools • Company specific set of rules on how automation should be done • Generic solution that is not dependent on individuals • Maintenance focused • Encapsulated by test automation centre • Enforced by framework • Ensured by the automation project Walkthrough We will now go through the test automation project Project

  40. Analysis and design • Evaluation • Scope • Analysis • Analysis based on existing documentation, test cases, use cases and requirements (static test review) • Provides the prioritization to a analysis structure template • Disposition • Decision making: if we go live what does and does not work? • What business processes are working and not working or not tested? • Often details on number of test cases passed and failed on integration and system level is not vital • Defect status itself often only aids prioritizing of resources/focus • Structure • Common and clear reporting on a level that enables correct business decisions • Structured test design and coverage analysis • Will control the test automation deliveries • Used in test management to create common priorities for both manual and automated tests • Implementation • After the analysis and approved structure design, the structure may be implemented using available tools in HP QC/ALM Walkthrough Project

  41. Analysis and design - evaluation • Value • Part of business case, added value by the test case, frequency and time to execute • Stable • Can the stability be impacted by sources out of control? • Test data • Is all test data needed defined or is a manual process involved? • Reusability • Can the business flow be reused in the automated component solution? • Well defined • Is the business flow well defined and no manual decisions needed? • Fully automated flow • No manual interventions needed? • Maturity • Is the functionality in maintainable where only the test data may be changed? • Defect frequency • Are there a lot of issues in the functional area? Walkthrough Project

  42. Analysis and design - the evaluation star Value Defectfrequency Stable Prio 1 test case Prio 2 test case Test data Walkthrough Maturity Project Fully automated flow Reusability Welldefined

  43. Analysisstructure- disposition • what is the system used for? • how is it done? • which are the requirements on process level? • which data conditionsare covered by use cases? Usecase Process Requirement Example Walkthrough Data condition 1 Mobile subscription Data condition 2 Web selfcare New Project Free surf Free SMS

  44. Analysisstructure- disposition Usecases/product Requirements Customer Account Subscription Folder area Process Description Prio Create Terminate New Modify Cancel New Walkthrough Sales Web shop 1 1 3 1 10 Web selfcare Customer operations Sales CRM 1 3 1 1 3 1 15 Sales Device App 2 3 Sales Device Online 2 1 1 1 Product configurations Product Inventory 1 Project

  45. Requirement and component design - refresh • All processes requires several components to enable E2E automation • Each component mainly automates one business process or interface • Validations may be performed in interfacing systems • A component may be developed in iterations • Components are used together – keyword driven • Components handle unlimited data conditions – data driven • Map which processes require which components Walkthrough Project

  46. Requirement and componentcoverage Components CRM Requirements Web Shop Customer Account Customer Folder area Requirement Process Data condition Test case Test case Prio Create Terminate New Modify Cancel Create Web selfcare R1.1 Create Customer TC23 Walkthrough Sales 1 X Customer op R2.3.4 Create Customer TC12 Sales 1 X Customer op R2.4.1 Terminatecustomer TC1 Sales 2 X Create bank account Customer op R4.1.1 TC45 Sales 2 X X Customer op R4.1.1 Createinvoiceaccount TC6 1 Sales X Project

  47. Coverageanalysis and status report • Subscriptionrequirement not completelycovered • New subscriptionrequirementcovered by web selfcare Status Not covered No run Not completed Failed Passed 1 Subscription 4 1 5 1 Customer operations 1 3 1 New 2 1 Not completed Modify Failed 1 1 1 Cancel 1 Not covered 3 Web selfcare 3 0 New 0 3 Passed

  48. Coverageanalysis and status report, HP ALM • Coverage in ALM • Test either • manual, • automated or • business component Walkthrough Project

  49. Component design template • Coverage • Covered test cases and requirements;Requirement and componentcoverage • Component description • Prerequisities • Data • Environment • Design description • Component type • Flow/usecasedescription • Detailed steps • Data table • Input/output data • Verification element Walkthrough Project

  50. Component data table Fieldname Input/output Mandatory/optional Description Entity linked from previous component O LinkEntity_In In M FirstName In First name of customer M In Family name Walkthrough FamilyName M In Private or Business OfficialName Business ID will be validated if provided BusinessID O In Out CustomerID_Out N/A The created CRM customer ID Project Out Scriptstate_Out N/A Status of execution

More Related