1 / 22

Economic Perspectives in Test Automation: Balancing Automated and Manual Testing with Opportunity Cost

Economic Perspectives in Test Automation: Balancing Automated and Manual Testing with Opportunity Cost. Paper By – Rudolf Ramler and Klaus Wolfmaier Presented By – Pallavi Gulati. Topics. Why Automated Testing? Simplistic Cost Model for deciding between Automated vs. Manual Testing

lyre
Download Presentation

Economic Perspectives in Test Automation: Balancing Automated and Manual Testing with Opportunity Cost

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Economic Perspectives in Test Automation:Balancing Automated and Manual Testing with Opportunity Cost Paper By – Rudolf Ramler and Klaus Wolfmaier Presented By – Pallavi Gulati

  2. Topics • Why Automated Testing? • Simplistic Cost Model for deciding between Automated vs. Manual Testing • Flaws in the Simplistic Cost Model • New Model based on Opportunity Cost • Benefits offered by the new model • Future Plans

  3. Why the need to Automate Tests? • Testing accounts for at least 50% of total project cost • Fast-paced iterative development • Shrinking schedules and budgets • Test Automation proposed to reduce testing costs

  4. Advantages - Automated Tests • Reduces overall cost of RUNNING tests • Though some initial cost required for writing automated tests • Increases the number of tests run • Increases frequency at which tests are run • Incremental cost of test running is less • Major concern in deciding whether to fix a particular bug or not

  5. Disadvantages - Automated Tests • Can not automate every test case – timing-related, appearance, user-friendly, some system & integration test cases, involving different hardware setups • May not catch new bugs, targeted towards regression bugs • Higher latency as compared to manual testing which can start immediately

  6. When to go for automation? • How do managers generally decide when to automate? • Decision is based on factors like: • Efforts required to run manual test • Frequency of testing • Cost of third-party / in-house developed tool • Efforts required to write automated tests

  7. Simplistic Cost Model Vm = Estimated Cost of designing manual tests Am = Overall cost of manual testing for some # of times Va = Estimated Cost of designing automated tests (may also include cost of test tool) Aa = Overall cost of automated testing for some # of times

  8. Flaws - Simplistic Cost Model • Manual & Automated Testing are not comparable • Exploring new functionality vs regression testing • Only costs are analyzed not benefits • Manual testing starts early in the process but some latency in automated testing • All test cases & executions are considered equal • Project Context (budget, resources) not considered • Misses additional cost factors like training on the testing tool, new process adaptation

  9. Why automated tests fail to provide expected benefits? • Due to flaws in estimation using the Simplistic Cost Model and other factors like: • Failure to understand that automated tests are targeted for regression bugs • Gross underestimation of the effort required to develop automated tests • Practically no allocation of efforts for maintaining automated tests

  10. When to automate? vs. What to automate? • It does not help to automate every test • Need to decide trade-off between automated and manual testing • Alternative - Opportunity Cost Model

  11. Opportunity Cost Model • Consider manual & automated test cost • Decide how many manual test cases should be automated • Pick those which will give maximum benefit / risk mitigation / coverage • Project budget & deadlines considered • Looks something like...

  12. Opportunity Cost Model • Va = 1 hour to automate 1 test • 0 min to run 1 automated test • 0 min to come up with 1 manual test • Dm = 15min to execute 1 manual test

  13. Restriction 1

  14. Restrictions 2 and 3

  15. Target Function Ra = Risk Mitigation for automated testing Rm = Risk Mitigation for manual testing This allows us to include the benefit obtained by automating a test

  16. Scenarios for Analysis

  17. Scenarios for Analysis

  18. Model can be extended to include Additional Influencing Factors • Tests that rely solely on automation • e.g. 100 concurrent transactions • Productivity changes over time • Causes an outward curved graph • Growing Test Effort in iterative development • Improves risk mitigation of automated test • Maintenance cost • Early / Late return on investment • Developing automated test early using data driven technique • Defect detection capability • random input invoking crash

  19. Advantages– Opportunity Cost Model • Draws attention to balancing automated and manual testing • Benefits as well as costs • Risk mitigation capability of tests • Budget and deadlines • More realistic comparison between automated and manual testing

  20. Future Plans • Reduce complexity & make it more practical to use • Evaluate preciseness and completeness • Identify & add more influencing factors • Analyze need to add more restrictions

  21. Summary • Advantages of Automated Test • Simplistic Cost Model • Flaws • Proposed Opportunity Cost Model • Benefits offered • Future Plans for expanding the model

  22. Wake up !!!

More Related