Regression Testing – What to Automate and how

1. Regression Testing – What to Automate and how Paul GerrardGerrard Consulting1 Old Forge CloseMaidenheadBerkshireSL6 2RD UK e: paul@gerrardconsulting.comw: http://gerrardconsulting.comt: 01628 639173 Assurance with Intelligence

2. In this session, I want to re-trace our early steps to automation Is life hard at the end because we didn’t think it through the beginning? Assurance with Intelligence

3. Automating regression tests Slide 4 Assurance with Intelligence • Regression tests are the most likely to be stable and run repeatedly so: • Automation promises big savings of time • Automation ensures reliable execution and results checking • Stable tests/software are usually easiest to automate • Hold on! Paradox ALERT!!!! • We regression test because things change • Chaotic, unstable environments need it the most • When things change automation is HARD!

4. What is the regression testing thought process? Assurance with Intelligence

5. How I think about regression testing Establish what we mean by regression testing Understand why regressions occur Combating regression – there are choices Who are the stakeholders for regression testing and what do they want? Context of regression testing Regression test policy Selecting what to regression test Selecting what regression tests to automate Assurance with Intelligence

6. A common experience? • Regression testing is • Painful • Boring • A waste of time • “Let’s get a tool in” • Automating 10% of tests is easy • Automating 90% of tests is hard (impossible?) • “Why are we doing this?” • “Let’s do it manually” Assurance with Intelligence

7. Let’s look at each thought process Assurance with Intelligence

8. (What is...) regression testing? Slide 9 Assurance with Intelligence • When a software fault is fixed, it often happens that 'knock-on' effects occur • We need to check that when changing the faulty code, the developer has not introduced new faults • There is a 50% chance of introducing regression faults • Regression tests tell us whether new faults have been introduced • i.e. whether the system still works after a change to the code or environment has been made. • When environments are changed, we might also regression test.

9. Regression testing 2 Slide 10 Assurance with Intelligence “Testing to ensure a change has not caused faults in unchanged parts of the system" Not necessarily a separate stage Regression testing most important during maintenance activities Regression tests are usually the first to be considered for automation.

10. How do regressions occur? • Something changes and affects ‘working’ software • An enhancement is implemented • A bug is fixed • Two bug fixes interfere with each other • One programmer overwrites another programmer’s fix (configuration management) • Something in the environment setup • Test data changes in some way • A test changes in some way • Test configuration/setup changes • Etc. etc. • Can you think of others? Assurance with Intelligence

11. Code fixes have a 50% chance of introducing side-effects in working software Ancient history – who cares whether it’s 10% or 90%? Regressions have a disproportionate impact on effort, confidence, morale Assurance with Intelligence

12. Combating regression - choices • Several options: • Don’t change anything! • Impact analysis (of code/data//test changes) • Regression testing (to detect unplanned changes) • Impact analysis is tricky; not 100% reliable but... • Compare source code – see what’s changed • Module dependencies – see what’s affected • Rigour in managing environments and data • Rigour in managing source code • Regression testing choices • Manual v automated • Test first or test last • All tests or a selection of tests • Others? Assurance with Intelligence

13. Who are our stakeholders? • Stakeholders are people interested in the outcome of test • What do they want? Ideally, nothing! • Sponsors, customers • Regressions are an immediate setback/delay just when they thought you could move the project forward • Users and testers • “All our good work is undermined!” • Developers • They’ve missed something obvious, embarrassing; unplanned work – could be trivial or substantial • Tech support • “Users are going to be furious!” • “We feel really let down by dev, maintenance, testers” Assurance with Intelligence

14. Context of regression testing • Test- or behaviour-driven development: • All tests are automated; these are our safety-net • Tests allow us to refactor with confidence • Integration/System/Acceptance testing 1st pass • RT after main testing done – confirmation that all tests pass with all fixes in place • Maintenance phase • Minor fixes (don’t get impact analysed) • Enhancements fit with existing system • Refactoring/conversion/interface changes. Assurance with Intelligence

15. Regression test policy • Common: • Business critical, high risk, error-prone areas • FIFA – first in, first to automate? • LIFA – last in first to automate • Easily automatable • But let’s step back a little • Focus on stakeholders • What do THEY actually want? Assurance with Intelligence

16. What would stakeholders want? • Sponsors, customers • Want to avoid delays • Is the most critical functionality working? • Tech support • Are we inconveniencing the users? • Is the most critical functionality working? • Users and testers • Can we trust the results of our early (high priority tests) • (Maybe we aren’t so worried about the rest?) • Is the most critical functionality working? • Developers • Want instant feedback (so they can fix with least hassle) • The best case for them doing their own automated tests! Assurance with Intelligence

17. If a developer ran your test team... • Whose problem is regression anyway? • It’s developers’ problem not the testers’! • Dev have been asked to fix/enhance/refactor code without screwing up • How would a developer address the regression problem? • Design a harness and code to execute transactions at the API level • Easier, more maintainable, less sensitive to cosmetic changes • A reusable asset that allows refactoring. Assurance with Intelligence

18. The real problem to solve? • If the problem is identifying change where there shouldn’t be any... • With a few caveats, it doesn’t matter what tests we run as long as we cover the important functionality – and that it behaves the same way as it did last time • We still need to demonstrate the user interface isn’t adversely affected and the end-to-ends still work • Is it POSSIBLE and BETTER to do this manually? • Why use a framework on top of a test execution tool on top of a browser to test server-based code? Too complex! • Where does that leave automated test execution – on the shelf? (It’s probably there already!) Assurance with Intelligence

19. Assurance with Intelligence

20. Techniques for Testing GUI Applications Eurostar 1997: the most popular paper on my website is referenced by Wikipedia http://gerrardconsulting.com/sites/default/files/Techgui.pdf Won’t repeat the whole story here... But most of the 1997 slides are still relevant Have we made NO progress since 1997? Assurance with Intelligence

21. Selecting what regression tests to automate 1997 Assurance with Intelligence

22. Manual v automatic scripts - very rough guideline

23. What aspects to automate? • Pareto law: • go for the easy scripts first • don’t waste time scripting low volume complex scripts at the expense of high volume simple ones • Consider hybrid approach: • use the tool to do navigation and data entry • do the checking on-line or at end of test.

24. Automated test scripts • Coded scripts • navigation through the system, some checklist test cases can be can be coded directly • relatively easy to maintain • Recorded scripts • automated scripts should be documented • then used to record the automated script • less easy to maintain • Integrate all scripts into automated suite.

25. Migration of manual scripts • Automated scripts are easier to maintain • if designed to be maintained • Means you have to build test automation scaffolding before you script tests • Automation projects are software development projects, not test projects • Allow for maintenance cost before deciding to migrate functional scripts • delay migration until system is stable • soak and regression tests, as required.

26. Conclusion • The regression problem is a developers’ problem that can be solved by: • Developer discipline • Better impact analysis • Automated testing at an API level • Manual/automated testing (critical/E2E level) for visibility to stakeholders • Is this what the big product companies do? Assurance with Intelligence

27. Regression Testing – What to Automate and how Paul GerrardGerrard Consulting1 Old Forge CloseMaidenheadBerkshireSL6 2RD UK e: paul@gerrardconsulting.comw: http://gerrardconsulting.comt: 01628 639173 Assurance with Intelligence