How to Eliminate Manual Feature Checking

1. How to Eliminate Manual Feature Checking @paul_gerrard Paul Gerrard paul@gerrardconsulting.com gerrardconsulting.com

2. Overview Checking Scope of Feature Checking Why Late, Manual Checking is Bad for Your Team and Your System Redistributed Testing How to Automate Feature Checking Summary Q&A Intelligent Definition and Assurance

3. Checking

4. What is check(ing) versus test(ing)? • There’s an ongoing debate about what checking is • Here’s my selected definitions of two words: • Test: A procedure for critical evaluation, a means of determining the presence, quality, or truth of something; a trial American Heritage Dictionary This is what I used in the testers-pocketbook.com • Check: A control to secure accuracy; a test or examination to see that something is correct or in good working order Oxford Reference Dictionary Intelligent Definition and Assurance

5. A check is… • A control to secure accuracy • A comparison with a required outcome (derived or computed) that tells us we align with some plan or intent • A test or examination to see that something is correct or in good working order • Correct – compared with some source of knowledge (a spec, a person, experience, standard or convention) • In good working order – aligned with someone’s view of how a system ought to work, in some context. Intelligent Definition and Assurance

6. Checking dominates manual testing By ‘manual’ we mean that the tester thinks their way through a test, but uses their hands (rather than tools) to drive it Most testing is actually checking (regrettably) Mechanical, procedural, pre-meditated process Performed by humans it can be error-prone Why do we ask human beings to perform an activity that is (obviously?) better done by tools? Intelligent Definition and Assurance

7. The shift to Agile is irresistible Agile approaches – BDD/TDD, user stories, multi-skilled teams, short iterations, rapid feedback, continuous integration, continuous delivery etc. are creeping into all projects Automated developer testing (BDD/TDD) Manual testing at the end is being squeezed Testers may have to ‘let go’ of manual checking and embrace redistributed testing Will automated checking be mandatory in future? Intelligent Definition and Assurance

8. Scope of Feature Checking

9. How to identify a feature • A software feature has these characteristics… • Meets a particular need of a user and/or be accessible to them (a screen, transaction etc.) • Provide a technical capability and be implemented as a function, method, class, module, unit • Can be described in isolation – like a component • Can be tested in isolation • Tests (checks) can be specified in advance. Intelligent Definition and Assurance

10. Scope of feature checking • Behaviour can be derived (from a requirement or conversations) in advance • A scenario can be defined in a systematic way: • Pre-conditions (an initial state, setup or preparation) • (Steps) inputs, commands or other stimuli • Post-conditions (outcome(s), outputs, final state…) • Similar to a traditional test case • Execution and comparison of actual to required behaviour comprises the check • DeFOSPAM is a method for deriving good scenarios • NB – checks focus on a selective subset of observable behaviour. Intelligent Definition and Assurance

11. Structured stories (other variations exist) Story Header • Feature: ship orders • As a orders clerk • I want to acknowledge and ship the order • So thatwe fulfil a book order • Scenario: ship a single book from stock • Given I select a valid order • And the ordered book is in stock • When I choose ‘acknowledge and ship’ • Then order status is changed to ‘shipped’ • And an address label is printed PRE-CONDITION(S) STEP(S) POST-CONDITION(S) Key word Story text Each Story has multiple Scenarios Scenarios can be data driven DeFOSPAM – Google it Intelligent Definition and Assurance

12. Success or failure • A feature succeeds (passes the check) if: • The pre-conditions are achieved AND • The system accepts the inputs, commands or other stimuli AND • The required outcome(s), outputs or final state are observed • A feature fails (fails the check)if: • The pre-conditions cannot be met, rejects or does not request the inputs, or the required outcomes are not observed. Intelligent Definition and Assurance

13. Why do checks fail? • The requirement was vague • Developer made bad assumption and user/tester ‘knows better’ • Or the tester made the bad assumption • System tests more thorough than dev tests? • Greater coverage, more focused, deeper checking • Risk-based approach • Deeper insight into user behaviour. Intelligent Definition and Assurance

14. Why Late, Manual Checking is Bad for Your Team and Your System

15. The usual process? Initial user Idea Requirement or story Refine Requirement Refined requirement Rework Initial Development Development Rework Initial Testing Re-Test Rework Diagnosis, discussion etc. Most rework is due to poor requirements than poor implementation Intelligent Definition and Assurance

16. Wasteful development We take a stab at the requirement, believing that users don’t really know what they want Developers make assumptions, and so do the testers (sometimes different) When software is available we show the users and they say, “that’s great, but what I really want is…” How wasteful! Intelligent Definition and Assurance

17. Validating requirements with examples • Wasteful: ask developers to create some code for users to critique the REQUIREMENT • We use developers to create this ‘straw man’, ‘prototype’, ‘starter for 10’ • The result is a new requirement and some code that doesn’t work • Instead, why don’t we create some proposed examples of the system in use? • It can encompass the same assumptions, proposals and errors as the straw man • Our users need just a little more imagination to validate the requirement • We achieve a refined requirement faster, cheaply, because we give users concrete examples and don’t ask developers to guess. Intelligent Definition and Assurance

18. Late feature checking by testers is bad • Devs have a safety-net so they test less/test badly • Testers do even more checking – a vicious circle • Testers join teams late, stakeholders not engaged early • Code and tests based on faulty, mistrusted requirements • Poor tests raise non-problems and miss real problems • Bigger test teams, lots of bugs, lots of rework • Testers shortcut and put checks in long, brittle, unmaintainable end-to-end test scripts • BTW, these tests could be manual OR automated. Intelligent Definition and Assurance

19. Redistributing Testing Checking

20. If a requirement drives today’s delivery it must be trusted Organisations demand more rapid and reliable delivery Continuous delivery is a hungry beast that eats requirements Requirements must be trusted to be mature, complete and coherent enough to deliver the business value envisaged by stakeholders TRUSTED, NOT PERFECT Intelligent Definition and Assurance

21. Redistributed testing (checking) Functional Requirements Build (+little testing) but lots of rework Testing (lots of checking + bugs) Trusted Requirements + Stories BDD/Test-Driven Testing SAVED TIME Trusted Requirements + Stories BDD/Test-Driven) Testing Intelligent Definition and Assurance

22. How to Automate Feature Checking

23. Redistributed = story-driven Output: Trusted requirements Requirement Validate Requirement Initial User Idea Stories Test/Behaviour -Driven Development Output: Continuously Checked code Less/no checking, better tests, fewer failures Testing (little checking) Intelligent Definition and Assurance

24. Varying levels of feature checking • ‘Key Examples’ • To verify that team understanding of the requirement is correct • Feature unit tests • To feed test-driven development • Feature acceptance criteria • To provide evidence so a feature can be accepted • Feature system tests • To ensure a feature is tested thoroughly • Integrated, end-to-end, user tests are really important • But they aren’t feature checks. Intelligent Definition and Assurance

25. Key tool requirements • Traditional RM tools don’t use stories • BSM • BDD tools • BSM • Integrated Requirements and Stories • Test code generation from stories Emerging tools support the left hand quadrants BDD tools support the right hand quadrants • Test Execution Automation • Execution feedback and reporting • BDD tools • xUnit • Other test frameworks • BSM • Custom interfaces required Intelligent Definition and Assurance

26. Business Story Platform Robot Framework Requirements Dictionary Keyword Definitions + Test Cases Story (structured) Test Export Test Outcomes System Under Test Test Library e.g. Webdriver executes tests • Test Status • Red • Amber • Green Write the Code + Refactor the Code Intelligent Definition and Assurance

27. Business Story Manager and Test Frameworks Business Story Manager STORY: Feature Header Requirement Business Scenario Given“Search page” When “I find a book” And “I add to basket” Then “basket has my book” Just enough to clarify understanding Test Scenario Pre-conditions “Search page” Steps “enter search text” “click search button” “scroll down to find book” “select book” “view more information to verify” “select add to basket” Post Conditions/outcomes “Basket has selected book” Manually maintained Generated Technical Code Per Environment Keyword Definitions 1 per application Keyword Definitions 1 per Feature Story Based Test Cases Test Config. Test Log Example: Robot Framework Drivers System(s) Under Test

28. Four execution automation choices Maintained code Generated code Current BDD Tools Step Definitions BDD Tool Steps Application API calls xUnit Test code Story Scenario Test Export Keyword Definitions/Library Calls Test Framework Keywords Proprietary Exec. Tools GUI Script Function Definitions GUI Test Tool Script

29. Four execution automation goals Goal Generated code Developer Testing BDD Tool Steps Developer Testing xUnit Test code Story Scenario Test Export System Feature Checking Test Framework Keywords Business Story Manager: interfaces for all four paths System Feature Checking GUI Test Tool Script

30. Getting started • Get testers to create (stories and) scenarios/acceptance criteria - EARLY • They would create test cases anyway • Give developers trusted requirements, stories and scenarios • Developers to adopt BDD/TDD • Testers: • Adopt Acceptance-Driven approach • Implement automated checking Intelligent Definition and Assurance

31. Summary Improved requirements ‘by example’ Easier route for developers to adopt BDD/TDD Automatable feature test cases and reduced demand for feature-level checking in system test Fewer testers, or testers released to do more end to end and exploratory testing Trusted, ‘live specifications’ for the lifetime of an application. Intelligent Definition and Assurance

32. Survey

33. References • Our THINKING: Business Story Method(overall method, DeFOSPAM etc.) • http://businessstorymethod.com • A Unified Approach to System FunctionalTesting (1993) • Testing Requirements (1994) • Our TECHNOLOGY • http://businessstorymanager.com • Free Story Platform for QA (a subset of BSM) • http://sp.qa • If you want to know how to implement the Business Story Method or tool – do get in touch Intelligent Definition and Assurance

34. How to Eliminate Manual Feature Checking @paul_gerrard Paul Gerrard paul@gerrardconsulting.com gerrardconsulting.com