Agile Test design & Automation of a Life-critical Medical Device

1. Agile Test design & Automation of a Life-critical Medical Device • Christian Nørlyng & Thomas Kauders The Copenhagen model of agile test automation

2. NY SEKTION • Introduciton – 2 slides

3. About us • CHNQ (Automator) • THKD (Architect) • MHUG (Primitives inventor)

4. Project summary • 700 Man-years • Over 11300 pages of specifications/plans

5. Perspective… Development time…

6. Perspective… Documentation…

7. NY SEKTION • The Challenge – 3 slides • Complex • Life critical • Volatile • Formal

8. Our challenge TESTING

9. Project Challenge • Changing requirements (R&D) • Software developed from unstabilised requirements • Complex, life-critical application • Unhappy management and frustrtation over all

10. Testing Challenge • Dynamic development environment • Stringent Validation and Verification criteria • Regular releases – that need to be tested • Small testing team • Manual test taking one full resource • just for regression • Automated test running on unstable test system • ”Self developed ad-hoc” • Automated test in maintenence over flow • due unstable test system and dynamic requirements

11. We had to do something!! • Final Test could not • be reachable without test automation for more regression. • be reachable without better manual regression testing. • be reachable without a smart way to structure test ware ”production”

12. Not reacheble without automated testing!!

13. The Team • 1 (newly arrived) test manager • 1 test architect • 2 manual testers • 1 test automateur • 1 model test automateur

14. NY SEKTION • The answer – 5 slides • Process descirption (upside-down process) • +Exploratory testing

15. Our answer The ”Copenhagen model of test automation”

16. Main characteristics • Cooperation of Test architect – Manual test team – and Automation test team • Reuse of test artefacts (TD-MTC-ATC) • Direct 1:1 tracebility from Req TD MTC ATC • Thinking automation into the process from the very beginning of testware production. • Granularity in TD/MTC/ATC using Primitives!

17. Meet the Primitives • System partitions, atomic level • Derived from requirements • Building blocks of TC´s • Reusable across TC´s • Extensible • Maintainable • 3 types: • Navigation primitives • Action primitives • Verification primitives

18. More about primitives…

19. Product of af module based test Normal Sequential test case Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Same but more maintainable and easing automation

20. The Concept Autmated Test TestCases possible to automate Primitives / Modules Analyzed GUI/MMI BuildUp TestCases Structure Modular mindset during test design Domain knowlegde Test Design Structure Requirements

21. Implementation Modular mindset during test design Requirements Review Test Design Using Primitives Review Manual testcases usingPrimitives Review Automating Manual testcases, Using Primitives

22. Process • Req in QC – Req module • TD in QC – Req module • Linked Req+TD • TD using primitives • TC in QC using primitives • Linked TD+TC • Automated TC using same primitives

23. Structured testing is NOT enough • The structured test effort MUST be complemented by a lightweight (Session based or Exploratory) testing process parallel to the structured one.

24. The Copenhagen model is not... It is not the solution: • For all automation tasks • ? • ? • ?

25. NY SEKTION • Real life (Screenshots) – 10 slides • Req/MMI, TD, Prim, Testplan, TestLab, Autx5

26. A live example of using the prims • Locally installed QC example - LIVE • Change 2 reqs • Change 3-4 TDs – mTCs, - aTCs

27. Tooling • ReqPro • CQ • CC • HP QC • Excel

28. The structure of the app • The application is structured acc. to a number of functional areas (subsystems) e.g. sms/contacts/dialing/set date/set alarm/ snooze alarm etc. • Each subsystem has own requirements

29. SRS

30. Test design using primitives

31. The primitives

32. Test cases – built by primitives

33. Test Lab view

34. What happens when we change a req? • Show tracebility from Req- to TC´s involved. • Change the Primitives involved • Show that all TC´s where primitive is used change ”in one go”.

35. NY SEKTION • Results – what we achieved – Experiences - 5 slides • Man-Aut test cooperation (re-use of resources), Granularity, Flexibility, Tracebility,

36. Result • Running 7-14 days of automateded regression every night • 7 days september • 14 days december?? • Improved effensiency • Team consists of • 6 testers and 1 manager • Every team member has an important role. • Happy Test! • Ackknowleged by project • Reacheble dead line for final test!!!!

37. What it solved • Full tracebilitytells you what to change • 1:1 tracebility gives quicker validation • Primitives make the test suite flexible

38. Why this works • When analyzing, break down till identifyable unique user action sequence parts. • When a sequence is unique, it is not present any where else in the system. • These sequenses can be put together in any order representing a complete user action

39. NY SEKTION • Pitfalls – 5 slides • Lag behind development, Heavy process, disciplin (in re-use of primitives)

40. What it did not solve • There is no quick fix for the analysis – change in req – requires re-analysis of the req and redesign of TD.

41. Pit falls • Not review with the right mind set! • This seems possible! • But is it? • M & A testsnot revied • Test Script base not reviewed • Break down not done proper • Break down done to proper • Will also cost time and effency • No ”Cheif of quality” in command of test ware base

42. NY SEKTION • Summary