Evolution of Structured Exploratory Testing at Philips Ultrasound

1. Evolution of Structured Exploratory Testing at Philips Ultrasound Doug Carlton and LeAnn Coker QASIG Presentation January11, 2006

2. Philips Ultrasound Philips Ultrasound is a subsidiary of the Dutch conglomerate, Philips Royal Electronics. • Consumer electronics • Lighting • Semi-conductors • Medical devices Philips Ultrasound has facilities in Bothell WA, Andover MA, Reedsville PA, Bangalore, India and Best, Netherlands for the research, design, development and testing of ultrasound imaging systems. Ultrasound medical devices must be SAFE, EFFECTIVE and RELIABLE

3. Scripted versus Exploratory Testing Scripted Testing • Using approved requirements, scripted tests are designed and formally documented. • The test scripts are executed at some later time (after the Verification Readiness Review at Philips). • Scripts may be executed by someone other than the original test designer. Exploratory Testing • In exploratory testing (ET), tests are designed and executed at the same time. • Quite often, test cases are not documented. Is one approach superior to the other or is it better to do BOTH?

4. The Testing Problem…. • Compliance with FDA’s Quality System Regulations (Federal Law applied to R&D of medical devices) mandates formal verification and validation techniques, with documented testing as a key activity. • Developing medical devices in today’s highly competitive marketplace requires cost-effective methods. Late defect discovery and re-design is very costly. • The formal release of requirements takes time, resulting in an inherent delay between documented test script generation and execution. • While defects may be found, formal Verification and Validation (V&V) does not emphasize or demand defect discovery. Formal V&V demonstrates that the medical device functions as specified in the intended use environment. (positive test cases) How do we identify the most important defects EARLY enough to get them fixed?

5. Our Solution: Structured Exploratory Testing • Structured exploratory testing may sound like a contradiction-in-terms, but "structure" does not mean the testing is pre-scripted. It means there are constraints on how the work will be performed and reported. • Early defect discovery requires process agility and tester freedom to hammer away on functionality with the intent of breaking the system-under-test. • By generating negative, stress-load and feature-to-feature interaction test cases, structured ET finds defects that traditional methods may miss. • Structured ET operates within a formal process that documents defects while providing metrics and reports to management. • Metrics and reports enable upper management to objectively evaluate the ROI of exploratory testing. Structured Exploratory Testing COMPLIMENTS the formal V&V!

6. Doug’s ET Roots • 1997 STARWEST Conference: James Bach presents “Good Enough Testing for Good Enough Software”. "I'm going to let the steamroller of reality explain this to you." • 1997 Neopath, Class III medical device manufacturer in Redmond, I asked the software dev manager to hire a full-time tester into an established software dev group specifically for testing early in product lifecycle..successful prototype but company went under! • 1999 My manager and I established the Software Integration Test (SIT) team within the software development group. This team was very successful at finding bugs, but eventually disintegrated. • 2000 STARWEST Conference: James Bach presents “Getting the Most Out of Exploratory Testing”, I knew this was the right approach, I was now more determined than ever to establish a dedicated exploratory testing team! • 2005 I was asked to manage the SIT team remnant. Started with experienced ET lead, swapped a rookie for a seasoned V&V veteran, lost a super-star tester to motherhood, drafted an Andover veteran, acquired an established ET all-star, Jon Bach provided “Session-based Testing” training to new team, and… Dedicated Structured Exploratory Team Established!

7. LeAnn’s ET Roots • 2000 I joined the Quality Assurance team at HealtheTech, a class II medical device company. This was my first experience where we were required to run test cases but also utilized ad hoc testing. • 2002 While on contract at a local company, I often stopped to read articles taped outside of someone’s office. The office turned out to be Jon Bach’s and the articles were some that he authored. This is the point that I could put a name to the testing that I enjoyed. • 2003 Thanks to Jon Bach’s referral, I joined the SIT Team at Philips Ultrasound where we performed scripted regression testing of daily builds, as well as, limited exploratory testing on new system features. • 2004 The Systems Engineering Group approached me to lead a proposed test activity that would focus on the interactions between ultrasound system modes and features. This activity became the foundation for the structure of our ET activities. • 2005 I became the lead of the newly established Integration Test (IT) Team and authored our new Process for Integration Test.

8. Integration Test (IT), what is it?

9. Integration Test Team Organization

10. Product Creation Process Phase III - Design & Implementation Process PhasesRisk Based Test PhaseFeature Interaction Test Phase

11. Risk Based Test Phase • Risks are two fold; safety hazards inherent to the use of the device and business risks such as functionality critical to sales. • Composed of System Test Sessions, Test Targeting Sessions and Focus Test Sessions. • Executed on a weekly basis, typically over many months. • Automated Tool used by both RBT and FIM phases. • Starts after the first Design Input Review and ends when the system and software achieves the Implementation Complete milestone.

12. Feature Interaction Matrix Phase History The first matrix test activity came out of the realization that we were doing a good job testing individual features but not enough thought was being put into how each feature interacted with other features in the system. • The matrix contained 276 feature interactions. • Each cell in the matrix represents an interaction to be tested. • Each cell in the matrix is assigned a risk rationale and value. • Each interaction has a corresponding data tracking sheet. • First FIM testing had 20 participants and identified 325 bugs. • Tracked progress in Excel – OVERHEAD NIGHTMARE!

13. Feature Interaction Matrix Phase • As a result of the number and quality of bugs found during the first matrix test activity, program management requested that we continue this activity for other software releases. • FIM starts when the system achieves the implementation complete milestone. • 100% of cells in matrix are tested. • Cells are weighted by risk (assisted by automated tool).

14. Feature Interaction Matrix Tool

15. Summary of Structured ET Evolution • The Bach brothers have made MAJOR contributions to the evolution! • Structured ET means there are constraints on how the work will be performed and reported not on how test cases are designed! • Structured ET produces metrics and reports that enable upper management to objectively evaluate its effectiveness. • Safe, effective and reliable ultrasound systems require a combination of exploratory and scripted testing. • Formally establishing structured ET will require a vision, champions, endurance, patience and time. ANY QUESTIONS ?

16. Contacts • Doug Carlton – doug.carlton@philips.com 425-482-8330 • LeAnn Coker – leann.coker@philips.com 425-908-2064