Principles of Software Verification and Validation for Medical Imaging

1. Principles of Software Verification and Validation for Medical Imaging Twin Spin October 7, 2010

2. Topics Short history of medical imaging How SW “Changed the Picture” Challenges and Solutions to Software Verification • 4 Areas of image trustworthiness • Importance of SW Engineering Principles Product Software Validation Summary

3. History Röntgen discovers X-rays in 1885, receives Nobel Prize in 1901. Rapid research and discovery leading to working prototype x-ray imaging system by Edison in 1901 Other modalities follow rapid development in 20th century.

4. Boom in x-ray-associated marketing.

5. Medical Imaging Today Multiple Modalities • Electron microscopy • Radiographic – standard x-rays, fluoroscopy • MRI • Nuclear medicine – PET, gamma • Thermography • Tomography – CT scans • Ultrasound • Photoacoustic imaging - lasers + ultrasound.

6. Medical Imaging Today

7. Demonstration • 2D images on e-film portable reader, courtesy my CT scan. • 3D, 4D Images from Osirix open source reader. Courtesy

8. For purposes of understanding these principles, this talk will focus on CT technology.

9. How Software “Changed the Picture”Yesterday: Hardware config and control Hardcopy output (film) Manual archiving Limited review Hardware positioning

10. Today: Digital raw data Dedicated digital signal processing Software patient positioning control Software config and control console DICOM data output Transmission over enterprise network Archival File Server (PACS) Advanced visualization workstation PACS reading station

11. The essential question: Can I trust these pictures?

12. To help answer the question, we ask two more questions: Q: What is essential in the image? A: This is validation. “What is the right information?” Q: How much error does the essential information contain? A: This is verification. “How do I insure the information is as error-free as possible?”

13. The essential question changes: From: Q: Can I trust these pictures? To: Q: How much can I trust these pictures?

14. Answering Question 1: An Art Lesson Visual artists have known that images are an interpretation of reality and exploit that fact to convey essential messages. Medical imaging is an interpretation of reality too, intentionally distorted through reconstruction algorithms to convey essentialdiagnostic information (i.e. right information). Interpretation involves subjectivity. Boy or girl?

15. How to create trust in a subjective process? A: Experience Through certification, advanced education, on-the-job training, etc.

16. Product Validation • Validation involves the customer or it isn’t validation. • Validation of images by medical imaging professionals is absolutely vital: radiologists, CT/MR technologists, etc. • Validation can be done at nearly all steps in the development. • Beta field phase absolutely essential. • Some issues can only be validated • Human factors: presentation of on-screen information, segmentation preferences, image fidelity preferences. • Visualization of small vessels or structures whose phantoms are too costly.

17. Product Validation Approaches • Internal panel of expert employees – training team, field application teams. • Internal panel of expert consultants – medical advisory board, focus groups. • Luminary sites willing to partner in product development. • Industry-acknowledged body of knowledge. Walter Reed colon datasets, Stanford bake-off.

18. Answering Question 2: Dataflow error Reconstruction algorithmic error “false images” Modality quantization, noise error User error Data transmission protocol error Database error 3D processing, user, & algorithmic errors User error, visual errors

19. Error in the dataflow.

20. The technical effort simplifies to reducing the impact of undetected functional error on the image to be less than or equal to the impact of inherent error on the image SO THAT the images and data derived from the images are trustworthy. Any “questionable” image abnormalities visual artifacts, or other deficiencies are due to laws of physics inherent in the modality.

21. The four areas for an image to be trustworthy from the user perspective: • Image orientation • Image fidelity • Measurements • Data integrity

22. Workstation error in the end image.

23. How to verify Image Orientation Manufacture an object of known dimension, HU values, orientation Scan multiple times Check the result The reference library of orientation datasets is ideal for regression automation.

24. Phantom manufacturers • http://www.universalmedicalinc.com/diagnostic-imaging/imaging-quality-control/phantoms/ct-phantom • http://www.phantomlab.com/rsvp_head.html • http://www.cirsinc.com

25. How to verify Image Fidelity • Use reference datasets • Phantoms • Synthetic data Check the result against reference image using image pixel checker DICOM xfer Problem: subtle differences due to video driver revs cause false failures. **Error below threshold of human eye** Tools must not flag errors that are not noticeable (or relevant) to the user. The sample image must be transferred through a daisy chain of imaging workstations to simulate the enterprise environment. Reprocessing the image can, in rare cases, lead to image degradation. User acceptance panels are another tactic – often for selecting default color tables.

26. How to verify Measurements • Use reference datasets • Phantoms • Synthetic data Check the result against reference measurement DICOM xfer In some cases it is necessary to manually complete a typical patient “workup” or workflow, transfer the patient record to a second workstation, and verify the measurements maintain consistency in data transfer & processing. All manual measurements are, by their nature, subject to human error. Define +- bounds

27. How to verify Data Integrity • Use reference datasets • Phantoms • Synthetic data • Large, small, • Multiple modalities • Load testing • Negative testing DICOM xfer DICOM Database Check the results at the database, not the user interface. We assume the user interface & visualization do not corrupt the data (it is prudent to verify this assumption if using this strategy). Data transfer testing is executed in isolation and in concert with orientation, fidelity, and measurement verification.

28. Other things that affect trustworthiness • Reliability • Stability • Security • Installation & Upgrades • System integration • Localization • Licensing • Performance • Manufacturing & distribution • Etc.

29. In other words…. Very few of the verification factors for trustworthiness are medical imaging-specific. Most are core software engineering principles and good quality engineering. • Good requirements development and management • Good code development and management • Good test case development and management Following good (not even best) practices automatically generates artifacts that audit agencies look for as proof of regulatory compliance.

30. Example of SW Engineering lapse. GE Healthcare, August 2008. Optovue, 2010

31. Risk-based verification • Everything can’t be tested • Organizations inevitably take a risk-based approach whether they know it or not. • “Good enough” by instinct to begin • “Good enough” by systematic classification by end. • Cross departmental effort to classify risks (patient risks, business risks) • Living document of decision-making as-you-go. • Complies with regulatory risk assessment deliverables. • High business value.

32. Risk-based verification derived from Business Needs/Risks All risks are secondary to patient risk. Recall vs. patch Design Mitigation Business Need: (product will not harm patient) Project Risk Assessment (schedule, market, patient hazards, etc) Unit testing Business Need: ( ) Business Need ( ) Risk-based system testing

33. Summary • The key to medical imaging is trustworthiness of the image – even with known error sources. • Verifying medical imaging software is an engineering task approaching image orientation, image fidelity, measurements, and data integrity • Validating imaging software is a clinical user task overlapping with human factors and artistic questions of interpretation and presentation. • All other challenges are familiar core software engineering problems.

34. Biography Alex Dietz has been applying product quality verification principles for 20+ years in telecommunications, data transmission, and medical imaging. He currently manages the Software Verification and Validation team for the EnSite cardiac mapping system at St. Jude Medical. He has spoken locally and nationally, most recently at the Software Design for Medical Devices conference in San Diego. adietz@sjm.com

35. References History • Naked To The Bone: Medical Imaging In The Twentieth Century by BettyanneKevles • http://en.wikipedia.org/wiki/X-ray#History Phantom Manufacturers • http://www.phantomlab.com/ • http://www.universalmedicalinc.com • http://www.cirsinc.com