Validation, Verification and Quality Control

1. Validation, Verificationand Quality Control Thomas Novicki Ph.D. D(ABMM) Marshfield Clinic Marshfield WI

2. Today’s Topics • Validation and Verification of ID and AST systems • Quality Control (QC) of Identification (ID) and Antimicrobial susceptibility test (AST) systems (We will not be discussing proficiency testing due to time constraints)

3. Disclosures • I have nothing relevant to this presentation to disclose

4. Validation and VErification

5. Validation and Verification • Validation: Documenting in vitro diagnostic device performance in your lab • Verification: An ongoing demonstration of the vendor’s performance claims using a variety of tools

6. Validation and Verification(or is it Verification and Validation…?) • Note that some entities (e.g. CLSI, CLIA) reverse the definitions • Initially Verify • Then periodically Validate The definitions used don’t matter, but their consistent use in your lab is important. What does your accrediting body use?

7. Who Guides Us? • Vendors of commercial systems • The Clinical and Laboratory Standards Institute (CLSI) • The Feds • CLIA – Clinical labs • FDA – Instruments • Your accrediting body (COLA, CAP, JC) (Did I miss anything…?)

8. Validation and Verification The Centers for Medicare and Medicaid Services (CMS), (a part of the US Department of Health and Human Services) regulates clinical laboratory testing through the Clinical Laboratory Improvements Amendments (CLIA) that became effective in 2003. (Recall that it all began in 1988)

9. Validation • For FDA cleared and approved tests, the lab demonstrates performance claims by the device manufacturer, typically found in the product label (‘package insert’) • For lab-developed tests (LDTs), a more rigorous set of studies demonstrating performance is necessary

10. Validation of LDTs • Will not discuss further except to say that any modification of an FDA IVD makes it an LDT • Be wary of modifying the more critical FDA-approved test (e.g. TB, HIV IVD devices ) • Qualitative tests easier to modify than quantitative • Simple cleared tests (e.g. MRSA chromo-agars for additional anatomical sites) are reasonably modified

11. iClick? 1 • CLIA requires a lab to validate which of the following for every new diagnostic test • Accuracy • Precision • Reportable and Reference ranges • All of the above • 1 and 2

12. iClick? 1 - Discussion • CLIA specifies that the user of an FDA IVD device of Moderate/High complexity will • Validate accuracy, • Precision, • Reportable range, and • Reference range • You must satisfactorily complete validation of the device before beginning its use. • You must document all validation study data and maintain it for at least 2 years after test retirement.

13. Validation – AST • Two methods for an unmodified FDA-cleared instrument • Compare to a reference method • Compare to an existing system • Method B probably the best fit for most labs with an existing system For method A, see Cumitech 31A (Cumitech 31A, Verification and validation of procedures in the clinical microbiology laboratory, ASM Press 2009)

14. DisclaimerWhat I’m about to say on validation… ‘… is more what you’d call guidelines than actual rules’

15. Validation – AST • Source of samples/isolates to test • > 30 well-characterized clinical isolates per panel that reflects the mix common to your population and also some with unusual AST patterns • Can include proficiency or commercial validation panels, but be wary: • Reflective of your population? • Incorrect sample matrix? • If from the device manufacturer, a self-fulfilling prophecy? • Note that with care a single panel can be used for both AST and ID validations

16. Validation – AST How to deal with discrepancies when two non-reference systems are compared?  No reference method, so grade results, then apply predetermined limits on the number and types of discrepancies

17. Validation – AST For each bug/drug result, calculate %s for… • Categorical (S/I/R) Errors • Minor Error I vs S or I vs R • Major Error S vs R or R vs S • No Very Major Error category: ‘True’ result is not known • Essential Errors (> + 1 MIC dil. difference) • Can only score when both methods generate discrete numbers (i.e. do not score < or > values)

18. Validation – AST Using the total number of evaluable bug/drug data points, an acceptable validation has: • < 5% Major Errors AND • < 10% Major + Minor Errors AND • < 10% Categorical Errors AND • < 10% Essential Errors Also look for trends in a particular drug or bug which may point towards a problem area Consult your vendor rep with any validation problems

19. Validation – AST • Precision (reproducibility) • Test five isolates X3 for three to five days • Calculate intra-and inter-run categorical and essential % agreements • Isolates should have known susceptibility profiles, such as ATCC strains • Use both Gram positive and Gram negative strains if validating GP and GN panels • Acceptable precision > 95%agreement

20. Validation – ID Test a range of Gram positive and Gram negative organisms ID’d by reference or current method(s) Minimum of 20 patient isolates; larger labs test more. Additionally, test QC strains • < 10% discrepancies between reference and new IDs

21. Validation – ID • Level of agreement may need to be adjusted e.g. • Old ID of ‘Coag-negative Staphylococcus sp’ • New instrument ID’s to species level • Precision • Test 2-4 isolates X3 for three to five days • Calculate % intra- and inter-run agreements • Accept > 95% agreements

22. Validation • What to do if validation fails? Do not use the new test and either… • Withdraw test from further consideration OR • Develop and implement a corrective plan with the manufacturer. Then, re-validate

23. Verification – Component of the QA Process An ongoing, multi-faceted process that assures the validated test continues to perform at the expected level

24. Quality Control I think, therefore I control quality

25. iClick? 2 You have an instrument that uses micro-broth dilution MIC plates in your lab. This system incubates and reads plates automatically but also has a manual plate reader. Your tech rep tells you to read a plate manually if QC fails on the instrument. If QC then passes, your QC is acceptable for the entire systemand no follow-up action is needed. • True • False

26. iClick? 2: discussion QC is performed to assure the accuracy of a test result by assessing all analytical components • Reagents • Instrument (where applicable) • Test precision (i.e. repeatability) • Operator technique • Data interpretation

27. CAP on AST QC ‘For AST of either disk or dilution type, control organisms are tested with each new lot or batch of antimicrobials or media and each day the test is performed thereafter.’ (Note that >3 failures per month per drug/bug combo is unacceptable)

28. CAP on AST QC ‘However, the frequency of test monitoring may be reduced to weekly …if the laboratory can document satisfactory performance with daily control tests as suggested by CLSI guidelines.’ The CAP checklist then goes on to briefly describe how to do so

29. AST QC Resources COLA, JC have similar verbiage, but is the checklist guidance enough? IMHO*, No! You need CLSI AST documents in your lab. (*In my humble opinion.)

30. CLSI AST Documents • M2 (disk diffusion) and M7 (MIC), 3 year cycle • Test performance •  QC (including daily-to-weekly QC)  • M100, annual cycle • Recommended drugs to test and report, interpretive breakpoints •  QC strains, QC troubleshooting, when to re-validate QC  • Others documents for other classes of microorganisms (Explore these and more at http://www.clsi.org/; choose Microbiology from the Shop drop down menu)

31. iClick? 3 How many of you have these CLSI documents in your lab: • 2012 M2 (disk diffusion), M7 (MIC) and M100 (Breakpoints, QC) • 2012 M7 and M100 • 2012 M2 and M100 • 2012 M2 and M7 • Any older versions of these documents • None of the above

32. Conversion to Weekly AST QC When day-of-use QC is being successfully performed at least once a week, a lab may consider weekly AST QC by a validation process: • Test QC strains 20 or 30 consecutive days • If 0-1 failures per drug-bug combo in 20 days OR 2-3 failures in 30 days • THEN Weekly QC may be implemented

33. Weekly AST QC • Perform QC testing once per week and with any new reagent • If weekly AST QC fails • If readily identifiable error (e.g. wrong QC strain; see M2 or M7 for more examples) correct error and retest once. If QC passes, document findings and continue weekly QC • If no error is found…

34. Weekly AST QC Re-validation Perform 5 consecutive days of QC on the failed drug-bug combo (We automatically take out a fresh QC strain from the freezer at this point) • If ALL 5 days pass, document and resume weekly testing • If a failure occurs again, STOP TESTING and thoroughly evaluate the process for error

35. Weekly AST QC Re-validation • Once a likely error has found, re-validate • Do not include data from the prior 5-day sequence • Must again pass these criteria • 0-1 failures per drug-bug combo/20 days OR • 2-3 failures per drug-bug combo/30 days • Remember that, during re-validation the lab is on a daily QC schedule-a failure means that that drug’s results are not released

36. AST QC References • Disk diffusion • M02-A11 Section 15 • M100-S22 Tables 3C, 3D • MIC • M07-A9 Section 16 • M100-S22 Tables 4F, 4G

37. AST QC Special Considerations M100-S22 Tables 3C & 4F, ‘Reference Guides to QC Frequency’ • When and how much re-validation to perform when a system change occurs(e.g. AST instrument repair, Abx formula change & Etc) Release of patient results when QC fails? • CLSI says OK with many qualifiers, but our lab generally just says No

38. iClick? 4 You are performing a weekly AST QC re-validation for drug X on your favorite automated ID/AST instrument when you incur a failure on the same AST panel for drug Y. What do you do? • Ignore the failure of drug Y and press on with the 5-day testing of drug X • Now treat the failure of drug Y as a second failure and start a 5 day test of drug Y • Stop testing, evaluate the whole process, then perform a new 20/30 day validation on the whole AST panel

39. iClick? 4 discussion The correct answer isn’t clear, but Marshfield Labs treats the failure of the second drug as a failure and starts a 5 day test on that drug also Why?  Subsequent failures may reflect a larger problem

40. ID QC CLIA requires a lab to show positive and negative reactions for each new lot and/or shipment of biochemical.  This includes kit and automated instrument ID methods!

42. ID QC There is a QC solution for automated ID instrument users: Streamlined QC Stay Tuned for the Next Talk!

43. The End Thank you… Questions?