Module PANEL TESTING

1. ModulePANEL TESTING High level

2. Content Overview • What is panel testing? • What is panel testing used for? • PT advantages and disadvantages • Preparation of test smears • Validation of panel batches • Panels’ composition • Organization of a panel testing round • Analysis of results; scoring system • Interpretation and feedback • Forms

3. What is Panel Testing? • One of EQA methods • System of sending stained and/or unstained smears from NRL to peripheral laboratories to check proficiency in performing AFB smear microscopy and reporting AFB results • Tests individual performance of a laboratory worker, not the laboratory overall

4. What is Panel Testing Used For? • Rapid assessment of performance of a laboratory staff to prioritize training and supervisory activities • Quick detection of problems associated with very poor performance • Evaluation of competency of laboratory technicians prior to and following training

5. What is Panel Testing Used For? (2) • A minimal step for EQA with limited resources • Monitoring performance of individuals in absence of a rechecking program • Supplements rechecking programs • Investigation of excessive errors found in rechecking

6. Advantages of Panel Testing • Low workload for a peripheral center • Improves laboratory credibility • Rapid response countrywide possible • Possible identification of gross deficiencies • Use of stained and unstained smears can help identify source of problem

7. Disadvantages of Panel Testing • Technicians know they are being evaluated • Does not measure routine performance • High workload for NRL • need foradditional resources -appropriate equipment, highly qualified staff to produce panels • a system for panel sets distribution, data collection, analysis and feedback • May not be motivating to improve daily performance

8. Preparation of Test Smears • Routine patient smears: • Problems with consistency • Only stained smears available • Smears of the required AFB quantification may not be easily available in the needed quantity • Specially manufactured smears: • Can be stained or unstained • Provide uniformity of the test for technicians • Provide known quantification of AFB required

9. Manufacturing Smears for PT • Should be done at NRL • requires a safety hood, centrifuge, vortex, water bath, lab supplies (pipettes, tubes, slides, boxes etc.) • BIOSAFETY MEASURES! • Smears are prepared from known positive and negative sputa • Reference for the manufacturing procedures: AFB Smear Microscopy EQA Guidelines • Requires time practice and expertise

10. Validation of Panel Batches • Mandatory requirement! • Pre-validation: • Validation of consistency of panel batches prior to sending test panels out to periphery • Post-validation: • Validation of panel slides / batches after receiving aggregate results from all laboratories • Keep accurate records of batches prepared and detailed results of the validation process

11. Pre-validation • Stain at least 6 slides from each batch to be examined independently by 3 or more technicians • Calculate the average results and standard deviation (SD) • The average minus 2 SD should be > 0 to accept the batch • Use the Validation Log to record results

12. Normal Distribution • All values are symmetrically distributed around the mean • Characteristic “bell-shaped” curve • Assumed for all quality control statistics Frequency x Variable

13. What is Standard Deviation? The principle calculation used in the laboratory to measure dispersion of a group of values around a mean Standard Deviation – Statistical Formula

14. For a set of data with a normal distribution, a value will fall within a range of: +/- 1 SD 68.2% of the time +/- 2 SD 95.5% of the time +/- 3 SD 99.7% of the time Laboratories use the +/- 2 SD criteria for the limits of the acceptable range for a control value When the QC measurement falls within that range, there is 95.5% confidence that the measurement is correct Frequency 68.2% 95.5% 99.7% -3s - 2s -1s Mean +1s +2s +3s Standard Deviation and Probability

15. Sample Form: Validation Log for AFB Panel Testing Slide Batches (pre-validation) • Intended positives should never be negative • Intended negative smear should never be positive • Quantification differences should not reach 2 steps on scale

16. Post-validation • The same smear error reported by a majority of technicians may represent a problem with the panel slide / batch: • Technical difficulties in preparing panel slides • Error in the pre-validation • Incorrect recording of the expected result • Fading of smears during transportation to peripheral sites

17. Logbook of Panel Slides Sets / Post-validation

18. Coding of Panel Smears • Ensure that result can not be guessed by an examinee – to avoid reading bias • Make identification of a panel smear clear to a supervisor in charge of a panel testing exercise Example of a smear’s code: 62-45-1 a smear serial number in a panel a panel set number a batch number

19. Panels’ Composition • The composition of a panel set is determined by NRL • Number and types of slides to reassure that correct or incorrect results are not accidental • At least 10 slides provides a valid and fair test • Batch of stained and unstained smears • Unstained smears: • Evaluate staining technique; provide information about stain preparation and quality

20. Examples of Panel Sets Compositions A panel test should represent a challenge in terms of difficulty: -some scanty and low-positive smears

21. Getting Started: Issues to Consider • System for sending slides • Frequency of testing • Forms to record and report results • Time allowed for technicians to complete PT • Availability of microscopes • Performance criteria • Feedback and corrective action if needed • Mechanism to resolve discrepant results

22. Implementation of Panel Testing • Responsibility of the NRL- from preparation of slides to analysis of results and feedback • Determine the number of AFB technicians who will participate in PT (ensure preparation of the needed number of panels) • Communicate with Public Health Directors regarding EQA activities • Prepare the schedule for panel testing in each location • Collaborate with intermediate laboratories

23. Sending Slides • Delivery system based on services, regulations, resources available: • mail/post • courier • supervisory visit • Turnaround time • Safe package to prevent breakage of slides: • strong plastic slide holders

24. A POSSIBLE SCHEME OF A PANEL TESTING ROUND National reference laboratory Intermediate laboratory Peripheral laboratories Peripheral laboratories

25. Performinga Panel Test Round • Frequency: at least one to two times a year • A standardized PT reporting form / an accompanying letter to provide instructions • Individual, not group work • No incentives or punitive actions as a result of the PT exercise • Time allowed to complete the PT exercise, maximum: • 2 hours for a stained slide set • 3 hours for an unstained slide set

26. Individual Results of Panel Testing / Feedback Form

27. Management of PT During a Supervisory Visit • Administration of PT during on-site visits: • can be effective in some circumstances • provides direct observation of work under PT exercise • corrective action may be easily facilitated • BUT: may be impractical in routine conditions • can be done in a special survey • Important:PT must not disrupt routine patients’ examinations, therefore consider: • Careful planning of a supervisory visit • Allocating sufficient time for a visit

28. Analysis of PT Results • A scoring system is to be developed prior to test • Distinguish major and minor errors • false positive/negative related to 1+, 2+ or 3+ errors are major errors • quantification errors (at least a 2 grade difference) and false positive / negative errors in the scanty group (1-9 AFB) are considered minor • Determine successful score • Determine plan of action for poorperformances

29. Types and Classification of Errors Correct: No errors QE Quantification error Minor error LFN Low False Negative Minor error LFP Low False Positive Minor error HFN High False Negative Major error HFP High False Positive Major error

30. Example of PT Scoring • Set of 10 slides, each slide is worth 10 points, total possible score = 100 • HFP and HFN scores 0 • LFP, LFN and QE scores 5 (QE = 2 grades difference) • Passing score = 80 – 90

31. Analysis of PT Results • Study the aggregate results from all laboratories • Post-validate panel slides/batches • Assure that poor performance is not due to panel slide problems • If a majority of technicians fail to report correct results for the same slide/batch it may represent a problem with panel slide preparation: • exclude this slide from scoring • check returned discrepant slides • detect problems in preparation of panel smears • undertake measures to improve the quality of panel smears preparation

32. PT– Interpretation of Results • False positive and negative errors should be considered separately • False positives - lack of proficiency / faulty microscope • False negatives - poor stain / inadequate examination time / poor microscope

33. Feedback to Laboratories on PT results • Timely and confidential • Individual and aggregate test results • Criteria for acceptable performance • Reports to TB program coordinator should provide appropriate background information and recommendations and not simply scores • Poor performance often requires a visit to laboratory

34. PT Aggregate Results of Multiple Laboratories

35. PT Aggregate Results Report: Example

36. Conclusion: Laboratories submitting unacceptable PT results with documented consistency and quality of PT slides experience serious problems with AFB microscopy. Additional resources should be obtained for supervisory visits, correction of problems identified, including replacement of faulty microscopes (and/or stains), retraining if needed, and follow-up panel testing.

37. Key Messages (I): • PT is an effective method when it is necessary to quickly obtain information about capabilities of individual laboratory technicians to read smears and report results according to standards approved by NTP. • PT is considered to be less effective than rechecking because it does not monitor routine performance; BUT • PT can be more effective than rechecking in the areas where prevalence of positives is low.

38. Key Messages (II): • The main prerequisite to start a panel testing program in a country is availability of a laboratory with a highly qualified staff capable to safely produce panel sets of appropriate quality and required composition. • Validation of panel batches (pre- and post-validation) is the mandatory requirement for PT. • A well functioning system should be established to distribute panels, collect and analyze data; provide timely feedback to peripheral laboratories.