1 / 11

THE POWER OF ERROR DETECTION OF WESTGARD MULTI-RULES: A RE-EVALUATION

THE POWER OF ERROR DETECTION OF WESTGARD MULTI-RULES: A RE-EVALUATION. Graham Jones Department of Chemical Pathology St Vincent’s Hospital, Sydney. Background. Westgard multi-rules are claimed to increase the power of error detection of laboratory QC procedures.

brock
Download Presentation

THE POWER OF ERROR DETECTION OF WESTGARD MULTI-RULES: A RE-EVALUATION

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AACB ASM 2003 THE POWER OF ERROR DETECTION OF WESTGARD MULTI-RULES: A RE-EVALUATION Graham Jones Department of Chemical Pathology St Vincent’s Hospital, Sydney

  2. AACB ASM 2003 Background • Westgard multi-rules are claimed to increase the power of error detection of laboratory QC procedures. • Power Function Charts can quantify the ability of these rules to detect changes in assay performance. • Examples of Power Function Charts are available on the Westgard QC website (www.westgard.com) • In this poster I re-evaluate the power of error detection of QC rules which require data from more than one QC run (Multi-run rules).

  3. AACB ASM 2003 Hypothesis • That the correct model for assessing the Power of Error Detection for multi-run QC rules should only show benefit for these rules if the error has not already been detected in QC runs required to gather data for those rules. • This hypothesis was modelled and compared to data on the Westgard website.

  4. AACB ASM 2003 Nomenclature • Single-run rules • All data is contained in a single QC run • For n=2 includes 13s and 22s • For n=4 includes 13s and 22s and 41s • Multi-Run Rules • Requires data from more than one QC run • For n=2 includes 41s and 10x • For n=4 includes 8x

  5. AACB ASM 2003 Methods • Power Function Charts were produced using a Microsoft Excel spreadsheet. • QC results were simulated using a random number generator with a normal distribution. • Changes in bias were modelled by adding various constants to the output. • QC rules were evaluated by the frequency with which they were triggered at changes in bias. • Westgard multi-rules with n=2 were evaluated for bias detection: 13s/22s/41s/10x. • Changes in random error were not modelled.

  6. AACB ASM 2003 Hypothesis - Graphical Display This display uses 10x as an example of a multi-run rule +3SD +2SD Mean -2SD 1 2 3 4 5 -3SD Change in assay bias QC run - within-run rules evaluate performance (13s/22s) QC run - within-run rules evaluate performance (13s/22s) - multi-run rule evaluates performance (10x across both materials) - Only adds benefit if shift NOT detected by QC events 1-4

  7. AACB ASM 2003 Results A Graph A - Original data from Westgard website B Probability for Rejection Graph B - Model of data from Westgard website. - Multi-run rules fire even if shift would have been detected previously. Shift in Bias (multiples of SD)

  8. AACB ASM 2003 C Graph C - Westgard data adjusted for hypothesis. - Multi-run Rules fire only if shift would NOT have been detected previously. Probability for Rejection D Graph D - Model of individual rules from Graph C - Multi-run Rules fire only if shift would NOT have been detected previously. Shift in Bias (multiples of SD) Shifts detected with 90% certainty from full multi-rules Shifts detected with 90% certainty from within-run rules.

  9. AACB ASM 2003 Results • A power Function Chart from the Westgard website showing multi-rules for bias detection with n=2 is shown in graph A. • The change in bias which Westgard claims full Multi-rules can detect with 90% certainty is about 2.0 times the SD of the assay (Graph A). • My model of the Westgard data, with multi-run rules triggered even if the change in bias would been previously detected, agrees well with the website data (Graph B). • In the Westgard model the multi-run rules (10x and 41s) enhance the error detection over the within-run rules (graphs A and B)

  10. AACB ASM 2003 • The model excluding multi-run rules when a shift would have been detected previously is shown in Graph C. • When these previously-detected shifts are removed from the data, the assay bias which can be detected with 90% certainty is reduced to about 3.3 times the assay SD (Graph C). • With this model the multi-run rules do not add to the within-run rules for confident error detection. • When the individual rules are plotted it can been seen that the multi-run rules never add to the error detection with 90% certainty. • The multi-run rules can be considered warning rules.

  11. AACB ASM 2003 Conclusion • The multi-run rules, as described on the Westgard website, give a falsely low estimate of the change in bias which can be detected with 90% certainty. • The 10x and 41s rules add little to the overall error detection at the 90% confidence level with 2 QC samples per run. • Multi-run rules are similarly non-contributory with 4 QC samples per run (data not shown).

More Related