1 / 18

Rapid Critical Appraisal of diagnostic accuracy studies

Rapid Critical Appraisal of diagnostic accuracy studies. Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford www.cebm.net. Diagnosis – what is the problem?. Log of reasons by several docs: Monitoring – has it changed? Prognosis – risk/stage within Dx

mateo
Download Presentation

Rapid Critical Appraisal of diagnostic accuracy studies

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Rapid Critical Appraisalof diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford www.cebm.net

  2. Diagnosis – what is the problem? Log of reasons by several docs: Monitoring – has it changed? Prognosis – risk/stage within Dx Treatment planning, e.g., location Stalling for time! What are tests used for?

  3. Is the test accurate? • To be accurate a test should be: • Reproducible • We get the same (wrong?) answer every time • P I-I question • Valid • We get the right answer • P I O question

  4. Organ Feature Agreement Kappa Reference Rectal Cancer Grading 50% to 69% 0.11 to 0.5 Thomas Hodgkins Classification 56% 0.44 Holman Melanoma depth 82%; 64% 0.68; 0.23 Breslow; Clark Breast cancer classification 73% 0.46 Stenkvist Reproducibility:Agreement of histopathologists Ken Fleming, Evidence-based pathology. EBM 1997

  5. Diagnostic Test Accuracy measurements Sensitivity is the probability of a positive test in a diseased person Specificity is the probability of a negative test in a non-diseased person.

  6. Is the test helpful (valid)?The Youden Index • Youden Index = sensitivity+specificity-1 • For a test to be useful, then • sensitivity + specificity > 1 (Youden Index > 0) • Examples: • Coin Toss with +ve = "heads"sensitivity = 0.5 specificity = 0.5 • Youden = 0

  7. Can a test rule-in or rule-out? SpPln Specific test, Positive rules Ineg: Rovsing's sign, ST elevation > 2mm SnNout Sensitive test, Negative rules Outeg: Erect abdominal film for obstruction, Elevated WCC in CSF (>5/mm )

  8. Can I trust the accuracy data from the study? RAMMbo Recruitment: Was an appropriate spectrum of patients included? (Spectrum Bias) Maintainence: All patients subjected to a Gold Standard? (Verification Bias) Measurements: Was there an independent, blind or objective comparison with a Gold Standard? Observer Bias; Differential Reference Bias

  9. Appraisal of Tests: RAMMbo was the evaluatioin fair? Population Index test OutcomeMeasures (Gold Standard) Maintained? OutcomeMeasures(Gold Standard) Representation? Comparator Test Blinded or Objective?

  10. QUADAS

  11. The Literary Digest PollLandon versus Roosevelt, 1936 % for Roosevelt • Literary Digest: 2.4 Million reader poll • Prediction for Roosevelt 43% • Gallup's 50,000 random sample • Prediction of the election result 56% • Gallup's 3,000 Digest readers • Prediction of Digest prediction 44% • Election result 62%

  12. Good sampling: needs a sample frame & unbiased selection Target Population Sample Frame Actual Sample Complete data

  13. High Threshold Low Threshold Were reference Measurements Blinded or Objective? Use standardised measurement strategy across ALL patients Index +ve Apparent difference Index -ve

  14. Sens=50% Spec=90% Sens=92% Spec=65% BNP screen of GP elderly patients • UK GP setting • 155 patients • 70-84 yrs old • Echocardiogram • 12 with CCF Smith H, et al BMJ, 2000

  15. Replication & Extension Study Potentially Eligible Systematic Reviews 1999-2002 N=191 34 systematic reviews with 39 meta analyses containing 678 original studies 157 systematic reviewsexcluded Assessment process systematic reviews 28 systematic reviews with 31 meta-analyses containing 545 original studies 6 systematic reviews with 8 meta-analyses excluded Assessment process original studies 31 meta-analyses containing 487 original studies 58 original studiesexcluded AWS Rutjes et al. 2005

  16. Study characteristics Severe cases and healthy controls Other case-control designs Differential verification Partial verification Non blinded studies Non-consecutive Retrospective data collection No description index No description reference No description population 0 1 2 3 4 5 RDOR 487 studies; 31 meta-analyses

  17. How well are diagnostic studies reported? • 112 studies in 4 major journals (1978-1993) Standard N (%) Spectrum composition 30 (27) Avoidance of workup bias 51 (46) Avoidance of review bias 43 (38) Test accuracy precision 12 (11) Indeterminate test results 26 (23) Test reproducibility 26 (23) Accuracy in subgroups 9 (8) Reid MC, Lachs MS, Feinstein AR. Use of Methodological Standard in diagnostic test research. JAMA 1995;274:645-651

  18. Using Evidence about Tests • Appraise the study - PICO • Always ask • Is it useful at all? (Youden Index > 0) • Usually ask • Can it “rule in” or “rule out” a disease? • Often ask • What is the post-test probability in the same situation as the study? • Rarely ask • Calculation of post-test probabilities in a different situation

More Related