1 / 54

Measures of diagnostic accuracy

Statistical Methods for Analysis of Diagnostic Accuracy Studies Jon Deeks University of Birmingham with acknowledgement to Hans Reitsma. Measures of diagnostic accuracy. Positive and negative predictive values Sensitivity and specificity Likelihood ratios Area under the ROC curve

roana
Download Presentation

Measures of diagnostic accuracy

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Statistical Methods for Analysis of Diagnostic Accuracy StudiesJon DeeksUniversity of Birminghamwith acknowledgement to Hans Reitsma

  2. Measures of diagnostic accuracy • Positive and negative predictive values • Sensitivity and specificity • Likelihood ratios • Area under the ROC curve • Diagnostic odds ratio

  3. Diagnostic accuracy studies • Results from the index test are compared with the results obtained with the reference standard on the same subjects • Accuracy refers to the degree of agreement between the results of the index test and those from the reference standard

  4. Basic Design Series of patients Index test Reference standard Cross-classification

  5. Clinical problem • Diagnostic value of B type natriuretic (BNP) measurement • Does BNP measurement distinguish between those with and without left ventricular dysfunction in the elderly? • Smith et al. BMJ 2000; 320: 906.

  6. Anatomy of diagnostic study • Target population: unscreened elderly • Index test: BNP • Target condition: LVSD • Final diagnosis (reference standard): echocardiography – global and regional assessment of ventricular function including measurement of LV ejection fraction

  7. Our example Elderly patients BNP measurement Echocardiography for LVSD Cross-classification

  8. Results of BNP study

  9. Measures of test performance • sensitivity • 11 / 12 = 92% < Pr(T+|D+) > • specificity • 93 / 143 = 65% < Pr(T-|D-) >

  10. Measures of test performance • positive predictive value • 11 / 61 = 18% < Pr(D+|T+) > • negative predictive value • 93 / 94 = 99% < Pr(D-|T-) >

  11. Sensivity and Specificity not directly affected by prevalence • sensitivity • 131 / 143 = 92% • specificity • 93 / 143 = 65%

  12. Predictive values directly affected by prevalence • positive predictive value • 131 / 181 = 72% • negative predictive value • 93 / 105 = 89%

  13. Do sensitivity and specificity vary with prevalence? • Test performance is sometimes observed to be different in different settings, patient groups, etc. • Occasionally attributed to differences in disease prevalence, but: • diseased and non-diseased spectrums differ as well. • e.g. using a test in primary care and secondary care referrals • the diseased group are different (cases more difficult) • the non-diseased group are different (conditions more similar) • sensitivity may decrease, specificity certainly decreases

  14. Likelihood ratios • Why likelihood ratios? • Applicable in situations with more than 2 test outcomes • Direct link from pre-test probabilities to post-test probabilities

  15. Likelihood ratios • Information value of a test result expressed as likelihood ratio

  16. Likelihood Ratio of positive test • How more often a positive test result occurs in persons with compared to those without the target condition

  17. Likelihood ratios • Likelihood ratio of a negative test result • How less likely a negative test result is in persons with the target condition compared to those without the target condition

  18. Likelihood ratios

  19. Calculate likelihood ratios from column percentages

  20. Interpreting likelihood ratios • A LR=1 indicates no diagnostic value • LR+ >10 are usually regarded as a ‘strong’ positive test result • LR- <0.1 are usually regarded as a strong negative test result • But it depends on what change in probability is needed to make a diagnosis

  21. 92% LR+ = 10 55% 10% 50%

  22. Advantages of likelihood ratios • Still useful when there are more than 2 test outcomes

  23. BNP is a continuous measurement • Dichotomisation of BNP(high vs. low) means loss of information • Higher values of BNP are more indicative of LVSD

  24. Results BNP study

  25. Likelihood ratios • Stratum specific likelihood ratios in case of more than 2 test results

  26. Compute LR from column percentages

  27. Bayes’ rule Post-test odds for disease = Pre-test odds for disease x Likelihood ratio

  28. Bayes’ rule • Pre-test odds • chance of disease expressed in odds • example: if 2 out of 5 persons have the disease: probability = 2/5 in odds = 2/3

  29. Bayes’ rule • odds = probability / (1 – probability) • probability = odds / (1 + odds)

  30. Bayes’ rulepatient with BNP >26.7 • Pre-test probability = 0.5 • Pre-test odds = 0.5 / (1-0.5) = 1 • LR(BNP >26.7) = 3.83 • Post-test odds = 1x3.83 = 3.83 • Post-test probability = 3.83 / (1+3.83) = 0.79

  31. Bayes’ rulepatient with BNP lower than 18.7 • Pre-test probability = 0.5 • Pre-test odds = 0.5 / (1-0.5) = 1 • LR(CK< 40) = 0.13 • Post-test odds = 1 x 0.13 = 0.13 • Post-test probability = 0.13 / (1+0.13) = 0.12

  32. Probability for LVSD after BNP

  33. 79% 52% 12% 50%

  34. 5% 17% 5% 1%

  35. Probability for LVSD after BNP

  36. Confidence intervals • Sample uncertainty should be described for all statistics, using confidence intervals + gives upper limit - gives lower limit Standard error of estimate estimate of effect Normal deviate (1.96 for 95% CI)

  37. Confidence Intervals for Proportions • Sensitivity, specificity, positive and negative predictive values, and overall accuracy are all proportions

  38. Exact or Asymptotic CI? • Asymptotic CI are approximations • Inappropriate when • proportion is near 0% or near 100% • sample sizes are small (confidence intervals are not symmetric in these cases) • Preferable to use Binomial exact methods • can be computed in many statistics packages • or refer to tables

  39. Comparison of Asymptotic and Exact Methods

  40. Confidence Intervals for Ratios of Probabilities and Odds • Odds ratios are ratios of odds • Likelihood ratios are ratios of probabilities

  41. CIs for study • Sensitivity = 92% (62%, 100%) • Specificity = 65% (57%, 73%) • PPV = 82% (70%, 91%) • NPV = 99% (94%, 100%) • LR(>= 26.7) = 3.8 (2.4, 6.1) • LR(18.7 < 26.7) = 1.1 (0.3, 4.1) • LR(<18.7) = 0.13 (0.02, 0.84)

  42. ROC-curve • ROC stands for Receiver Operating Characteristic • ROC-curve shows the pairs of sensitivity and specificity that correspond to various cut-off points for the continuous test result

  43. Continuous diagnostic test results

  44. Heterogeneity in Threshold

  45. Heterogeneity in Threshold

  46. Heterogeneity in Threshold

  47. Heterogeneity in Threshold

  48. Heterogeneity in Threshold

  49. Threshold effects Decreasing threshold increases sensitivity but decreases specificity Increasing threshold increases specificity but decreases sensitivity

  50. Change in cut-off valueand effect on sens & spec

More Related