1 / 21

ROC

ROC. Receiver Operating Characteristic- historic name from radar studies Relative Operating Characteristic - psychology, psychophysics Operating Characteristic - preferred by some. Disease -. Disease +. TN. TP. FN. FP. Test -. Test +. Specificity TNF = TN/(TN+FP) 0.86

misha
Download Presentation

ROC

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ROC Receiver Operating Characteristic- historic name from radar studies Relative Operating Characteristic - psychology, psychophysics Operating Characteristic - preferred by some

  2. Disease - Disease + TN TP FN FP Test - Test + Specificity TNF = TN/(TN+FP) 0.86 TNF + FPF = 1 Sensitivity TPF = TP/(TP+FN) 0.73 TPF + FNF = 1 Threshold for Positivity

  3. Test Characterization • SENSITIVITY of a test is its ability to detect disease within a diseased population. It is calculated as the fraction of diseased patients correctly identified by the test. Also called the True Positive Fraction and TPF= TP/(TP+FN) where (TP + FN) is the number of patients with the disease. • SPECIFICITY of a test is its ability to identify the absence of disease in a disease free population. It is calculated as the fraction of non-diseased patients correctly identified by the test. Also called True Negative Fraction and TNF= TN/(TN + FP) where (TN + FP) is the number of patients that are disease free. • ACCURACY is the fraction of correct test results or diagnoses. It is calculated as the number of patients with correct test results divided by the whole patient population (TP+TN)/(TP+FP+TN+FN). • PREVALENCE of the disease is is calculated as the fraction of patients who have the disease (TP+FN)/(TP+FP+TN+FN).

  4. Example: A study shows 90 true positives, 80 false positives, 20 true negatives and 10 false negatives. What are the sensitivity and specificity of the test? Sensitivity = TPF TPF = TP/(TP + FN) TPF = 90/(90 +10) = 0.90 Sensitivity = 90% Specificity = TNF TPF = TN/(TN + FP) TPF = 20/(20 +80) = 0.20 Specificity = 20% Accuracy = ?

  5. ROC Curves • The receiver operating characteristic (ROC) curve is used to compare overall performance (sensitivity and specificity) of a test. It can be used where imaging systems and observers (radiologists) both play a role in the test outcome or with purely objective tests such as with blood levels of cholesterol (LDL and or HDL). • An ROC curve is a graph of the True Positive Fraction (sensitivity) vs. False Positive Fraction (1-specificity) of a test as the threshold for positive result is changed.

  6. ROC Curve TPF FPF D- mean = 150, SD = 50 D+ mean = 250, SD = 75

  7. ROC - Diagnostic Imaging Threshold criteria are established using an ordinal scale of 0-4, ranging from under-reading (0) to over-reading (4). • At the most restrictive criterion (under reading or high threshold for positive), both sensitivity and the false-positive fraction are near zero (lower left on ROC). • At the most lax criterion (over reading or low threshold for positive), both the sensitivity and the false-positive fraction are near 1 (upper right on ROC). • In practice the operating point is a compromise between increasing sensitivity and decreasing specificity (increasing FPF).

  8. Hypothetical ROC curve 1.0 An experienced radiologist Over-reading Less experiencedreader True positive fraction (sensitivity) Useless Test How can this be? Under-reading 0 1.0 False positive fraction (1 - specificity)

  9. Non-diseased cases TPF, sensitivity Threshold less aggressive mindset Diseased cases FPF, 1-specificity

  10. Non-diseased cases moderate mindset TPF, sensitivity Threshold Diseased cases FPF, 1-specificity

  11. Non-diseased cases more aggressive mindset TPF, sensitivity Threshold Diseased cases FPF, 1-specificity

  12. Threshold Non-diseased cases Entire ROC curve TPF, sensitivity Diseased cases FPF, 1-specificity

  13. Entire ROC curve chance line TPF, sensitivity Reader Skill and/or Level of Technology FPF, 1-specificity

  14. Quantifying ROC Curves • The area under an ROC curve is a measure of overall performance. • The maximum area is 1.0 • Useless test is the diagonal line from 0.0 to 1.0 and has area under ROC =0.5, so a more meaningful measure is the area in excess of 0.5. • As test performance improves, the curve moves towards the upper left corner and the area under ROC increase.

  15. TPF vs FPF for 108 US radiologists in study by Beam et al.

  16. ( Chest film study by E. James Potchen, M.D., 1999 )

  17. Dilemma:Which modality is better? 1.0 Modality B Modality A True Positive Fraction 0.0 0.0 1.0 False Positive Fraction

  18. ROCs (one outcome) 1.0 B is better than A Modality B True Positive Fraction Modality A 0.0 0.0 1.0 False Positive Fraction

  19. ROC (another outcome) 1.0 B same as A Modality B Modality A True Positive Fraction 0.0 0.0 1.0 False Positive Fraction

  20. ROC (another outcome) 1.0 A is better than B Modality B Modality A True Positive Fraction 0.0 0.0 1.0 False Positive Fraction

More Related