1 / 16

ASSESSING THE PERFORMANCE OF MEDICAL DIAGNOSTIC SYSTEMS: THE RECEIVER OPERATING CHARACTERISTIC (ROC) CURVE

ASSESSING THE PERFORMANCE OF MEDICAL DIAGNOSTIC SYSTEMS: THE RECEIVER OPERATING CHARACTERISTIC (ROC) CURVE. JOSEPH GEORGE CALDWELL, PH.D. 27 FEBRUARY 2007. BACKGROUND. TWO-ALTERNATIVE STATISTICAL DECISION PROBLEMS: HOW TO ASSESS ACCURACY?

Mercy
Download Presentation

ASSESSING THE PERFORMANCE OF MEDICAL DIAGNOSTIC SYSTEMS: THE RECEIVER OPERATING CHARACTERISTIC (ROC) CURVE

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ASSESSING THE PERFORMANCE OF MEDICAL DIAGNOSTIC SYSTEMS: THE RECEIVER OPERATING CHARACTERISTIC (ROC) CURVE JOSEPH GEORGE CALDWELL, PH.D. 27 FEBRUARY 2007

  2. BACKGROUND • TWO-ALTERNATIVE STATISTICAL DECISION PROBLEMS: HOW TO ASSESS ACCURACY? • FROM A SERIES OF TEST RESULTS (E.G., BLOOD TESTS, X-RAYS, MRI IMAGES), MUST DECIDE WHETHER SOMEONE HAS A PARTICULAR DISEASE OR CONDITION • STATISTICAL CONSULTING REPORT: “DESCRIPTION OF THE STATISTICAL SUBSYSTEM OF THE AUTOMATED RECEIVER OPERATING CHARACTERISTIC (ROC) SYSTEM”

  3. THE TWO-ALTERNATIVE STATISTICAL DECISION PROBLEM THE STIMULUS-RESPONSE MATRIX: TYPE 2 ERROR TYPE 1 ERROR

  4. HOW TO REPRESENT THE ACCURACY OF THE DECISION PROBLEM? • BY VARYING THE DECISION CRITERION, THE PROBABILITIES OF THE TWO TYPES OF ERROR MAY BE ADJUSTED -- A TRADE-OFF • BY CHANGING THE DIAGNOSTIC PROCEDURES (E.G., ADDING TESTS, IMPROVING TESTS), THE PROBABILITIES OF BOTH TYPES OF ERRORS MAY BE REDUCED • A PROBLEM: TOO MANY STIMULUS-RESPONSE MATRICES. HOW TO SUMMARIZE ACCURACY MORE SUCCINCTLY?

  5. THREE EXAMPLES

  6. THE RECEIVER OPERATING CHARACTERISTIC (ROC) CURVE

  7. AN EXAMPLE

  8. AN EXAMPLE (CONT.)

  9. EXAMPLES OF ROC CURVES

  10. APPLICATION TO LOGISTIC REGRESSION LOGISTIC REGRESSION MODEL: ESTIMATE DECISION CRITERION: DECIDE “YES” IF pest > c AND “NO” IF pest <= c VARY c AND CALCULATE THE PROPORTION OF HITS AND FALSE POSITIVES, AND PLOT ON A ROC GRAPH (EACH VALUE OF c YIELDS A DIFFERENT POINT ON THE CURVE).

  11. ROC CURVES FOR LOGISTIC REGRESSION MODELS (SAS OUTPUT)

  12. ROC CURVES FOR LOGISTIC REGRESSION MODELS (SAS OUTPUT)

  13. REFERENCES

  14. REFERENCES

  15. REFERENCES 1. Swets, John A., and Ronald M. Pickett, Evaluation of Diagnostic Systems: Methods from Signal Detection Theory, Academic Press, New York, 1982 2. Green, David M. and John. A Swets, Signal Detection Theory and Psychophysics, Wiley, 1966, reprinted by Peninsula Publishers, Los Altos, CA, 1988 3. Dorfman, Donald D. and Edward Alf, Jr., "Maximum-Likelihood Estimation of Parameters of Signal-Detection Theory and Determination of Confidence Intervals -- Rating Method Data", Journal of Mathematical Psychology, vol. 6, 1969, pp. 487-496 4. Ogilvie, John C. and C. Douglas Creelman, "Maximum-Likelihood Estimation of Receiver Operating Characteristic Curve Parameters," Journal of Mathematical Psychology, vol. 5, 1968, pp. 377-391. 5. Rau, C. R., Linear Statistical Inference and Its Applications, Wiley, New York, 1965, pp. 302-309 6. Ralston, A., A First Course in Numerical Analysis, McGraw Hill, New York, 1965, p. 348 7. Caldwell, J. G., Description of the Statistical Subsystem of the Automated Receiver Operating Characteristic System, 1995 8. Logistic Regression Examples Using the SAS System, SAS Institute, Cary, NC, Version 6, First Editioin, 1995 9. Bamber, D. (1975), “The Area Above the Ordinal Dominance Graph and the Area Below the Receiver Operating Characteristic Graph,” Journal of Mathematical Psychology, 12, 387-415. 10. Hanley, J. A. and McNeil, B. J. (1982), “The Meaning and Use of the Area Under a Receiver Operating Characteristic (ROC) Curve,” Radiology, 143, 29-36.

More Related