140 likes | 268 Views
Royal College of Surgeons in Ireland Coláiste Ríoga na Máinleá in Éirinn. Indices of Performances of CPRs Nicola Motterlini, Borislav D Dimitrov, Tom Fahey. Contents. CPR derivation Analytical approach CPR validation Indices of discrimination Indices of calibration
E N D
Royal College of Surgeons in IrelandColáiste Ríoga na Máinleá in Éirinn Indices of Performances of CPRs Nicola Motterlini, Borislav D Dimitrov, Tom Fahey
Contents • CPR derivation • Analytical approach • CPR validation • Indices of discrimination • Indices of calibration • Performance of a CPR in several validation studies (systematic review with meta-analysis)
Analytical approachto CPR(an example) Target disorder (presence or absence) Multivariable approach (e.g., logistic regression model) Total score: 3+2+1 = 6 Prediction LOW RISK % Total score ≤ 3 MODERATE RISK % 4 ≤ Total score ≤ 7 HIGH RISK % Total score ≥ 8 Decision NO TREATMENT FURTHER DIAGNOSTIC TESTING TREATMENT
Thresholds approach to diagnosis Score 100% HIGH RISK Treatment 10 9 8 7 6 5 4 3 2 1 Diagnosis / test threshold Probability of disease Further diagnostic testing Test / reassurance threshold 0% LOW RISK No treatment
Performance of a CPR can be assessed like a performance of a diagnostic test! TP, number of true positives, FP, number of false positives, FN, number of false negatives, TN, number of true negatives, NT+, the number of patients with positive test results, NT-, the number of patients with negative test results, ND+, the number of patients with the disease, and ND−, number of patients without the disease
Indices of performance • For discrimination (classification) • Ability to discriminate between diseased individuals and non-diseased individuals (e.g., sensitivity, specificity, ROC curve, odds ratio, positive and negative predictive values, likelihood ratio etc.) • For calibration (comparison) • Agreement between predicted probabilities and observed probabilities (e.g., risk ratio, Hosmer-Lemeshow goodness-of-fit test etc.)
Discrimination – sensitivity and specificity Sensitivity = TP/ND+ Specificity = TN/ND-
Calibration – risk ratio Predicted Disease risk (%) Observed Low Moderate High Score risk <1 “under-prediction” =1 “accurate prediction” >1 “over-prediction” Predicted Risk ratio -> Observed
Performance of a CPR in several validation studies (systematic review with meta-analysis)
Pooled analysis • When the 95% CI includes 1 there is no significant statistical difference between the observed and predicted by CPR • The smaller the interval, the more accuracy are the results
Index of heterogeneity I2 value over 50% is a strong indicator for heterogeneity
Summary • CPR derivation • Multivariable approach to develop a CPR • CPR validation • Indices of discrimination (sensitivity, specificity) • Indices of calibration (risk ratio) • Systematic review with meta-analysis of validation studies of CPRs