340 likes | 535 Views
WHICH RISK ADJUSTMENT MODEL SHOULD WE USE? A FINNISH POINT OF VIEW. 16.3.2011 Matti Reinikainen North Karelia Central Hospital, Joensuu. THE FINNISH INTENSIVE CARE CONSORTIUM. 1994. 2007. So far, benchmarking in the Finnish Intensive Care Consortium has been mainly based on SAPS II
E N D
WHICH RISK ADJUSTMENT MODEL SHOULD WE USE? A FINNISH POINT OF VIEW 16.3.2011 Matti Reinikainen North Karelia Central Hospital, Joensuu
THE FINNISH INTENSIVE CARE CONSORTIUM 1994 2007
So far, benchmarking in the Finnish Intensive Care Consortium has been mainly based on SAPS II • Based on “The Severity Study” • 13 152 patients (720 from 7 Finnish hospitals) • Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 1993; 270: 2957-63. • APACHE II data is also collected
APACHE II vs. SAPS II • same basic principle, values of physiologic parameters from the first 24 hrs in the ICU • APACHE II (Acute Physiology And Chronic Health Evaluation II): the diagnostic category weight is added to the logit • SAPS II (Simplified Acute Physiology Score II): the diagnosis is not needed; instead the type of admission (scheduled surgical, unscheduled surgical, medical) affects the score
ARE THE OLD MODELS GOOD ENOUGH? • APACHE II - from 1985 - not always easy to choose the right diagnostic category • SAPS II - from 1993 - advantage: no diagnosis needed - disadvantage: does not take into account the diagnosis
DOES THE RISK PREDICTED BY SAPS II REFLECT REALITY? • A patient example: • HR 110/min • SAPs 84 mmHg • Tc 38 ºC • consciousness, renal function, blood cell counts, electrolytes quite OK • HCO3-18 mmol/l • age 65 years • no difficult chronic diseases • a medical admission • respiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33.3 kPa) • PROBABILITY OF IN-HOSPITAL DEATH ?
DOES THE RISK PREDICTED BY SAPS II REFLECT REALITY? • A patient example: • HR 110/min • SAPs 84 mmHg • Tc 38 ºC • consciousness, renal function, blood cell counts, electrolytes quite OK • HCO3-18 mmol/l • age 65 years • no difficult chronic diseases • a medical admission • respiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33,3 kPa) • SAPS II score 32 points → probability 0.128
SAPS II –score 32 → probability 0.128 • the database of the Finnish Consortium, 1998-2007, readmissions excluded: 2319 patients, with a SAPS II score of 32 points • hospital mortality 8.4%
SAPS II –score 32 → probability 0,128 • the database of the Finnish Consortium, 1998-2007, readmissions excluded: 2319 patients, with a SAPS II score of 32 points • hospital mortality 8.4% • diabetic ketoacidosis (n = 26): mort 0% • drug intoxication (n = 108): mort 0.9% • congestive heart failure (n = 49): mort 22.4%
CAN SAPS II STILL BE USED? • It overestimates the risk of death – leads to ”grade inflation” • If most intensive care units are graduating with honors, is it genuine quality or grade inflation? Popovich MJ, Crit Care Med 2002 • Recalibrations are needed
SMR 1998 – 2007, FINNISH INTENSIVE CARE CONSORTIUM SMR based on new calibration SMR based on original SAPS II model Päivitetty 09.04.2008
CAN SAPS II STILL BE USED? • It can be used for monitoring changes in a unit’s own results • Can be used for benchmarking purposes if the units to be compared have similar case-mix • Should not be used to compare results of units with major differences in case-mix
SAPS 3 WAS CONSIDERED IN FINLAND TOO - IS IT A GOOD ALTERNATIVE? • Values of physiological parameters ± 1 h of ICU admission • Reason for ICU admission documented more precisely than in SAPS II • Takes into account pre-ICU care • Prognostic performance? • Quality of data collected??
The SAPS 3 StudyMetnitz et al ICM 2005: 31:1336-1344. (Part 1)Moreno et al. ICM 2005: 31:1345-1355. (Part 2) • At first 22,791 admissions • Exclusions: readmissions (1455), < 16 yrs (628), those without ICU admission or discharge data (1074) and those that lacked an entry in the field ”ICU outcome” (57) - SAPS 3 basic cohort: 19,577 patients
The SAPS 3 StudyMetnitz et al ICM 2005: 31:1336-1344. (Part 1)Moreno et al. ICM 2005: 31:1345-1355. (Part 2) • SAPS 3 basic cohort: 19,577 patients • More exclusions: patients with a missing entry in the field of ”vital status at hospital discharge” (2540) and those still in hospital (253) • SAPS 3 Hospital outcome cohort: 16,784 patients • Quality of data? – at first, 5.5% of patients excluded because of missing data; then 13% of the remaining population excluded because of missing data on vital status
The SAPS 3 StudyMetnitz et al ICM 2005: 31:1336-1344. (Part 1)Moreno et al. ICM 2005: 31:1345-1355. (Part 2) • How about data completeness? • ”Data completeness was found to be satisfactory with 1 [0-3] SAPS II parameter missing per patient” • How many SAPS 3 parameters were missing? • ??? • Were the physiological values obtained within ± 1 h?
SAPS 3 – even if data quality in the study was less than perfect, does it work?
Ledoux D et al. SAPS 3 admission score: an external validation in a general intensive care population. Intensive Care Med 2008; 34: 1873-7. single-centre (Belgium), 802 patients “the SAPS 3 … model customised for Central and Western Europe … was not significantly better than the SAPS II.” Poole D et al. External validation of the Simplified Acute Physiology Score (SAPS) 3 in a cohort of 28,357 patients from 147 Italian intensive care units. Intensive Care Med 2009; 35: 1916-24. “…the SAPS 3 score calibrates inadequately in a large sample of Italian ICU patients and thus should not be used for benchmarking, at least in Italian settings”
Sakr Y et al. Comparison of the performance of SAPS II, SAPS 3, APACHE II, and their customized prognostic models in a surgical intensive care unit. Br J Anaesth 2008; 101: 798-803. • single-centre (Germany), 1851 patients • “… the performance of SAPS 3 was similar to that of APACHE II and SAPS II. Customization improved the calibration of all prognostic models.” • Metnitz B, Schaden E, Moreno R, Le Gall JR, Bauer P, Metnitz PG; ASDI Study Group. Austrian validation and customization of the SAPS 3 Admission Score. Intensive Care Med 2009; 35: 616-22. • 22 ICUs in Austria, 2060 patients • “The SAPS 3 … general equation can be seen as a framework … For benchmarking purposes, region-specific or country-specific equations seem to be necessary...”
2 ICUs in Norway, 1862 patients • “The performance of SAPS 3 was satisfactory, but not markedly better than SAPS II.”
SAPS II showed better discrimination • SAPS 3 equations showed better calibration • “…in our experience the scoring process is more time-comsuming and complex than that for SAPS II.”
SAPS 3, CONCLUSION: Does it work? – Yes! However, prognostic performance is NOT better than that of SAPS II the scoring process is more time-comsuming and complex than that for SAPS II (experience from Norway) on the other hand: according to many studies, the calibration of SAPS II is poor and customisation is needed
QUESTION DISCUSSED IN FINLAND: Should we implement a new risk-adjustment model (SAPS 3) that is not better than the old ones is more time-consuming would require customisation Or should we go on with one of the old models (that also require customisation)?
FINNISH (at least temporary) SOLUTION: OWN CUSTOMISED PREDICTION MODEL • One objective: no need to exclude patient groups for benchmarking • neuro- and cardiac surgical patients are not excluded • We did not want to increase the burden of data collection – no new parameters added • SAPS II –based data collection preserved • possible to compare the results with those of previous years • possible to describe the population using a well-known scoring system
OWN CUSTOMISED MODEL - M Reinikainen, P Mussalo, V Kiviniemi, V Pettilä, E Ruokonen • Based on patients treated in 2007-2008 • Readmissions excluded • Age ≥ 18 yrs • Those discharged to another ICU excluded • n = 25 801
OWN CUSTOMISED MODEL • Outcome variable (to be predicted) ”DEATH IN HOSPITAL” • Explaining covariates: • Emergency admission or planned beforehand • Surgical postoperative or medical • SAPS II score without admission type points • ln ((SAPS II score without admission type points) + 1) • Diagnostic groups having an independent impact on the probability of death • First a binary variable (0,1) was made of every APACHE III –dg group; everyone of these was tested separately • 31 dg groups with an independent effect were included in the model
LOGISTIC REGRESSION ANALYSIS • logit = β0 + β1X1 + β2X2 + … + βiXi • - the regression analysis produces the constant β0 and the coefficients βi • the logit can be calculated when the parameter values Xi are known • the logit (log odds) can also be expressed as and thus
LOGIT = -7,796 + 0,049 x (SCORE_SAPS_WITHOUT_ADM_TYPE_POINTS) + 1,013 x (ln(SAPS_WITHOUT_ADM_TYPE_POINTS + 1)) + 0,767 (if emergency admission) - 0,219 (if post-operative admission) + 1,229 (if DG_NONOP_CARDIOGENIC_SHOCK) + 0,364 (if DG_NONOP_CARDIAC_ARREST) – 0,796 (if DG_NONOP_RHYTHM_DISTURBANCE) + 0,348 (if DG_NONOP_ACUTE_MYOCARDIAL INFARCTION) + 0,422 (if DG_NONOP_BACTERIAL_OR_VIRAL_PNEUMONIA) – 1,619 (if DG_NONOP_MECHANICAL_AIRWAY_OBSTRUCTION) + 0,306 (if DG_NONOP_OTHER_RESP_DISEASES) + 0,795 (if DG_NONOP_HEPATIC_FAILURE) + 0,703 (if DG_NONOP_GI_PERFORATION_OR_OBSTRUCTION) + 0,643 (if DG_NONOP_GI BLEEDING_DUE_TO_VARICES) + 0,431 (if DG_NONOP_OTHER_GI_DISEASES) + 0,790 (if DG_NONOP_INTRACEREBRAL_HAEMORRHAGE) + 0,654 (if DG_NONOP_SUBARACHNOID_HAEMORRHAGE) + 0,400 (if DG_NONOP_STROKE) – 1,427 (if DG_NONOP_NEUROLOGIC_INFECTION) - 1,266 (if DG_NONOP_SEIZURE) – 0,486 (if DG_NONOP_OTHER_NEUROLOGIC_DISEASES) - 0,679 (if DG_NONOP_MULTIPLE TRAUMA_WITHOUT_HEAD_TRAUMA) – 0,658 (if DG_NONOP_METABOLIC_COMA) – 2,126 (IF DG_NONOP_DIABETIC_KETOACIDOSIS) – 2,245 (if DG_NONOP_DRUG_OVERDOSE) – 1,150 (if DG_NONOP_OTHER_METABOLIC_DISEASES) – 0,752 (if DG_NONOP_OTHER MEDICAL_DISEASES) + 0,340 (if DG_POSTOP_DISSECTING_OR_RUPTURED_AORTA) – 0,701 (if DG_POSTOP_CABG) + 0,701 (if DG_POSTOP_PERIPH_ARTERY_BYPASS_GRAFT) + 0,470 (if DG_POSTOP_GI_PERFORATION_OR_RUPTURE) + 0,411 (if DG_POSTOP GI_OBSTRUCTION) – 0,522 (if DG_POSTOP_SUBDURAL_OR_EPIDURAL_HAEMATOMA) – 0,885 (if DG_POSTOP_CRANIOTOMY_FOR_NEOPLASM) – 1,620 (if DG_POSTOP_OTHER_RENAL_DISEASES) PROB = EXP(LOGIT) / (1 + EXP(LOGIT))
PATIENT EXAMPLE A patient example: • HR 110/min • SAPs 84 mmHg • Tc 38 ºC • consciousness, renal function, blood cell counts, electrolytes quite OK • HCO3-18 mmol/l • age 65 years • no difficult chronic diseases • a medical admission • respiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33.3 kPa) • SAPS II score 32 points → probability 0.128
PATIENT EXAMPLE • SAPS II score 32 → probability 0.128 • New customised model: • If none of the diagnoses included in the model: probability 0.082 • dg bacterial pneumonia: probability 0.12 • dg drug intoxication: probability 0.0094
AUROC • APACHE II: 0.84 • SAPS II: 0.84 • new customised model: 0.87 • H-L test for new model: p = 0.127
CONCLUSIONS SAPS 3 works, but its prognostic performance is not better than that of SAPS II If you want to use SAPS 3, you should probably customise it If you want to use SAPS II, you should probably customise it Idea for future research: to create a Nordic risk adjustment model, predicting 6-month or 1-year mortality