Can Different Approaches to Monitoring of Shock be Tested in a Rigorous Scientific Fashion?

1. Can Different Approaches to Monitoring of Shock be Tested in a Rigorous Scientific Fashion? Jason D. Christie, MD, MS Assistant Professor of Medicine Assistant Professor of Epidemiology Division of Pulmonary, Allergy, and Critical Care Center for Clinical Epidemiology and Biostatistics University of Pennsylvania, Philadelphia, USA

2. Roadmap • Reliability of Measures • Which Questions to Ask? • Mechanistic Inference vs. Prediction • Case heterogeneity within shock syndromes • Refining the Clinical Definition of Shock • Impact studies • Prediction • Effectiveness

3. CCEB Reliability of Measures • Reliability • ability to provide consistent results with repeat use. • Components • consistency with repeat testing by a single observer • “test-retest reliability” • consistency between different observers • “inter-rater reliability”

4. Reliability of hemodynamic measures Bland, Altman Stat Meth Med Res 1999

5. CCEB Reliability of Measures • Reliability for measures that produce a number is less important than calibration and drift • The Reliability of operator dependent measures should be determined rigorously in clinical research

6. “All who drink of this remedy recover in a short time, except those whom it does not help, who all die. Therefore it is obvious that it fails only in incurable cases.” - Galen, Second Century Study Design is Important

7. “All models are wrong, some are useful.” - George E.P. Box

8. Theoretical Model of Potential Studies of Monitoring in Shock MR MO Outcome (e.g. Death) Exposure SHOCK MP MT Chemoprevention Treatment Risk of Shock Progression/Outcome

9. Questions for Monitoring Studies • Does monitoring affect risk of shock among subjects with sepsis, MI? • Can a new modality be used to predict response to therapy to prevent shock? • Does implementation of monitoring lead to less death?

10. Mechanistic Inference vs. Utility • Mechanistic Inference • Risk Factor - “Mediator” • Clinical Utility • Predictor - “Marker” • Part of a “Predictive Model” including clinical indices

11. Questions for Monitoring Studies • Risk Factor - “Mediator” • Mechanism of Disease • Clinical state defined by the measurement independently associated with outcome • Multivariable analysis is focused on adjusting for confounding and interaction • Implies Causality • Goal may be to intervene on the physiological process defined by the measurement • e.g. nitroglycerine for microcirculation

12. Questions for Monitoring Studies • Risk Factor - “Mediator” - Examples • Is elevated plasma IL-6 level independently associated with shock among patients with bacteremia? • Is sublingual capnometry independently associated with mortality in patients with septic shock?

13. Prediction of shock is enticing Babaev JAMA. 2005; 294:448-54.

14. Questions for Monitoring Studies • Predictor - Examples • Does IL-6 level at a particular cutoff predict subsequent development of shock among patients with bacteremia? • Can we use sublingual capnometry to clinically to predict future shock in those without shock? • In these cases the identification of high risk subjects may lead to interventions unrelated to IL-6 or regional circulation • Similar to using BNP in heart disease

15. How to report Prediction • Sensitivity • All those who die are deemed high risk by the predictive rule • Specificity • All those who do not die are deemed low risk by the rule • Positive predictive value • Those deemed high risk die • Negative predictive value • Those deemed low risk don’t die

16. Problem with These Measures • Clinically, we almost never dichotomize patients into: “you will be dead” vs. “you won’t be dead”

17. Accuracy of Prediction • We’d rather ask: • How often are we correct about assigning risk status? • If we say you’re higher risk, are you higher risk than someone who we say is at lower risk? • How close is our prediction to reality? • If we say you have a 10% chance of a complication, do you really have a 10% chance?

18. How often are we correct about assigning risk status? • If we say you’re higher risk, are you higher risk than someone who we say is at lower risk? Discrimination C-statistic = Area under the ROC curve

19. Discrimination = Area Under the ROC curve = C-statistic

20. What does the C-statistic mean? • Given two patients (1 who will have an event and 1 who won’t), how often can you correctly identify, based on risk factors, which one is which? • 100%, perfect discrimination • C-statistic = 1.0 • 50%, perfect guessing (worthless) • C-statistic = 0.5

21. C-statistic • Tells you nothing about calibration • Also, probably tells you nothing about how well you predict • “Relative Ranking of Patients is in correct order” (Justice, Berlin)

22. How close is our prediction to reality? • If we say you have a 10% chance of a complication, do you really have a 10% chance? • Calibration • Hosmer-Lemeshow Statistic • P<0.05 is poor calibration • “There is a significant difference between predicted and observed” • Graphical Plots of Predicted vs. Observed

23. Good Calibration and Discrimination

24. Poor Calibration, Good Discrimination

25. Why Not Both? • The goodness of fit tends to go down as the area under the ROC curve goes up! • Calibration gets worse as discrimination gets better Diamond. J Clin Epidemiol. 1992;45: 85-89

26. What’s Most Important? • Depends on what you want • Clinically • Both • You want to risk stratify your patients • Discrimination • How worried are you about getting the wrong likelihood death? • Calibration

27. Roadmap • Reliability of Measures • Which Questions to Ask? • Mechanistic Inference vs. Prediction • Case heterogeneity within shock syndromes • Refining the Clinical Definition of Shock • Impact studies • Prediction • Effectiveness

28. Sepsis and Cardiogenic Shock • SHOCK Trial 2005 • 18% of subjects with cardiogenic shock had suspected sepsis as cause of shock • indicated by leukocytosis • inappropriately low systemic vascular resistance. • Of these subjects with suspected sepsis, 74% had culture-positive infection Kohsaka Arch Intern Med. 2005; 165:1643-50

29. Heterogeneity in Cardiogenic Shock Hochman, Circulation. 2003;107:2998-3002

30. Heterogeneity of Septic Shock • Late Septic shock has longer duration than Early Septic Shock • Roman-Marchant Chest 2004 • Different infections, predisposing factors • Impaired microvascular flow in resuscitated individuals • Classified as “NO SHOCK”

31. CCEB Do we care if clinical shock definition is heterogeneous? • OK for entry criteria or study of incidence and outcomes • Cardiac revascularization studies use it • Sepsis trials have employed it • However, it is flawed when looking for association when shock is the outcome or comparator variable

32. Theoretical Model of Potential Studies of Monitoring in Shock MR MO Outcome (e.g. Death) Exposure SHOCK MP MT Chemoprevention Treatment Risk of Shock Progression/Outcome

33. Outcome Definition • Fuzzy Outcomes lead to Fuzzy Associations www.furby.com

34. Copeland, AJE, 1977

35. CCEB Outcome Misclassification • Really Important • 10-20% of misclassification leads to double to triple the sample size • Although mortality is not a perfect outcome, it is probably better to study than risk of shock

36. Theoretical Model of Potential Studies of Monitoring in Shock MR MO Outcome (e.g. Death) Exposure SHOCK MP MT Chemoprevention Treatment Risk of Shock Progression/Outcome

37. Roadmap • Reliability of Measures • Which Questions to Ask? • Mechanistic Inference vs. Prediction • Case heterogeneity within shock syndromes • Refining the Clinical Definition of Shock • Impact studies • Prediction • Effectiveness

38. Validity of clinical criteria SBP<90 Despite adequate volume load

39. Validity of clinical criteria SBP<90 Despite adequate volume load

40. CCEB Gold standard for shock? • What are we trying to define here? • Can newer technologies refine the definition to make it more useful? • Enhance predictive validity • Discriminate between those who die and those who will not? • Define patients classified as “Not shock” who would benefit from intervention

41. Perspectivism and “Shock” “We are the music makers and the dreamers of the dreams” - Willy Wonka

42. Perspectivism and Shock Definition • Perspectivism: • “Given the that there is no transcendent truth, how can a philosopher [intensivist] make any claims at all which are valid outside his personal perspective?” • Nietzsche • Can we ever know if our clinical syndrome is what we think it is in the setting of all this uncertainty?

43. Perspectivism and Shock Definition • Perspectivism: • “Given the that there is no transcendent truth, how can a philosopher [Intensivist] make any claims at all which are valid outside his personal perspective?” • Nietzsche • Can we ever know if our clinical syndrome is what we think it is in the setting of all this uncertainty? SBP<90

44. Philosophy of Paradigm • Perspectivism and Paradigm • Paradigm: “collection of beliefs shared by scientists, a set of agreements about how problems are to be understood” • Kuhn: The Structure of Scientific Revolutions • “Shock” is a paradigm we have bought into, and it has taken on a life of its own • But the paradigm which is useful for incidence may not be for mechanistic or therapy studies • Time to shift the paradigm?

45. ALI patterns

46. Evolution of definition of MI • Q-waves • 1979: AHA/WHO consensus conference • Sx, ECG, CK > twice normal • 1990s: Troponin introduced to uncover • “unsuspected myocardial necrosis” • Classified as “No MI” • 2000: AHA/WHO Consensus Conference • Troponin added • NSTSMI nomenclature, etc. defined

47. Evolution of Definition Churg-Strauss • 1951 Allergic Angiitis on autopsy • 1980s-1990s: Antineutrophil cytoplasmic antibodies (ANCA) become available • 2005 French Vasculitis Study Group • ANCA + Churg Strauss • renal involvement, peripheral neuropathy, and biopsy-proven vasculitis • Can this evolution of Syndrome definition happen in Septic Shock? Sable Fourtassou Ann Int Med 2005 Nov 1;143(9):632-8

48. Roadmap • Reliability of Measures • Which Questions to Ask? • Mechanistic Inference vs. Prediction • Case heterogeneity within shock syndromes • Refining the Clinical Definition of Shock • Impact studies • Prediction • Effectiveness

49. Impact and Prediction • Initial mortality prediction studies are available for some of these measures/technologies • Capnography • Respiratory variation • Next steps are to test impact

50. Justice, Ann Int Med 1999;130:515-524.