Clinicians and Probability

1. Clinicians and Probability Michael A. Kohn 11/16/2006

2. Outline • Review of Bayesian updating described in Chapters 3 and 4 • Discussion of applying Bayesian updating in clinical practice • Course review

3. Probability Updating 1) Convert the patient’s pre-test probability of disease, P(D+), to pre-test odds, Odds(D+): Odds(D+) = P(D+)/[1 – P(D+)] 2) Find the likelihood ratio for the patient’s test result, LR(r): LR(r) = P(r|D+)/P(r|D-) 3) Get post-test odds, Odds(D+|r), by multiplying pre-test odds by likelihood ratio: Odds(D+|r) = Odds(D+) x LR(r) 4) Convert post-test odds to post-test probability: Prob(D+|r) = Odds(D+|r)/[1+Odds(D+|r)]

4. Threshold Probability If B the benefit of treating a D+ individual (or the cost of failing to treat) and C is the cost of treating a D- individual unnecessarily, then Ptt = C/(B+C)

5. Threshold Probability If Ctreat is the treatment cost (in $), and the maximum cost-effectiveness ratio is CER* (in $/bad outcome prevented), then Ptt = Ctreat/(ARR x CER*) Where ARR is the absolute risk reduction of treating D+ individuals, usually obtained from an RCT of the treatment in D+ individuals. By definition D+ individuals have a positive gold standard test. Assumes treatment does not reduce risk at all in D- individuals.

6. Using Test Results Treat if the updated probability of disease is greater than the treatment threshold: Treat when P(D+|r) > Ptt

7. Need 3 Things • Pre-test probability: P(D+) • Likelihood ratio of test result: LR(r) • Treatment threshold: Ptt

8. Example A 58 y.o woman presents to the E.R with an episodic pressing/burning chest pain that began two days earlier for the first time in her life. The pain started while she was walking, radiates to the back and is accompanied by nausea, diaphoresis and mild dyspnea, but is not increased on inspiration. The latest episode of pain ended half an hour prior to her arrival. She has had three normal deliveries and had two abortions. Risk factors: hypertension known for years partially treated (in the past), truncal obesity (height–161 cm, weight–85 Kg ). She denies smoking, diabetes mellitus, hypercholesterolemia or a family history of heart disease. She currently takes no medications On physical examination upon arrival: appears to be in distress, pulse regular 100/min, B.P 135/80, 18 respirations/min, temperature 36.7°. The lungs are clear, the heart sounds are normal with no murmurs or extra sounds, the abdomen is soft with no organomegaly. No pedal edema is noted and the peripheral pulses are normal. On the E.C.G: normal sinus rhythm 101/min, axis 45°, borderline ST elevation of 0.5 mm in leads V2-V4. Admit to telemetry? Cahan A, et al. Qjm. 2003 Oct;96(10):763-9.

9. Homework/Exam Problem Pretest Probability 10% Consider this patient’s ECG “suggestive” Admit if probability of ACS is 1 in 5 or greater

10. Homework/Exam Problem • Pre-test Odds = 0.1/(1-0.1) = 0.11111 • LR(“Suggestive”) = 20%/5% = 4 • Post-test Odds = 0.111 x 4 = 0.4444 • Post-Test Prob = 0.44/(1+0.44) = 31% • 31% > 1/5 = 20%, so admit

11. Homework/Exam Problem Could also calculate no ECG-ECG threshold probability • Threshold Odds = 0.2/(1-0.2) = 0.25 • LR(“Diagnostic”) = 30%/1% = 30 • Pre-test Odds = 0.25/30 = 0.0833 • Post-Test Probability = 0.083/(1+0.0833) = 0.82% If pre-test probability < 0.82%, don’t do ECG.

12. Homework/Exam Problem Could also calculate ECG-admit threshold probability • Threshold Odds = 0.2/(1-0.2) = 0.25 • LR(“Normal”) = 1%/30% = 1/30 = 0.033 • Pre-test Odds = 0.25/0.03330 =7.5 • Post-Test Probability = 7.5/8.5 = 88% If pre-test probability > 88% admit even if ECG is normal.

13. Does any clinician really do this in daily practice?

14. Real Clinical Practice Pretest Probability 10% (????) Consider this patient’s ECG “suggestive” (??) Admit if probability of ACS 1 in 5 or greater (????)

15. Problem 1: Pre-Test Probability • In contrast with the homework examples, pre-test probabilities are never given. • Patients have widely varying pre-test probabilities. • We clinicians are very bad, even irrational, at estimating them.

16. Problem 2: Likelihood Ratios • In contrast with the homework examples, likelihood ratios are rarely known. • We clinicians are very bad, even irrational, about using them.

17. Problem 3: Treatment Thresholds • In contrast with the homework examples, treatment thresholds are never given. • Consequences of error very widely from patient to patient and depend on multiple diverse factors

18. Pre-test probabilities What is the probability (in percents), in your opinion, that the patient has: • Active coronary artery disease? • A dissecting aortic aneurysm? • Reflux esophagitis? • Biliary colic? • Anxiety disorder? Cahan A, et al. Qjm. 2003 Oct;96(10):763-9.

19. Note: Ranges from 1-99% Figure 3. The range of estimated probabilities for each of the five diagnoses suggested to the participants. For each diagnosis, the ranges of `crude' and `standardized' probabilities are shown as the left- and right-hand lines, respectively. The means are shown as dots.

20. Figure 2. Frequency distribution of the total probabilities assigned by participants. The mean total probability was 136.7% (± 53.9%). Sixty-five percent of participants had a total probability > 100% (i.e. exhibited subadditivity)

21. Why so high? Figure 3. The range of estimated probabilities for each of the five diagnoses suggested to the participants. For each diagnosis, the ranges of `crude' and `standardized' probabilities are shown as the left- and right-hand lines, respectively. The means are shown as dots.

22. Using Likelihood Ratios • Prevalence of a disease is 1/1000 • Dichotomous test has false positive rate of 5% • What is the probability of disease in a person with a positive result? (Assume that you know nothing about the person’s symptoms or signs) Casscells W, et al.N Engl J Med. 1978 Nov 2;299(18):999-1001

23. Using Likelihood Ratios Consider 2x2 table method.

24. Using Likelihood Ratios • Pre-test Prob = 1/1000 • Pre-test Odds = 1/999 • ASSUME False positive rate = 1 – Spec and Sens = 100% • LR(+) = 100%/5% = 20 • Post-Test Odds = 1/999 x 20 = 20/999 • Post-Test Prob = 20/1019 = 2% Casscells W, et al.N Engl J Med. 1978 Nov 2;299(18):999-1001

25. Using Likelihood Ratios 11/60 surveyed gave 2% for the answer BUT, 27 gave 95% for the answer, SO Maybe the problem was misunderstanding about what was meant by “false positive rate.” They interpreted it as 1 – PPV. Casscells W, et al.N Engl J Med. 1978 Nov 2;299(18):999-1001

26. Decision-making vs. Probability Estimation A 58 y.o woman presents to the E.R with an episodic pressing/burning chest pain… Ask MDs for pre-test probability of ACS and you get answers ranging from 1% to 99% Ask them to choose a next step*, and 100% will get an ECG. *You could give them options: a) send patient to cath lab, b) admit, c) get ECG, and d) send home.

27. Decision-making vs. Probability Estimation On the E.C.G: normal sinus rhythm 101/min, axis 45°, borderline ST Ask MDs for post-test probability of ACS and you get answers ranging from 1% to 99% Ask them to choose a next step*, and most will get additional tests, such as troponin I, repeat ECG. *You could give them options: a) send patient to cath lab, b) admit, c) get more tests (e.g., troponin I, repeat ECG, and d) send home.

28. MD 1 • Prob ACS: 90% • Next Steps: call cardiology - pt needs admission or at least serial trops then stress testing prior to d/c home.

29. MD 2 • Prob ACS: My estimated probability for aortic arch aneurysm is 5%, acute MI is 10% and unstable angina is 95%. • Next Steps: check troponin, routine labs, CXR (?mediastinum?) and discuss with cards for risk stratification (e.g. stress echo) when possible.

30. MD 3 • Prob ACS: (not stated) • Next Steps: she needs cardiac markers. If she is pain free and her markers are negative and serial EKG's are negative, I'd get a stress echo. Certainly cardiology consult.

31. MD 4 • Prob ACS: Greater than 80% • Next Steps: Troponin, aspirin, consider ntg and/or morphine if the patient is in severe pain, beta-blockade, cxr, repeat ECG in 10 minutes, repeat troponin, [Cardiology consultation, admit]

32. MD 5 • Prob ACS: I'm not good at ascertaining probabilities • Next Steps: Chest x ray, D Dimer, serial troponins, and would be on the phone with cardiology now. ASA would be given as well as a Beta Blocker, and she would be most likely be admitted no matter the outcome of any single test short of coronary angiography.

33. MD 6 • Prob ACS: less than 50% (35%) but too high for me to send home • Next Steps: iv, o2, monitor, ecg (serial), cxr, cbc, chem 7, trop. ASA, beta blockers, nitrates for cp only. regardlessof trop number would call cardiologist to eval.

34. MD 7 • Prob ACS: 80%mi, Pericarditis, 10%, Colecystitis or Coledochalithiasis 10% • Next Steps: Troponin and repeat ecg

35. MD 8 • Prob ACS: I'd give her a high probability (> 85%) of ACS. • Next Steps: I'd go down the therapeutic algorithm - ASA, B-blocker, etc., call Cardiology, and pursue/rule-out alternative diagnoses (GI, vascular causes, etc.) all the while expecting that she would be going to the cath lab. In the absence of another better alternative and with no contraindication to angiography, I think a negative EST would be a false-negative and a waste of time.

36. Why teach Bayes’s rule if we can’t or won’t use it in our clinical practice?

37. Why teach EBD to you? You will -- • develop clinical policies (decision rules and guidelines), • evaluate proposed clinical policies, • develop diagnostic tests, • evaluate diagnostic tests.

38. Does understanding the mathematical process help clinical decision making, even though we don’t use it explicitly in our clinical practice?

39. Course Review

40. Kappa

41. Kappa Observed Agreement: 27/71 = 38% Expected: (20/71)23 + (14/71)13 + (13/71)12 + (18/71)13 + (6/71)(10) = 15.4 15.4/71 = 21.7% Kappa = (38%-21.7%)/(100%-21.7%) 0.209

42. ROC Curves

43. ROC Curves

44. ROC Curves

45. Biases in Studies of Dx Test Accuracy • Overfitting Bias – “Data snooped” cutoffs take advantage of chance variations in derivations set making test look falsely good. • Incorporation Bias – index test part of gold standard (Sensitivity Up, Specificity Up) • Verification/Referral Bias – positive index test increases referral to gold standard (Sensitivity Up, Specificity Down) • Double Gold Standard – positive index test causes application of definitive gold standard, negative index test results in clinical follow-up (Sensitivity Up, Specificity Up) • Spectrum Bias • D+ sickest of the sick (Sensitivity Up) • D- wellest of the well (Specificity Up)

46. Biases in Studies of Screening Tests Biases that occur only when comparing outcomes in those diagnosed by screening with those diagnosed by clinical presentation: Lead-Time (Zero time-point shift) Length (Differing natural history)

47. Biases in Studies of Screening Tests Stage migration bias occurs only when comparing stage-specific outcomes/prognosis. Volunteer bias can occur when comparing entire screened to unscreened (if not randomized to screening) “Sticky diagnosis bias” makes screening look bad by selectively attributing bad outcomes to disease in the screened group.

48. Biases in Studies of Screening Tests Pseudodisease/Overdiagnosis: Compare outcome in those diagnosed with disease rather than in all those screened vs. all those not screened. Kind of like stage migration bias, where there are only two stages: Stage 1 and Stage 0. You are comparing Stage 1-specific outcomes.

49. Multiple Tests

50. RCTs: ARR, NNT, and BOTE CEA