Lost in Translation: Innumeracy in Medicine and Dealing with Uncertainty

1. Lost in Translation: Innumeracy in Medicine and Dealing with Uncertainty Brian Chan MD IM PGY3 Senior Talk Jan 25 2013

2. Objectives • Characterize innumeracy and difficulties in risk communication • Illustrate physicians own innumeracy and approach to uncertainty • Describe tools to deal with innumeracy to help ourselves and our patients • Demonstrate how we use statistical thinking in practice of modern medicine

3. Relevance:

4. Data Requires Context:

5. Case 1: 41 year old female “Ms. Odds’ with history of hypertension, otherwise healthy comes into clinic for follow up. She has a friend whose mother died from breast cancer and is wondering if she should get a screening mammogram…

6. Choices: • She returns a month later, stating she went ahead and got a mammogram from a screening fair-- its come back positive, what is her chance she has breast cancer? A. 90% B. 70% C. 50% D. 30% E. 10% F. 1%

7. Background Data: • Age 40-50 yo women • Incidence of breast cancer in 10 years= 1.4% • Sensitivity of mammogram test (If B+ then M+) = 75% • False positive rate of mammogram (If B- then M+) =10% • If Ms. Odds has a positive mammogram, what is her chance of breast cancer?

8. Choices: A. 90% B. 70% C. 50% D. 30% E. 10% F. 1%

9. Encountering a Cougar

11. Answer: A. 90% B. 70% C. 50% D. 30% E. 10% F. 1% Out of 100 women in age 40 group with a positive screening mammogram, 10 will have diagnosis of breast cancer

12. Who is this man and why should we care? Google 2013

14. Innumeracy • Inability to reason about uncertainties and risk* • Illusion of certainty • Ignorance of risk • Miscommunication of risk • Clouded thinking/drawing inference

15. Uncertainty versus Risk Risk is the quantification of uncertainty into a probability or frequency based on empiric data Quantification of risk can take a number of forms….

16. How We Quantify Risk • Degrees of belief - subjective probability • Propensity - intrinsic properties of an object • Frequency - “incidence” • Relative frequency of event in specified reference class (icu population, population of BMT patients, population of intubated pts) • For frequentists, risks can only be defined in situations where a large body of empirical data exists (what about in situations of few cases)

17. Risk “mis”communication • A. single event probabilities • “you have 30% chance of side effect” • Lack of reference causes confusion B. Comparing treatment effects • RRR - largest effect, drug companies • ARR - how patients like to hear risks from MD • NNT - how we communicate to peers C. Conditional probabilities - sensitivity is not the same as PPV of a test

18. Overcoming innumeracy • 1. Defeat illusion of certainty • 2. Learn about the actual risks of relevant events and actions • 3. Communicate the risk in an understandable way and draw inferences without falling to “clouded thinking”

19. Illustrating innumeracy with my favorite San Diegan Anchorman. Dreamworks picture. Accessed from youtube.

20. Back to our case: Data: age 40-50 yo women P (B+) = 1.4% (incidence) If B+ then M+ = 75% (sensitivity) If B- then M+ =10% (false positive) If Ms. Odds has a M+, what is her chance of B+ P(B+|M+) ?

21. Bayes Theorem • P(disease|positive) = a / a+b • P(disease|positive) = p(disease)p(positive|disease) / • P(disease)p(positive|disease) + p(1-disease)p(positive|no disease)

22. Conditional probabilities versus natural frequencies Method 1: probability (x,y,z) P(disease) = x = .014 P(pos | disease) = y = .75 P(pos | no disease) = z = .10 = xy / (xy + (1-x) z = .014 x .75 / (.014 x .75 + .986 x .10) = 10%

23. Method 2: P(B+|M+) = 12 / (12 + 99) = 10%

24. Physicians And Innumeracy

25. A Way Forward to Numeracy Presenting data in the right context can lead to insight 1. Single event probabilities 2. Conditional probabilities 3. Relative Risks

26. Common Scenarios Adapted from gigerenzer, edwards, “simple tools for understanding risks: from innumeracy to insight.” BMJ 2003

27. Photo break

28. Part II.From Numeracy to Prediction Numeracy allows us to realize uncertainty in medicine Probability and prediction as a waypoint between ignorance and knowledge Prediction = hypothesis testing Separating ‘Signal’ and ‘Noise’

29. Bayes (again): clinical reasoning 1. “Prior” (pre-test probability) - “x” 2. Conditional probability of new data given prior hypothesis is true - “y” 3. Conditional probability of new data given prior hypothesis is false - “z” ________________________________ = Posterior probability xy / [ xy + z(1-x) ]

30. Dealing with uncertainty

31. Probability of a terror attack, given plane crashed into the WTC “prior”: 1 in 20,000 (.005%) terror attacks New event occurs: plane hits WTC probability that a plane hits a building if terrorists were to attack (100%, conservative) Probability that a plane hits a building if terrorists were not to attack (1 in 12,500, .008%) Posterior probability that terror attack occurred given plane crash is now is 38%

32. 2nd plane crash Prior P(terror attack): .005%--> 38% New event: plane crash P(Crash | terror) 100% P(Crash | non-terror) .008% P(terror attack | plane) --> 99% pre-test probability is key

33. Bayes vs Fisher Fisher = frequentism - the p value Bayesian prior is “too subjective” Develop methods to free that bias Uncertainty is derived from sampling, intrinsic to the experiment, not intrinsic to our understanding of the world Collect more data to decrease uncertainty

34. Applicability to clinical medicine • Can we truly leave subjectivity out of clinical reasoning? • Prior experience with patients • Our medical knowledge • Our values of practice

35. Photo break 2

36. Conclusion:Risks of Data Overload Explosion of new diagnostic tests heighten risks for false positives Explosion of published research (most which may be questionable*) - how can we sort through signal and noise? False positives make our predictions more prone to fail “Before we demand more of our data, we need to demand more of ourselves.” Ionnidis Jpa. Why most published research findings are false. Plos med 2005 august;2(8): e124

38. Conclusions (cont) A first step is to appreciate innumeracy exists in patients and clinicians alike. • innumeracy prevents meaningful quantification of uncertainty • Impairs our ability to communicate risk to patients and ourselves

39. Conclusions (cont) Acknowledge subjectivity and uncertainty in medicine, yet strive to minimize bias, minimize irrationality through practice • Making predictions based on our beliefs allows us to test our priors • Use evidence based medicine wisely • Prediction, testing, revision will allow us to converge toward truth/signal

40. In Summary .. [BS] is inevitable in situations that require us to talk or write about something we do not understand. This is one of the dangers in perpetuating the myth of the omnisicent physician In some situations, we may not know with sufficient confidence what is true and what is not. In those cases, we should be as truthful as we can… we must be vigilant in our efforts to assess the truth of what we suppose we know. When we are wrong, we should be the first to admit it. In all cases, it is vital that we commit to veracity.”

41. thanks • Sima desai • Marc gosselin • Elizabeth allen • Laura zeigen - ohsu library services • The senior residents whose shoulders I stand upon: marissa, darcy, sam, sarah, nancy, andre • Jess bordley, my first co-intern • Pete sullivan

42. references • Calculated Risks. Gerd gigerenzer. Simon&schuster. 2002 • The signal and the noise. Nate silver. The pengin press 2012 • Presentation. Is a picture worth a 1000 words? Communicating evidence to patients. Courtesy of elizabeth allen. • Bayes calculator http://psych.fullerton.edu/mbirnbaum/bayes/BayesCalc.htm • http://www.cdc.gov/cancer/breast/statistics/age.htm • Gigerenzer g, galesi m. why do single event probabilities confuse people. BMJ Jan 12. 1-3 • Galesic M, Gerg Gigerenzer. Natural frequencies help older adults and people with low numaracy to evaluate medical tests • Ahmed H, Naik G. Communicating risk. BMJ 2012; 344: e3996. • Kent DM, Hayward RA. Limitations of applying summary results of clinical trial to individual patients. JAMA,Sept 12, 2007- vol 298. 10. • How doctors think. Karthyn montgomery. Excerts. 2006 • Gigerenzer, G, Edwards A. Simple tools uor understanding risks: from innumeracy to insight. BMJ vol 327. 27 September 2003 • Odette w, schwartz L, et al. do physician understand cancer screening statistics? A national survey of primary care physicians in the US. Annals of Internal medicine 2012;156:340-349. • Editorial. What we don’t know can hurt our patients: physician innumeracy and overuse of screening tests. Annals of internal medicine 392-3 • Education and the art of uncertainty. Richard Gunderman. Radiology 2005. • Sonnenberg A, Gogel. “translating vaguecomplaintsinto precise systmes.” euro journal of gastroenterology and hepatology 2002; 14317-321. • Gunderman RB. Bullshit. Journal of the american college of radiology 2010. vol 7, issue 1. p 13-15

43. Innumeracy is a barrier to risk communication • Miscommunication of single event probabilities (lacking reference class) • Ex. 30% chance of side effect • Out of 10 patients, 3 patients tend to report side effect • Risk communication requires a clear statement of a what a probability refers to • Frequency statements can help reduce confusion

44. Use Framing Wisely

45. Final Exam Heme occult test -(ER GI bleed special) Prevalence 0.3% Sensitivity 50% False positive rate 3% Positive predictive value?

46. Example Using Conditional Probabilities* • Probability that person has insulin dependent diabetes 0.5% • IF patient has diabetes, will have a 95% positive screening genetic test. • IF patient does not have diabetes, 50% still test positive on a screening genetic test. • Estimate probability that a patient has diabetes if positive test. • Galesic, Gigerenzer. Natural frequences help older adults and people with low numeracy to evaluate medical screening tests. • Med Decis making 2009 29;368

47. Case 2: • You get a call from the surgical floor intern about a 79 yo patient here for LOA with increasing oxygen requirement likely needing bipap. No other history (of course!). Cxr obtained before night states: • “support hardware noted. Bilateral consolidations are appreciated, concern for pneumonia, chf, hemothorax, ILD, malignancy cannot be excluded. Clinical correlation is advised.”