Knowledge Translation: The steep path between evidence generation and application

2. Knowledge Translation:The steep path between evidence generation and application Brian Haynes Health Information Research Unit Dep’t of Clinical Epidemiology and Biostatistics McMaster University

3. KNOWLEDGE IS THE ENEMY OF DISEASE The application of what we know will have a bigger impact on health and disease than any single drug or technology likely to be introduced in the next decade. Sir Muir Gray, UK National Library for Health

4. Knowledge Translation… …the organization, retrieval, appraisal, refinement, dissemination, and uptake of knowledge (eg, important new knowledge from health research)

5. Generalizable knowledge for better clinical practice and healthcare • knowledge from research (sometimes called evidence) • knowledge from the analysis of routinely collected and audit data (sometimes called statistics) • knowledge from the experience of clinicians and patients.

6. Cost-effectiveness of warfarin* • Warfarin for atrial fibrillation • $25CDN saved per stroke averted • Aspirin for atrial fibrillation • $65CDN saved per stroke *Gustafsson C, et al. Cost effectiveness of primary stroke prevention in atrial fibrillation: Swedish national perspective. BMJ. 1992;305:1457-60.

7. What proportion of patients with atrial fibrillation do not receive anticoagulants? 50% Bradley BC, et al. Frequency of anticoagulation for atrial fibrillation and reasons for its non-use at a Veterans Affairs medical center. Am J Cardiol. 2000 Mar 1;85(5):568-72.

8. In Hamilton, Ontario, “The Clot Capital of the Universe,”the proportion of medical inpatients receiving clot prevention according to guidelines is… …33%

9. Current guideline adherence for diabetes Intervention: Ophthalmology assessment… 46% - 80% Proteinuria assessment… 35% - 82% Foot assessment… 30% - 72% HbA1c… 16% - 87% Cholesterol assessment… 55% - 68% Smoking status assessment… 25% - 87%

10. In all, 73% of microalbuminuric patients were not on ACE-I/ARB. Hypertensive type II diabetic patients were often left untreated and only a minority of those treated were optimally controlled. The importance of an elevated systolic pressure is underestimated and the number of antihypertensive drugs prescribed, insufficient. Screening and treatment of albuminuria are inadequate.

11. The routine application of what we know can prevent or minimise: • unknowing variation in clinical practice • errors of commission and omission • unsatisfactory patient experience

12. Evidence (from research) is necessary but, of course, not sufficient… ...it has to be combined with the circumstances of the individual patient and the values of each patient. But without evidence it is improbable that patients, professionals, and those who manage resources, will to make good decisions.

13. researchers decision makers

14. application generation synthesis policy 5 4 decisions 3 1 MRC CIHR a c b 2  Knowledge Translation  Steps from evidence generation to clinical application Steps: 1. generation of evidence from research; 2. evidence summary and synthesis; 3. forming clinical policy; 4. application of policy; 5. individual clinical decisions, including a) patient’s circumstances, b) patient’s wishes, and c) evidence from research

15. Step 1. Generating Research Evidence

16. Step 2. Synthesizing Research Evidence

17. How much synthesis do we need? “..at least 10 000 Cochrane reviews are needed to cover a substantial proportion of the studies relevant to health care that have already been identified” Susan Mallett & Mike Clarke ACP Journal Club. 2003 Jul-Aug;139:A11.

18. When will we have our 10,000 reviews? Growth of Cochrane Reviews and Protocols 2003 Non-Cochrane reviews: >50% of all reviews “…between 2010 and 2015”. Mallett&Clarke, ACPJC 2003 2500 completed mid-2005 protocols 2000 completed mid-2004 1995 reviews

19. Step 3. Developing Policy

20. Step 4. Applying evidence in practice

21. The McMaster PLUS project • only a tiny proportion of all research is “ready for application” • only a tiny fraction of the “ready” research is “relevant” to the practice of a given clinician • only a tiny proportion of the “relevant” research for a given practitioner is “interesting” in the sense of being something new, important, and actionable.

22. Critical Appraisal Filters Evidence-Based Journals ~2,500 articles/y meet critical appraisal and content criteria (95% noise reduction) 50,000 articles/y from 120 journals

23. Clinical Relevancy Filter (MORE) McMaster PLUS Project ~20 articles/yr for clinicians (99.96% noise reduction) ~2,500 articles/y meet critical appraisal and content criteria (95% noise reduction) ~5-50 articles/y for authors of evidence-based clinical topic reviews

24. Dear Dr. Jones,We want to alert you to NEW articles in the PLUS system. These articles that have received very high relevancy and newsworthiness scores: We hope that you will find these articles of value in your clinical practice.Best wishes from the PLUS Team

26. PLUS Trial – Northern Ontario Physicians 344 consent eligible 134 non- respondent 7 refused consent 203 randomized: 10 communities 6 small clusters 4 large clusters Group 1 (3) Group 2 (3) Group 1 (2) Group 2 (2) 2 left study

27. Self Serve Version Ovid Stat!Ref Pyramid of Evidence Full Serve Version Ovid Stat! Ref Pyramid of Evidence PLUS Email Alerts PLUS Search Engine Intervention • Randomization to 2 different trial interfaces

28. RCT begins Control cross-over begins PLUS Preliminary Findings: % of Participants Using PLUS by Month Self-servevsFull-serve Baseline (5 mo) Full-Serve 70 60 50 40 30 20 10 0 Percentage Using PLUS Relative increase 58.7%, P=0.001 Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun 03 03 04 04 04 04 04 04 04 04 04 04 04 04 05 05 05 05 05 05 Month Self-serve Full-Serve

29. Free EBM literature updating service http://www.bmjupdates.com Free at www.bmjupdates.com! (sponsored by BMJ Publishing Group)

30. Step 4. Applying evidence in practice

31. Step 4. Applying evidence in clinical decisions

32. WHO estimates US$100B/yr for health-related research • not enough is for application research • the balance is shifting slowly • should there be a Nobel Prize for applied research?

33. Step 5. Making better clinical decisions

34. Effects of Computerized Clinical Decision Support Systems on Practitioner Performance and Patient Outcomes A Systematic Review Amit Garg MD, Neill Adhikari MD, Heather McDonald MSc, Patricia Rosas-Arellano MD,PhD, Phillip J. DevereauxMD,, Joseph Beyene PhD, Justina Sam, R. Brian Haynes MD, PhD Departments of Clinical Epidemiology and Biostatistics, McMaster University Departments of Medicine, McMaster University, University of Toronto, and University of Western Ontario Department of Biostatistics and Epidemiology, University of Western Ontario Ref: Garg et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005;293:1323-38.

35. Context – Computerized Clinical Decision Support Systems • Software designed to directly aid in clinical decision making in which characteristics of individual patients are matched to a computerized knowledge base for the purpose of generating patient specific assessments or recommendations. Rules / Algorithms • INPUT • Patient characteristics • Automated through EMR • By extra research staff • By existing health care staff • By the patient • By the practitioner Computer • OUTPUT • Recommendations • delivered to health • care provider • Directly by computer • By pager • By extra research staff • By existing health care staff • Outcomes • Provider performance • Patient outcomes integrate into workflow

36. Are CDSSs clinically effective?

37. Did CDSS improve practitioner performance? • 100 studies • “counting positive results on ≥ 50% outcomes measured” • In 16 of 21 (76%) reminder systems • In 24 of 37 (65%) disease management systems • In 19 of 29 (66%) drug dosing or prescribing systems • In 4 of 10 (38%) diagnostic systems Examined in 97 studies, 63 cited improvement (65%)   

38. Did CDSS improve patient outcome? • Update 100 studies • most had inadequate power to detect important difference • none proven to improve definitive outcome such as mortality • surrogate outcomes such as BP and HbA1C not meaningfully improved in most studies Examined in 52 studies, 7 cited improvement (13%)

39. Reminder Systems 40 studies Improved Practitioner Performance - 76% - Improved Patient Outcome - 0% - • Screening, counseling, vaccination, testing, medication use, or the identification of at-risk behaviors • CDSS successes were typically demonstrated in ambulatory care, although one successful system was used in hospitalized patients

40. Disease Management Systems 37 studies Improved Practitioner Performance - 62% - Improved Patient Outcome - 19% - Most are RECOMMENDATIONS. Range of problems, for example: - diabetes care - cardiovascular prevention - incontinence in the elderly - advanced directives - ventilator support - infertility - corollary orders - reduce unneeded health care utilization

41. Step 5. Improving health care decisions

42. The weakest links • Policy - especially at the local level • Coordination - 4P • Helping practitioners to recommend effective treatments • Helping patients to follow effective treatments

43. The strongest link • Organization of health care knowledge according to the hierarchy of evidence (evidence-based medicine)

44. The evolution of Evidence-Based information systems Examples Computerized decision support Evidence-based textbooks Evidence-based journal abstracts Systematic reviews Original journal articles

45. KNOWLEDGE IS THE ENEMY OF DISEASE The application of what we know will have a bigger impact on health and disease than any single drug or technology likely to be introduced in the next decade. Sir Muir Gray, UK National Library for Health