What makes a clinical reference evidence-based?

1. What makes a clinical reference evidence-based? Brian S. Alper, MD, MSPH May 15, 2006 Canadian Health Libraries Association

2. Who am I? • “A funny thing happened to me on the way to rural family practice.” • Current roles: • Editor-in-Chief, DynaMed • Medical Director of Clinical Reference Products, EBSCO Publishing, Inc.

4. Our Goal in Medicine • Provide the best care… • Provide patients the best information to guide health care decisions… • Improve health outcomes… • …based on the “truth” – separating medical knowledge from folklore

5. How do we know medicine? • WE = society • MEDICINE = clinical knowledge • Scientific investigation • Original research published in journals • Systematic reviews • EVIDENCE-BASED MEDICINE

6. How do we know medicine? • WE = practicing clinicians • MEDICINE = clinical knowledge • Consultants Colleagues • Lectures Textbooks • Rounds Precepting • Guidelines CME • Experts Experience • PRACTICAL choices selected for efficiency

7. Apprentice-style Memorization Soak up information now Retrieve information during practice Knowledge Mastery Find information when you need it Adult learning Problem-oriented Clinical application “Just-in-Case” vs. “Just-In-Time”

8. Using Traditional Textbooks • Not always written for clinical practice • May not provide best research evidence (Are opinions well supported, or selectively supported?) • May be years out-of-date • Finding specific information within the text can be time-consuming

9. Using Guidelines • Not always written for my clinical practice • May not provide best research evidence (Are opinions well supported, or selectively supported?) • May be years out-of-date • Finding specific information within the text can be time-consuming • May be unavailable for specific question

10. Using Journal Articles • Individual journal articles • may not relate to specific information need • may not provide complete picture • Finding one journal article can be time-consuming, let alone finding all the relevant articles • Articles are written to promote research findings, not often written for clinical application • BUT THIS IS OUR EVIDENCE SOURCE

11. Clinical Questions are Common • Typical primary care physician has • 2 questions for every 3 outpatient visits • 3 questions for every 1 inpatient admission • Ideal: Answer these questions with the best available evidence.

12. Previous research onanswering clinical questions • MEDLINE searches can answer clinical questions from primary care physicians • 43% and 46% questions answered by medical librarian searches • Mean search times were 27 minutes and43 minutes • J Fam Pract. 1996;43:140-144. • Bull Med Libr Assoc. 1994;82:140-146.

13. Barriers to Finding Best Evidence • 6 most important obstacles to answering doctor’s questions with evidence • Time required to find information • Difficulty modifying original question • Difficulty selecting optimal search strategy • Failure of resource to cover the topic • Uncertainty when all relevant evidence found • Inadequate synthesis of multiple bitsof evidence • BMJ 2002 Mar 23;324(7339):710

14. Result • Busy clinicians use references considered • Fast • Likely to provide answers • PREFERRED over evidence databases • By family physicians (BMJ. 1999;319:358-361.) • By family physician residents(Fam Med. 2003;35:257-260.)

15. The Need • An evidence database must be: • Fast • Able to answer most questions To make EBM feasible in routine practice.

16. Evidence-Based Medicine • Definition: Integration of best research evidence with clinical expertise and patient values • Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-Based Medicine. How to Practice and Teach EBM. 2nd ed. London: Harcourt Publishers Ltd. 2000. p. 1.

17. Best Research Evidence • Comprehensive – evidence only known to be best if all the available evidence is known • Valid – critical appraisal determines potentialfor bias • Systematic – selection and evaluation of evidence by protocol reduces author bias, investigator bias • Current – every day new evidence could be best • Synthesized – one study vs. the whole picture

18. Ideal Sources for Clinicians • Comprehensive – covering all relevant articles AND information needs. • Valid/Systematic – driven by best available evidence. • Relevant – focus on clinically relevant information. • Easy to use – organized for use by clinicians. • Affordable

19. Validity (“Truth”) • Consider validity of original research • PLUS validity of translation • Was the best evidence identified? • Was the best evidence selected? • Was the evidence evaluated well? • Is the summary accurate? • Are conclusions based on the underlying evidence?

20. What makes a clinical reference evidence-based?

21. Acknowledgements • David Tovey • Editorial Director, BMJ Knowledge • Editor, Clinical Evidence • Jon Brassey • Editor, TRIP Database • Bruce Arroll • Head of Department, Dept of General Practice and Primary Health Care, University of Auckland

22. What makes a clinical reference evidence-based? Evidence-Based = conclusions based on best available evidence

23. “Evidence-based clinical reference” requires the following: • Systematically identifying all applicable evidence If you don’t identify all the evidence, how can you know the evidence you cite is the best available evidence? Start with the question – systematic search -OR- Start with the evidence – systematic cataloguing

24. “Evidence-based clinical reference” requires the following: • Systematically identifying all applicable evidence • Systematically selecting the most valid, relevant evidence from that identified If author selects…what bias is applied? Protocol to select best available evidence… based on the bias of selecting for rigorous methodology and patient-oriented outcomes

25. “Evidence-based clinical reference” requires the following: • Systematically identifying all applicable evidence • Systematically selecting the most valid, relevant evidence from that identified • Systematically evaluating the selected evidence (critical appraisal) Article abstracts are often wrong or misleading Most randomized trials have significant methodologic limitations.

26. “Evidence-based clinical reference” requires the following: • Systematically identifying all applicable evidence • Systematically selecting the most valid, relevant evidence from that identified • Systematically evaluating the selected evidence (critical appraisal) • Accurately summarizing the evidence and its quality What does it really tell us? Based on what? Level of Evidence labeling – Is it accurate? Most randomized trials have significant methodologic limitations.

27. “Evidence-based clinical reference” requires the following: • Systematically identifying all applicable evidence • Systematically selecting the most valid, relevant evidence from that identified • Systematically evaluating the selected evidence (critical appraisal) • Accurately summarizing the evidence and its quality • Making conclusions dependent on the evidence “proven to be effective in a case report”

28. “Evidence-based clinical reference” requires the following: • Systematically identifying all applicable evidence • Systematically selecting the most valid, relevant evidence from that identified • Systematically evaluating the selected evidence (critical appraisal) • Accurately summarizing the evidence and its quality • Making conclusions dependent on the evidence • Synthesizing multiple bits of evidence for overall conclusion

29. “Evidence-based clinical reference” requires the following: • Systematically identifying all applicable evidence • Systematically selecting the most valid, relevant evidence from that identified • Systematically evaluating the selected evidence (critical appraisal) • Accurately summarizing the evidence and its quality • Making conclusions dependent on the evidence • Synthesizing multiple bits of evidence for overall conclusion • Changing the conclusions when new evidence alters thebest available evidence

30. Systematic Evidence Identification Systematic Literature Search • Commonly called Systematic Review • Start with question, search for best available answers Systematic Literature Surveillance • Also called Research Monitoring • Start with literature being published, determine how best to use

31. Systematic Literature Search • Start with clinical question • Define inclusion criteria • Search for all relevant information • Assess methodology (validity) • Combine results • Determine clinical conclusion • Apply to practice

32. Using Systematic Reviews to Answer Clinical Questions • Cochrane Reviews • may be “gold standard” when available and current • many sources use Cochrane reviews • Other Reviews - check whether process (search, evaluation) is systematic

33. Using Systematic Reviews to Answer Clinical Questions • More difficult for non-RCT answers • Consider additional info • Most valid method for answering an individual question • Not adequate to meet most information needs

34. Systematic Literature Surveillance • Start with defined literature set • Define inclusion criteria • Select all relevant information • Assess methodology (validity) • Summarize results • Determine clinical conclusion

35. Systematic Literature Surveillance - Uses • Alert clinicians to critical info (EBM article summaries) • Catalog for future searches (EBM search engine) • Update clinical reference information (Integrated EBM)

36. Systematic Literature Surveillance - Advantages • Cover all types of information - Diagnosis - Treatment - Etiology - Prognosis • Keep up with current research • Address current questions

37. Systematic Literature Surveillance - Disadvantages • Can miss the big picture • 1 study may be “rebuttal” • “ping-pong” between sources • overemphasis of study results • Need to read collection of studies • Miss older research (unless retrospective literature surveillance)

38. Ideal Sources • Comprehensive – covering all relevant articles AND information needs. • Valid/Systematic – driven by best available evidence. • Relevant – focus on clinically relevant information. • Easy to use – organized for clinician uses. • USE SYSTEMATIC REVIEWS AND SYSTEMATIC LITERATURE SURVEILLANCE

39. Evidence for EBM at the POC • Randomized trial of access to a synthesized evidence database by primary care clinicians • Reference – Annals of Family Medicine 2005 Nov-Dec;3(6):507-513 • www.annfammed.org/cgi/content/full/3/6/507

40. Design and Process principles of database tested 1. Systematic literature surveillance 2. Relevance selection filter 3. Validity selection filter 4. Concise synopsis 5. Direct reference citation 6. Synthesis of evidence (outlines and overviews reflect all evidence summarized) 7. Standardized template 8. Additional content listings (reviews, guidelines)

41. Evidence for EBM at the POC • Randomized trial • Target: 50 primary care clinicians, each asking 20 clinical questions during practice • Randomized on per-question basis to • Usual information sources alone • DynaMed plus usual information sources • Outcomes • Proportion of questions answered • Proportion of questions answered that change clinical decision-making • Time to answer

42. Per-Question Randomization • Solves “Allocation concealment” • Solves “Statistical need” • Per-question analysis is descriptive only because questions are dependent on searcher skill • Within-subject analysis necessary to avoid confounding by searcher skill

43. RCT Results • Among 52 subjects asking 698 questions • 347 questions assigned to Intervention • 263 (75.8%) answered • 351 questions assigned to Control • 250 (71.2%) answered • p = 0.05 (within-subject analysis) • When the synthesized evidence database was used, answer was foundin the database for 69.1% of questions

44. RCT Results • 347 questions assigned to Intervention • 224 (64.6%) answered AND noted to make a difference in clinical decision-making • 351 questions assigned to Control • 209 (59.5%) answered AND noted to make a difference in clinical decision-making • p = 0.01 (within-subject analysis)

45. RCT Results • Median time for searching (Intervention) • 4.95 minutes • Median search time (Control) • 4.98 minutes • No significant difference • Subjects noted they found answers faster, but then read more information so overall time was not reduced

46. Contact Information • Brian S. Alper, MD, MSPH Editor-in-Chief, DynaMed www.DynamicMedical.com 10 Estes St. Ipswich, MA 01938 Editor@DynamicMedical.com Balper@epnet.com (978) 356-6500 ext 749

47. Thank You