Is it True: Evaluating Medical Reviews

1. Is it True: Evaluating Medical Reviews

2. “The review article itself should be the product of scientific investigation in which the participants are original investigations (research) rather than patients”

3. Usefulness • Work: Low, good source for POEMs • Relevance: If title and abstract or article conclusion hold promise of POEM, continue • Validity: Uncertain

4. Reviews- Three Basic Types • Textbooks • Academic Reviews • “Translation” Journals

5. Textbooks • Collection of review articles • Minimal, if any, supporting evidence • Questionable validity, long lag time to publish • Average 1-2 years • Most useful for retracing, less hunting and foraging

6. Academic Reviews • Summary: “Broadly paint landscape” • Validity uncertain- begin with conclusions and find supporting references • References often inaccurate and out of date • **Expertise of author varies inversely with quality of review- Oxman/Guyatt** • More later • Must confirm POEMs with original research, increasing work

7. Academic Reviews • Synthesis: Systematic reviews • Meta-analysis or overviews • Answer one or two specific questions • Review primary literature with strict criteria • Conclusions supported by available evidence • Meta-analysis: Achieve power not possible by single study

8. Academic Reviews • Excellent source for hunting and foraging • The Cochrane Library - “Database of Systematic Reviews” • Clinical Evidence (BMJ-BMA) • Clinical Inquiries (FPIN)- SORT • AFP EB Reviews- SORT • Dynamed - SORT • Essential Evidence Plus- SORT

9. Translation Journals • Quick reads for retracing and sporting • Low work, but with low validity, may be zero usefulness • Hunting/foraging: Entering jungle on starless night

10. Translation Journals • Common POEMs need original data for verification, greatly increasing work • Watch for “weasel words”, based on DOEs and anecdotes • “it seems”, “may be effective”, “so one may assume”, “it appears”, “in my experience”

11. Weasel Words Patient Care

12. Translation Journals “Buyer Beware: Unsystematic reviews lead to unsystematic conclusions. Readers looking for a shortcut to understanding evidence about health problems and patient care should at least look for reviews by those who have not taken shortcuts”

13. Determining the usefulness of reviews Onto the worksheets!

14. Reviews: Determining Relevance A. Addressing specific clinical question? A. Patient-oriented evidence? B. Common problem? C. Change your practice?

15. Reviews: Determining Validity • Answer ALL worksheet questions • Stop = “fatal flaw” • Notice how hard this is! Average time for a good systematic review- 2 years! • Much different from your “usual review/CME talk”

16. Worksheet Qs: Finding the studies • Clearly stated? • Terms appropriate? MESH-linked? None missing? • Comprehensive? • Medline + another • MEDLINE misses >50% of articles • Cochrane registry is especially good source • Science Citation Index • Bibliographic review • Unpublished literature • conference abstracts, personal correspondence with important investigators or pharmaceutical companies • Done by more than one person and compared

17. Worksheet Qs: Selecting the studies • Inclusion Criteria • Established a priori • Minimum factors: Population/problem; intervention/comparison; outcomes; study design • Prefer no language restriction • Sometimes validity criteria incorporated (random, blinded, appropriate follow-up, gold standard, etc.) • Best if done independently by 2 investigators • Possibly blinded to author/journal/study results

18. Worksheet Qs:Validity of included studies • Appropriate criteria? • Assurance that criteria specific to type of article employed (therapy, diagnosis, prognosis, etc) • If therapy: randomization, blinding, concealed allocation, follow-up • Process independent by > 2 authors? • Surprising differences! • Why blinding may be important: • 2 sample articles, same study methods • One finds benefit, other does not • “serious flaws” in article without benefit

19. Worksheet Qs: Validity • Were the included studies valid? • Garbage in = garbage out • If yes, no problem • If no, how did authors handle this? • Exclusion/inclusion criteria for quality of study • Subanalysis with comparison of results • Need to consider how these flaws affect results/conclusions

20. Worksheet Qs: Analyzing the data • Homogeneity vs Heterogeneity: just finding the words and an explanation most important • If NOT homogeneous? • Need qualitative explanation. Is it due to chance vs study design, population, exposure, or outcome?

23. Worksheet Qs: Analyzing the data • Appropriateness of combining data: • “Vote” count not usually appropriate • Important to include ‘magnitude’ of the overall effect • Cannot be done without some common ground- outcome • Publication bias • Small, negative trials less likely to be published • Examined by funnel plot • Number needed to change results

24. Funnel plot examples From: Cooper & Hedges: The handbook of research synthesis. 1994

25. From: Cooper & Hedges: The handbook of research synthesis. 1994

26. Reviews: Major Points • Validity traps to avoid • Assertions based on DOEs -- avoid perpetuating medical gossip • Unassessed validity -- Personal experience unreliable as a basis for therapeutic interventions • Missing pieces -- **Quality of the review varies inversely with the expertness of the writer** • Failure to identify level of evidence – Look for LOE’s/ SORT

27. Reviews- Three Basic Types

29. Effect on Patient-Oriented Outcomes • Symptoms • Functioning • Quality of Life • Lifespan SORT A SORT B • Effect on Disease Markers • Diabetes (microalbuminuria, GFR, photocoagulation rates) • Arthritis (ESR, x-rays) • Peptic Ulcer (endoscopic ulcers) SORT C Relevance of Outcome • Effect on Risk Factors for Disease • Improvement in markers (blood pressure, HbA1C, cholesterol) Uncontrolled Observations & Conjecture • Physiologic Research • Preliminary Clinical Research • Case reports • Observational studies • Highly Controlled Research • Randomized Controlled Trials • Systematic Reviews Validity of Evidence