1 / 48

DON’T BELIEVE EVERYTHING YOU READ: AN EDITOR’S VIEW

DON’T BELIEVE EVERYTHING YOU READ: AN EDITOR’S VIEW . JAMES R. SCOTT, MD I have no conflict of interest to disclose. OBJECTIVES. TO CLEARLY UNDERSTAND EVIDENCE BASED MEDICINE TO BE ABLE TO ASSESS VALIDITY OF NEW TREATMENTS

glen
Download Presentation

DON’T BELIEVE EVERYTHING YOU READ: AN EDITOR’S VIEW

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DON’T BELIEVE EVERYTHING YOU READ: AN EDITOR’S VIEW JAMES R. SCOTT, MD I have no conflict of interest to disclose.

  2. OBJECTIVES • TO CLEARLY UNDERSTAND EVIDENCE BASED MEDICINE • TO BE ABLE TO ASSESS VALIDITY OF NEW TREATMENTS • TO APPLY BEST-EVIDENCE IN YOUR CLINICAL PRACTICE

  3. JOHN IOANNIDIS • WHY MOST PUBLISHED RESEARCH FINDINGS ARE FALSE. PloS Med 2005;2(8):124 • “The most downloaded document of all time on PubMed”

  4. SCOTT’S RULE IT WILL NEVER WORK AS WELL IN YOUR PRACTICE AS REPORTED IN THE LITERATURE

  5. "BELIEVE NOTHING THAT YOU SEE IN THE NEWSPAPERS [TV,INTERNET].... IF YOU SEE ANYTHING IN THEM THAT YOU KNOW IS TRUE, BEGIN TO DOUBT IT AT ONCE." Sir William Osler

  6. 16000 1400 14000 1200 12000 1000 10000 800 8000 600 6000 400 4000 200 2000 0 0 1973 1983 1993 2003 1973 1983 1993 2003 EVIDENCE BASED MEDICINE: A RECENT PHENOMENON Meta-Analyses RCTs 2012 – 426,853 RCTs2012 – 53,042 Meta-Analyses

  7. NUMBER OF PUBLISHED PAPERS WITH “EVIDENCE BASED” MEDICINE IN TITLE • 2013 – NOW > 100,000

  8. EVIDENCE-BASED MEDICINE DE-EMPHASIZES • INTUITION • EXPERIENCE • PATHOPHYSIOLOGY EMPHASIZES • PROOF • OUTCOMES

  9. PRINCIPLES OF EVIDENCE-BASED MEDICINE • FIRST DO NO HARM • WHAT ARE THE RESULTS? • ARE THE RESULTS VALID? • HOW DO I APPLY THEM TO MY PATIENTS?

  10. LEVEL OF EVIDENCE I RANDOMIZED CONTROLLED TRIAL – The “Gold Standard” II-2 COHORT OR CASE CONTROL STUDY II-3 LARGE CASE SERIES III CASE REPORT, EXPERT COMMITTEE, RESPECTED AUTHORITY, CLINICAL EXPERIENCE

  11. RULES TO IMPROVE PUBLISHED PAPERS • LEVELS OF EVIDENCE • GUIDELINES – CONSORT, PRISMA, STROBE • TRIAL REGISTRATION • CROSSCHECK FOR PLAGIARISM • DECLARE ALL CONFLICT OF INTERESTS • DETECT FRAUD

  12. PREFERRED STUDIES - RANDOMIZED CONTROLLED TRIALS (RCT) BEST • MOST RELIABLE EVIDENCE • CONSORT GUIDELINES • DESIGNED FOR EFFICACY ONLY • MY RULE - INCLUDE ABSOLUTE RISK • ACTUAL NUMBERS, PERCENTAGES • NUMBER NEEDED TO TREAT (NNT) • NOTE – 25% LATER REFUTED

  13. PROBLEMS TRANSLATING RCTs INTO PRACTICE • DIFFERENT PATIENTS • COSTS MORE • INSURANCE WON’T COVER • PATIENT DOESN’T WANT IT • YOU ARE BETTER AT SOMETHING ELSE • UNANTICIPATED COMPLICATIONS OR SIDE EFFECTS

  14. CHALLENGES WITH NEW SURGICAL AND OBSTETRIC PROCEDURES • RCTs DIFFICULT TO DO • BLINDING NOT POSSIBLE • COHORT/CASE SERIES NEXT BEST • UNETHICAL NOT TO COMPARE WITH CURRENTLY ACCEPTED METHOD CAUTION: IN YOUR OWN PRACTICE USE IRB & INFORMED CONSENT

  15. CLINICAL JUDGEMENT STILL IMPORTANT • WOMEN UNDERGOING ABDOMINAL SACROCOLPOPEXY RANDOMIZED TO BURCH OR NO BURCH(Brubaker et al. N Engl J Med 2006;354:1557) • BURCH LOWERED INCIDENCE OF SUI FROM 44% TO 24% (20%) • REAL WORLD – PT WITH VARICOSITIES  BLEEDING, HEMATOMA & 8 UNITS OF BLOOD

  16. IMPORTANT TO REMEMBER EFFICACY – UNDER IDEAL CONDITIONS EFFECTIVENESS – IN REAL CLINICAL SETTING (YOUR PRACTICE) • NO RCT FITS EVERY PATIENT • ALMOST NO RCT FITS ANY PATIENT PERFECTLY • NOT GOOD AT DETECTING COMPLICATIONS

  17. META-ANALYSIS: STATE-OF-THE-ART REVIEW ADVANTAGES • STANDARD RULES • STATISTICAL POWER DISADVANTAGES • SUBJECTIVE ASSUMPTIONS • COMPLICATED, ARTIFICIAL • ONLY AS GOOD AS THE TRIALS USED • ODDS RATIOS & CONFIDENCE INTERVALS • TAKEN AS GOSPEL

  18. COCHRANE REVIEWS • MOST WELL DONE • PUBLICATION BIAS • LONG & BORING • SOME OUTDATED • LIMITED ACCESS • ODDS RATIOS ONLY

  19. STATISTICS A FELLOW WITH ONE LEG FROZEN IN ICE AND THE OTHER LEG IN BOILING WATER IS COMFORTABLE – ON AVERAGE. J. M. Yancey

  20. REALITY NO RANDOMIZED TRIALS OR META-ANALYSES FOR ABOUT > 50-60% OF WHAT WE DO

  21. NO RANDOMIZED CONTROLLED TRIALS • VBAC • CESAREAN ON REQUEST • SOME GYN SURGICAL PROCEDURES • MANY NEW MATERIALS & DEVICES

  22. EXAMPLE: THERE ARE NO RANDOMIZED TRIALS TO PROVE THAT PARACHUTES WORK RECOMMENDATION: EVIDENCE-BASED GURUS SHOULD PARTICIPATE IN A DOUBLE- BLIND RCT Parachute Use to Prevent Death and Major Trauma. Smith GCS et al. BMJ 2003;327:149

  23. COHORT STUDIES & CASE SERIES STILL USEFUL • MORE CHANCE OF BIAS • STROBE GUIDELINES • HOW IT WORKS IN THE TRENCHES • LONG-TERM FOLLOWUP • FIRST REPORTS OF ADVERSE EVENTS OR RARE COMPLICATIONS

  24. LANDMARK OBSERVATIONAL STUDY FIRST DESCRIPTION OF AIDSBASED ON CLINICAL FINDINGS IN FIVE PATIENTS GOTTLIEB – 33 YR OLD ASSISTANT PROF. SCHROFF – RESIDENT IN INTERNAL MED. Gottlieb MS, Schroff et al. N Engl J Med 1981;305:1425 CITED 2,532 TIMES

  25. SMALL CASE SERIES: OBSTETRIC COMPLICATIONS ASSOCIATED WITH THE LUPUS ANTICOAGULANT D. WARE BRANCH MD JAMES R. SCOTT MD NEIL K. KOCHENOUR MD ED HERSHGOLD MD N ENGL J MED 1985;313:1322

  26. SIGNIFICANCE: Based on 8 Patients • NEWLY RECOGNIZED SYNDROME • TREATABLE CAUSE OF FETAL DEATH • MAJOR ANTIPHOSPHOLIPID ANTIBODY RESEARCH EFFORT NEXT DECADE • CITED 583 TIMES

  27. GOOD CASE REPORT • FIRST SUCCESSFUL EMBOLIZATION FOR SEVERE POSTPARTUM BLEEDING* * BY CHIEF RESIDENT @ U OF UTAH BROWN BJ et al. Uncontrollable Postpartum Bleeding: A New Approach to Hemostasis Through Angiographic Embolization. ObstetGynecol 1979;54:371. • CITED 132 TIMES

  28. WORTHY OF HEALTHY SKEPTICISM • GUEST SPEAKERS • DATABASE STUDIES – Inaccurate • LARGE EPIDEMIOLOGIC STUDIES – Clinically Irrelevant • DECISION ANALYSES – Soft data, assumptions • DRUG COMPANY SPONSORED – Ghostwriters Bias, Overstated conclusions • ADVERTISEMENTS - Embarrassing

  29. ALL TOO COMMON: • WIDELY QUOTED IN MEDIA • SAME DATABASE AS WITH MAGNETIC FIELDS • ? BIOLOGIC PLAUSIBILITY NO INCREASED RISKIN EPIDEMIOL. SAME MO. (no publicity) • 86 Papers – Conflicting Results • MY TAKE: ALL NONSENSE CONCLUSION – COFFEE ASSOCIATED WITH MISCARRIAGE

  30. WAKEFIELD 1998 PAPER IN LANCET RELATED MMR VACCINATIONS IN CHILDREN TO AUTISM • DECLARED FRAUDULANT AND RETRACTED IN 2010. • DID TREMENDOUS HARM • “SHOULD NEVER HAVE BEEN PUBLISHED” Richard Smith, Former Editor of BMJ • SAME PATTERN OCCURING AGAIN MULTIPLE WEAK ASSOCIATIONS --- PUBLICIZED BY MEDIA

  31. LARGE DATABASE • LOGESTIC REGRESSION “MODELING” • Odds Ratios (95% CI) = 1.21 (1.01-1.46) • WIDELY PUBLICIZED BY THE MEDIA CONCLUSION: INDUCTION OF LABOR AND AUGMENTATION ASSOCIATED WITH AUTISM

  32. INTERPRETIVE BIAS AND OVERSTATED CONCLUSIONS MISLEADING RESULTS - “3-FOLD INCREASED RISK …” REALLY 1/MILLION VS 3/MILLION - STATISTICALLY SIGNIFICANT BUT CLINICALLY IRRELEVANT

  33. OTHER BORDERLINE “ASSOCIATIONS” WITH AUTISM DURING PAST YEAR Flu & Fever During Pregnancy Obese Mothers, Weight gain Short Pregnancy Interval Maternal Thyroid Dysfunction ICSI (IVF) Older Fathers Antidepressants, Gluten Sensitivity F.H. of Autoimmune Disease Lyme Disease Air Pollution, Pesticides

  34. TRUTH ALL OF THESE EPIDEMIOLOGIC STUDIES WITH SMALL ODDS RATIOS SHOULD END WITH THE STATEMENT: “WE FOUND A WEAK ASSOCIATION AND WE HAVE NO CLUE WHAT IT MEANS” David Grimes

  35. CAUTION 75% CLINICAL TRIALS INDUSTRY FUNDED PURPOSE - MAXIMIZE FINANCIAL RETURN ONLY 1/3 NOW DONE IN UNIVERSITIES WHO CONTROLLED DATA & WROTE PAPER TIES WITH COMPANIES - MANY AUTHORS & 59% OF EXPERTS WRITING GUIDELINES 5X > CHANCE OF FAVORABLE OUTCOME WHEN COMMERCIALLY SPONSORED

  36. PREDATORY OPEN ACCESS JOURNALS • 8250 JOURNALS – 438 PUBLISHERS, 1/3rd IN INDIA • ADVERTISE FOR PAPERS • CHARGE AUTHOR FEE TO PUBLISH • 304 VERSIONS OF FLAWED AND FICTITIOUS WONDER DRUG PAPER SUBMITTED  ACCEPTED BY 157 (>50%)SCIENCE 2013;342:60-65.

  37. SUBTLE PROBLEMS DESPITE THE RULES AS INDUCTION AND MAINTENANCE THERAPY FOR ULCERATIVE COLITIS. NEJM 2013;369;699 • Complicated RCT – 211 centers in 34 countries • Compared drug vs placebo • 16 Authors – 10 with multiple ties to industry – 7 Company Employees • Company held & analyzed data • No mention of probable cost ($100,000 per year) BOTTOM LINE: MODERATE EFFECT AT 6 & 52 WEEKS

  38. Company sponsored RCTs • Marketing in the guise of research – to publicize expensive drugs • Open Label, No Control Group, Short-term, Physician payment Example: VIOXX

  39. WHAT NEEDS TO HAPPEN: • INDEPENDENT INVESTIGATORS • COMPARE TO PRESENT DRUG/Rx • INCLUDE COST COMPARISON • REJECT EPIDEMIOLOGY STUDIES WITH ODDS RATIOS < 3-4 • REQUIRE ABSOLUTE NUMBERS OR NNT • TRANSPARENCY – DISCLOSE ALL CONFLICTS INCLUDING EDITORS

  40. CLINICALLY RELEVANT • VALID STUDY DESIGN • STRINGENT REVIEW PROCESS • RESULTS PHYSICIANS CAN TRUST

  41. DON’T WASTE YOUR TIME - OTHER RELIABLE AND USEFUL SOURCES • COCHRANE LIBRARY www.cochrane.co.uk • ACOG PRACTICE BULLETINS www.greenjournal.org • UP TO DATE www.uptodate.com • MEDICAL LETTER www.medicalletter.com

  42. CHALLENGE TO TRANSLATE IMPERSONAL & DOGMATIC STATISTICS INTO PERSONALIZED CARE OF REAL FLESH & BLOOD PEOPLE.

  43. STILL IMPORTANT COMPASSION EMPATHY COMMUNICATION CLINICAL JUDGEMENT COMMON SENSE ACCESS

  44. REFERENCES • Scott JR. Show me the evidence. Obstet Gynecol 2002;100(3):403-4. • Ioannidis JPA. Why most published research findings are false. PLoS Med 2005;2(8)e124 • Scott JR. Improving systematic reviews for clinicians: a journal editor’s view. Paediat Perinat Epidemiol 2008;22(1):38-41. • Scott JR. Evidence-based medicine under attack. Obstet Gynecol 2009;113(6):1202-3. • Grimes DA, Schulz KF. False alarms and pseudo-epidemics. The limitations of obsevational epidemiology. Obstet Gynecol 2012;120(4):920-7. • Smith R. The Trouble With Medical Journals. The Royal Society of Medicine Press Ltd. Edward Arnold Publishers. 2011, London, UK • Bohannon J. Who’s afraid of peer review? Science 2013;342:60-7.

More Related