Can Addiction Research Be Trusted?

1. Can Addiction Research Be Trusted? Introducing EBAM*(*Evidence Based Addiction Medicine) Stewart B. Leavitt, PhD, EditorAddiction Treatment Forum Sponsored by an educational grant from Mallinckrodt Inc.

2. EBAM Guides Available • “Can Addiction Research Be Trusted?” – report • Available online at www.atforum.com • Look under Addiction Resources tab • Slides from this presentation also available • Called “EBAM AATOD Slides” • PowerPoint format

3. “EBAM for Practitioners” • Available at… www.atforum.com under “Addiction Resources” tab. • A more technical explanation for persons who need to critically assess research evidence.

4. Can Addiction ResearchBe Trusted?Yes? No? Sometimes? • Research must be critically examined and interpreted for particular clinical needs and patient benefits. • Just because something appears in print does not mean it is worthwhile or valid.

5. Why Should You Care? • Research directly or indirectly affects the practice of addiction medicine on a daily basis. • There is an urgent need forgood science to overcome stigma, prejudice, and misunderstanding. Trepanation once was a “scientific cure” formental illness.

6. What is EBAM?Evidence-Based Addiction Medicine • The use of current best evidence in making decisions about the care of patients. • Involves combining clinical expertise with valid research on a topic of concern. • Differentiates between practices based on sound evidence versus those founded more on 1) tradition, 2) outdated beliefs, or 3) “junk” science.

7. Why EBAM? EBAM approaches allow you to respond more convincingly to the following questions…. • Where did you hear about that treatment? • How do you know the information is worthwhile and true? • What do you propose doing and what results do you expect? • What are the costs versus benefits?

8. Essential Understandings • Medical research is an imperfect science. • Medical writing/reporting is often biased. • Scientific research does not “prove” anything to be true or not true.

9. The Nature of Clinical Research • Compares experimental treatment (intervention) with standard therapy or placebo. • Research subjects represent only a tiny portion of the population in question – statistical analyses estimate effects in the larger population. • All things are possible (some patients recover without treatment), and chance can play an important role in the results.

10. Interpreting Research • Interpretation is a matter of perception. • Not everyone “sees” a research study in the same way. • Becoming a more critical consumer of research is essential.

11. Problems with Medical Research Articles Overall Surveys of research have found startling results… • On average, half or more of journal articles contain errors – from missing data to major design flaws. • Roughly, up to 2/3 of article abstracts may contain data that is either inconsistent with or missing from the main body of the article. • In one investigation, 80% of systematic reviews & meta-analyses examined had serious flaws.

12. Concerns About Addiction Research Studies • Many articles appear to be written for other researchers, rather than practitioners. • There are often too few subjects and/or inadequate effects for accurately detecting significant group differences. • Subjects are often selected to maximize outcome results, but they are not typical patients. • Cases are presented as if they are evidence of causality or applicable to broader groups. • There may be a biased leap of interpretation from laboratory experiments to potential effects in humans.

13. What Really Matters? • “Not everything that can be counted counts, and not everything that counts can be counted. What really mattersis not always obvious.”– Albert Einstein • Just because data were collected, analyzed, and reported does not mean they are relevant to the clinical question at hand – but it mattered to somebody.

14. What Are Significant Outcomes? • Statistical Significance – the likelihood (probability) that the results represent true differences between groups. • Clinical Significance – importance of the results for providing meaningful benefits to patients. • Statistical significance does not automatically transfer to clinical significance.

15. Taking A Closer LookAtClinical Research Articles

16. Anatomy of a Research Article Abstract Introduction Methods Results Discussion Conclusion

17. Abstract • Presents overview: who, what, where, when, why, and how. • Structured abstract is a mini-article… • Summarizes major sections of article. • Limited to 250 words. • Abstract may be inaccurate & misleading.

18. Introduction • States purpose of study, including: • clinical background, • review of previous research, • rationale for current investigation. • Identifies questions (hypotheses) to be investigated in present study.

19. Methods • Defines the patients (subjects) included in the study and the sample sizes. • Describes in detail how the study was conducted (treatment protocol). • Defines outcomes of importance and how they were measured. • Describes how data was analyzed (statistics).

20. Results • Describes in detail what was found: • differences in treatment effects (outcomes), • their significance (p values), • and, confidence limits. • Should show how statistical procedures were performed and any data adjustments (eg, for noncompliant/dropout subjects).

21. Discussion • Recaps results (the “payoff”). • Interprets findings in comparison with other research on the topic. • Describes study limitations. • Notes other factors that might have influenced outcomes.

22. Conclusion • Presents case for “external validity” – how, when, where, and in which patients the results might be applied in clinical settings. • Often provides suggestions for further research on the subject.

23. Hierarchy of EvidenceClinical Therapy Research Systematic Reviews / Meta-Analyses of RCTs. Randomized Controlled Trials (RCTs). Cohort Studies Case-Control Studies Cross-Sectional Surveys Case Reports “Perspectives” Articles

24. “Perspectives” Articles • Aka* overviews, reports, commentary, editorials, or interviews. *Also known as… • Most common in addiction field; often cited as evidence. • Summarize or comment on research, and are the most subject to bias. • Usually, favors one viewpoint (perspective) over another.

25. “The Emperor’s NewPerspective (Clothes)” • “He thought it better to continue the illusion that anyone who couldn’t see his clothes was either stupid or incompetent.” – H.C. Anderson • Sometimes, the higher the “authority” of the source, theless its perspective can be trusted as being valid and unbiased.

26. Case Reports • Aka case histories, case series, or anecdotes. • Draw upon personal observations or medical records to report unexpected events. • Subject to errors in observation or interpretation, or inadequate documentation. • May arrive at unsupported conclusions. • “The plural of anecdote is not evidence.” – Leshner

27. Cross-Sectional Surveys • Aka prevalence or epidemiology studies. • Examines relationships of medical conditions, treatments, and/or other factors in a patient population at a particular time. • Can establish associations but not causality. • May be problems with recall and extraneous factors (confounders).

28. Case-Control Studies • Aka case-referent, case-comparison, or retrospective studies. • Identifies patients with outcome of interest (Cases) and Control patients without the outcome. • Researchers look back in time to see how the two groups differed (especially, regarding treatment or exposure received). • Subject to recall bias or inconsistent records.

29. A “Classic” Case-Control Study • 25 persons (Cases) exposed to Tutankhamen’s tomb in 1923-26. • Compared with 11 explorers in Egypt at that time not exposed (Controls). Results: No significant differences in mean age at death or in years survival after exposure. Conclusion: Mummy’s curse was an urban legend.

30. Cohort Studies • Aka followup, incidence, longitudinal, observational, or prospective studies. • Most common trial in addiction medicine. • An Experimental group (cohort) receives treatment of interest; Control group does not. • Outcome differences between groups are compared and contrasted. • Nonrandomized, many factors uncontrolled.

31. A First (?) Clinical Cohort Trial • In 1774, a ship’s doctor took 12 sailors with scurvy and treated them daily, 2 at a time, with either… • …cider, elixir vitriol, vinegar, sea water, nutmeg, or oranges/lemons. • Compared with untreated patients. • Citrus fruit provided rapid relief.

32. Randomized Controlled Trials • RCTs considered “gold standard.” • Subjects randomized to Experimental and Control conditions; then followed for outcomes. • Extraneous factors (confounders) are equally distributed across groups. • Subjects/investigators usually “blinded.” • May be ethical concerns about Control condition.

33. Systematic Reviews& Meta-Analyses • Gather highest quality evidence on a subject. • Compare and contrast studies to reach summary statements and conclusions. • Meta-analyses combine statistical evidence from multiple studies for greater precision. • Conclusions tend to be reliable and accurate. • Clinical guidelines often come from these studies.

34. A Closer LookAtClinical Research Validity

35. What is Validity? • The degree to which the research represents the most accurate and bias-free treatment effects. • Internal Validity – refers to effects that are valid for the subjects in the particular study. • External Validity – refers to the relevance of a study for typical addiction treatment settings. • Validity is determined by looking at a study’s design and sources of potential bias.

36. What is Bias? • Bias is anything that influences conclusions about the groups under investigation and potentially distorts comparisons. • The question is not if there is bias, buthow much and whether it is sufficient to make the study invalid for a particular purpose.

37. Publication Bias – Trials with “positive” results are more likely to be submitted and published more quickly. Patient Selection – Who was excluded or included? Were groups equivalent? Subject Management – Were groups treated exactly the same, except for the intervention under study? Sources of Potential Bias - 1

38. Sources of Potential Bias - 2 • Randomization – Subjects should have an equal chance of assignment to each study condition. • Blinding (Masking) – Avoids preconceived ideas about treatment effects. • Placebo Effects – Mere expectation of benefit (or harm) may influence outcomes.

39. Sources of Potential Bias - 3 • Confounding Factors – Extraneous factors, known or unknown, may cause certain effects to occur or not to occur. • Compliance/Follow-up – Adherence to protocol and retention in study are critical. • Completeness of Data – Are all or only some subjects analyzed? (ITT vs Per Protocol)

40. Overcoming “Statistophobia” • When most people hear the wordstatistics they want to hide. • It is not necessary to be a biostatistician to understand and assess addiction research.

41. Example: How Large Is TheTreatment Benefit? • RCT comparing 8 mg/d buprenorphine (BUP) vs 80 mg/d methadone (M80). • Average illicit-opioid+ urinalyses at 26 weeks. • M80 produced significant 17% reduction.(55% - 38% = 17%; p < .01) Ling et al. Arch Gen Psychiatry. 1996;53:401-407.

42. Example: How Many Patients Need To Be Treated? • Treatment effect = 17% reduction with M80. • For every 100 patients treated with M80, rather than BUP, 17 more would benefit. • That is… …for every 6 patients (100 ÷ 17) treated with M80, one additional patient would remain illicit-opioid abstinent than if treated with BUP.

43. What The Article Didn’t Say In the 8 mg/d BUP vs 80 mg/d Methadone trial – • The patients selected were ideal; rather than typical. • There was a 95% chance that the “true” treatment effect was between 7% and 28% (vs 17% reported). • Not enough subjects enrolled (n=150) to accurately detect the relatively small treatment effect of 17%. • So many subjects dropped out that the validity should be questioned. (48% M80, 65% BUP gone at 26 wks)

44. Why DoesGood ResearchGo Bad?

45. Researcher-WritersFace Challenges • Insufficient funding – researchers may choose to do “something” when funds are not available to do it “right.” • Inadequate training in research & reporting. • Lack of technical support – many researchers do not have access to data processing and statistical support staff.

46. Researcher-WritersFace Challenges (Cont.) • “Publish or perish” pressures – influence poor quality in design and reporting. • Political agendas – promote certain lines of research looking for “desired” results. • Journal space limitations – foster inadequate descriptions of analyses and results. • “Too many cooks” – multiple authors and peer reviewers complicate the process.

47. Research Writing Run Amok When too many people are involved in a research article, it reminds us that… “A camel is a horse that was designed by a committee.”

48. Dangers of “Data Dredging” • Searching for “significant results” after the study is over and the findings were not as expected or desired (post hoc analysis). • Analyzing the same data in different ways until something “important” turns up. • Cranking-out several articles from a single small clinical study.

49. Laboratory vs Clinical Research • Test tube & animal results do not necessarily apply to people. • “Bench work” can be invaluable for guiding clinical trials in humans. • But the ultimate test is in patients “at the bedside.” ©1980, David Sinclair, PhD - used by permission

50. Distortions in the Mass Media“Medical Propaganda” • Medical journalists usually lack understanding of research methods and shortcomings. • News editors seek sensational health stories to grab the public’s attention. • Research organizations and publications should be more cautious in their reporting of research to the press. • Readers need to become more critical.