Drug and Therapeutics Committee

1. Drug and Therapeutics Committee Session 3. Assessing Medicine Efficacy

2. Drug Efficacy

3. Understand the importance of determining efficacy and evaluating the clinical literature Discuss the major types of medicine study design Describe the key components of a journal article Understand how to evaluate and interpret results of a randomized controlled trial Discuss the use of systematic reviews and meta-analyses in evaluating medicines Objectives

4. Introduction Assessing medicine studies in the clinical literature Systematic review and meta-analysis Activities Summary Outline

5. DTC medicine selection responsibilities— Evaluate and recommend appropriate medicines for the formulary based on efficacy, safety, quality, and cost Screen ineffective or costly medicines that have no increased benefit Introduction

6. The first step in assessing whether a new medicine is effective is to formulate the clinical question that is important to your DTC. This question should be constructed to specify— The patient population (P) The medicine intervention that you are interested in (I) The comparative treatment already available (C) The outcome that is important to clinicians and patients (O) Example: In a diabetic patient over age 60 (P), does metformin (I) compared to glibenclamide (C) reduce the risk of stroke (O)? Getting Started in Evaluating an Article: Formulating the Question

7. What kind of evidence: Secondary research or reviews—overviews, systematic reviews, meta-analysis Studies from primary literature—medicine trials, observational studies, surveys, experiments For the DTC, the most reliable evidence is the systematic review that contains several randomized clinical trials and a meta-analysis Finding it: Depends on your resources Many online databases (e.g., PubMed, Medline, Cochrane) Evidence: What Kind and How to Find It

9. Abstract—summary information about the article Introduction—why the authors decided to do the research Methods—how they did it and how they analyzed their results Results—what they found Discussion—what the results mean, in the author's opinion Conclusion— are these justified by the results? References—list of references used in the study Acknowledgments—study funding sources, potential conflict of interests, authors’ affiliations Assessing the Quality of the Evidence: What Makes a Good Clinical Trial?

10. Evaluation of a Clinical Medicine Study

11. Why was the study done; what is the clinical question? Introductory statement should clearly state the study question and the hypothesis that the authors have decided to test When presented in a negative statement, it is referred to as the “null hypothesis” Example – “Antibiotics do not improve the symptoms and recovery time of upper respiratory tract infections” Checklist for Evaluating Medicine Studies (1)

12. What type of study was done? Many different types of studies Reviews, experiments, trials, and surveys Observational studies Cohort (prospective or retrospective; cross-sectional or longitudinal) Case control (used primarily to evaluate an adverse event thought to be medicine-related) Case series (single case report) Randomized controlled trials (experimental) Randomized controlled trial (RCT) Most reliable study design Should use this type of study - If not used, must determine why they have elected not to use Checklist for Evaluating Medicine Studies (2) Gold Standard

13. Example of an RCT (1) Target population R Intervention Group A Group B % reduction in morbidity or mortality Outcome (example) % reduction in morbidity or mortality Comparison

14. Example of an RCT (2)Meropenem vs. Imipenem/Cilastatin Prophylaxis for 176 patients R Imipenem/Cilastatin Meropenem Intervention % reduction in infection (11.4%) % reduction in Infection (13.6%) Outcome (example) Comparison Manes et al. (2003); Pancreas

15. Example of an RCT (3)Meropenem vs. Imipenem/Cilastatin (Continued) Treatment for 182 patients with lower RTI, UTI and other infections R Imipenem/Cilastatin Meropenem Bacterial Eradication (86%) Efficacy rates (87%) Bacterial Eradication (86%) Efficacy rates (90%) Comparison Fang et al. (2002); Chinese Medical Journal

16. Was the study design appropriate to the research question? Consider whether the study design used is the preferred design for the research question, examples: Testing efficacy—RCT New diagnostic test—RCT plus cohort study Screening—cross-sectional cohort study Prognosis (outcome of a disease over time)—longitudinal cohort study Causation—toxin, ADRs, adverse drug events—case control or cohort study Checklist for Evaluating Medicine Studies (3)

17. Was the design of the study sensible? What specific intervention was being considered and what was it compared with? Example—is it reasonable to compare a new product for hypertension with a half dose of an ACE inhibitor or medicines no longer used because of side-effects (e.g., reserpine)? What outcome was measured in the study and how? Is it an outcome that is clinically important? Checklist for Evaluating Medicine Studies (4)

18. Who is studied? How were the subjects for the study recruited? Who was included in the study? Representative of the population in which the medicine will be used? Who was excluded from the study? Is it likely to lead to false conclusions about the effect of an intervention What was the setting of the study and does it relate to the local environment? Checklist for Evaluating Medicine Studies (5)

19. How well was the study conducted? Was systematic bias avoided or minimized? Very important part of a critical appraisal Important elements to determine how well a study was conducted—some important definitions: Bias–anything that leads to deviation of the results from the truth or processes leading to such deviation Randomization—process of assigning patients to treatment groups (new medicine, comparator medicine, placebo) by chance Observer blind—the person measuring the outcomes in a study is not told what treatment patients have received Checklist for Evaluating Medicine Studies (6)

20. Important definitions (continued) Double-blind—neither the observer or the patient in a trial knows what treatment the patients received Allocation—the process of assigning patients to treatment groups Intention-to-treat population—the total number of patients assigned to receive a particular treatment, irrespective of whether they actually received it or not Confounding factor or variable—a variable that can cause or prevent the outcome of interest, is not intermediate variable, and is associated with the factor under investigation Checklist for Evaluating Medicine Studies (7)

21. Example: Summary of Key Sources of Bias in RCTs (Source: Greenhalgh, T. 1997. British Medical Journal 315:305–08)

22. Features that are most important in determining reliability of a RCT Methods used to randomize patients Blinding (double blind) Are all patients followed up and included in the analysis? Quality of RCTs: What to Look for (1)

23. Randomization and concealment of allocation Assigning patients by chance removes the likelihood that the investigator will select patients, either consciously or unconsciously, for the experimental treatment who are more or less likely to respond to it. If the paper you are reviewing does not tell you how patients were randomized and how the allocation process was concealed, it may have unreliable results. Quality of RCTs: What to Look for (2)

24. Double-blind versus open trials Three people who can influence a trial Patients Treating physician Person measuring the outcome Ideally, patient, physician, and observer all must be unaware of the treatment group. In summary, double-blind, placebo, controlled trials with objectives and outcomes judged by an independent outcome committee are the gold standard. Quality of RCTs: What to Look for (3)

25. Inclusion of all patients in the statistical analysis A trial is less likely to have bias if all patients recruited and allocated to treatment are accounted for. Trials of new interventions that do not report what happened to all patients and do not report an intention-to-treat analysis (includes all recruited patients even if they did not receive treatment) should be treated with more uncertainly than those that do. Quality of RCTs: What to Look for (4)

26. Non-randomized trials—main difference with RCT is that risk of selection bias is higher. Judgment needs to be made concerning the significance of selection bias in the study. If selection bias is significant, then the study must be suspect. No amount of statistical adjustment can overcome the fundamental bias that is introduced by the absence of randomization. Non-Randomized Trials: What to Look For

27. What did the authors think they would find? A trial should be big enough and long enough to have a high chance of detecting an effect of treatment. What did the authors decide was an important difference before they did the trial and how many patients did they calculate they would need ? How are the results described ? Simplest description of trial results is to use proportions, that is, the number of patients in treatment group who have the outcomes compared with the total number in the group. Understanding the Numbers (1)

28. Different types of data need different statistical tests. Comparing the effect of treatment in one group relative to the effect in the other is necessary. Comparing the absolute value of the results in one group with those in the other is critical. The difference in the effects of treatment (if any) can be described as the estimate of effect size: Confidence interval expresses the range of plausible results. P-value expresses the probability that the difference is real and not due to chance. Understanding the Numbers (2)Key Concepts for the DTC to Consider

29. P-value <0.05 indicates that there is— 5% chance (1 in 20 probability) of observing a result that does not truly exist 95% chance that any observed result is a true result (e.g., difference in outcome with different medicines) that exists in the population A 95% CI shows the range of results that 95 times out of 100 will include the mean (i.e., the study result) 5% chance that the true result will fall outside the range The larger the sample size, the smaller the CI, and the more confident we are that these results are reliable Understanding the Numbers (3)P-Values and Confidence Intervals (CI)

30. Understanding the Numbers (4)

31. Event rate (over specified time period) Relative risk (RR) (same as rate ratio or risk ratio) Absolute risk difference No. of patients with event Total no. of patients Event rate in treatment group Event rate in control group* Event rate in treatment group – event rate in control group* Understanding the Numbers (5) * or comparator medicine group

32. Number needed to treat (No. patients that need to be treated during the study period before an effect may be realized) Relative risk reduction (RRR) (this is not a helpful statistic) Odds ratio (OR) ________1___________ Absolute risk difference 1 – Relative risk No. patients who have the event No. who do not have the event Understanding the Numbers (6)

33. An overview of individual studies that contains an explicit statement of objectives, materials, and methods and has been conducted according to explicit and reproducible methodology Use of explicit methods limits bias in identifying and rejecting studies Not same as narrative reviews done to prove a point Conclusions are generally more reliable and accurate because of the methods that are used Large amounts of information can be assimilated quickly by DTC Cochrane Collaboration undertakes many systematic reviews; abstracts are free of charge at www.cohrane.org Systematic Reviews (1)

34. Results of different studies can be formally compared to establish the generalizability of findings and consistency of results. Reasons for heterogeneity (inconsistency in results across studies) can be identified and new hypotheses generated about particular subgroups. Where appropriate, results of individual studies can be statistically combined using meta-analysis to provide a single summary estimate of the effect of an intervention. Systematic Reviews (2)

35. To fully understand and interpret systematic review, one must consider— How the trials included in the systematic review were found, and the potential for publication bias: search strategies and inclusion criteria The use of meta-analysis The use of sensitivity analysis in interpreting the results Analysis re-run with different values for key variables and the impact of these changes on the results is observed Tests the robustness of assumptions made in analysis Interpreting inconsistent results (heterogeneity) Systematic Reviews (3)

36. Refers to the statistical techniques used to combine the results of a clinical trial into a single estimate of effect Can be thought of as a weighted average effect Used to calculate pooled or summary estimates for all metrics (RR, ARD, NNT, OR) Presented graphically as a forest plot Meta-analysis

37. Meta-analysis—Forest Plot Source: Dale, K.M. et al. 2006. Journal of the American Medical Association 295:74–80

38. Comparing medicines Medicine is tested against placebo, not the standard medicine in its class. Medicine is tested against a medicine with poor performance. Insufficient information about the disease outcomes and effects of the medicine study are given Clinically unimportant outcomes may be used. Potential Clinical Study Problems: Objectives

39. Study sample problems Study patients are not representative of the population that will actually take the medicine. Number of participants is too small. Randomization Patients are not randomized correctly to the treatment, control or comparator groups. Dropouts Patients with more side-effects or less effect are more likely to drop out and not complete the study. Potential Clinical Study Problems: Methods (1)

40. Confounding factors have not been controlled rigorously and results of the study may be from the confounding factor Bias introduced by the researcher May have an extremely important effect but be difficult to assess Blinding Blinding is not done effectively Statistical significance of a trial is valid, but clinical significance is weak Potential Clinical Study Problems: Methods (2)

41. Dosing regimens Efficacy and safety are based on one dosage regimen and do not provide opportunity to learn more from various doses. Medicine studies use fixed doses to compare different medicines. Outcomes (end points) Outcomes in the analysis are not optimal or were selected after study completion. Not all outcome measures are reported. Outcome measures may be discordant (some show improvement and others do not). Potential Clinical Study Problems: Methods (3)

42. Study is not subject to a peer-review process Throw away journal Symposia proceedings Study funding is provided by a pharmaceutical company Misleading data presentation and analysis Published data can be represented in different ways and taken out of context. Conclusions do not agree with the results Potential Clinical Study Problems: Results and Conclusions

43. Meta-analysis does not answer the clinical question. The search process and inclusion criteria of articles is not defined and may be incomplete. Negative studies not included so giving overly positive results Appraisal of individual studies not described In meta-analysis, how were data combined statistically and was a sensitivity analysis done? How was inconsistent results (heterogeneity) interpreted? Systematic Review Problems

44. Activity 1–Comparing Antimicrobial Medicines for Pneumonia Activity 2–Interpreting the Data: The Helsinki Heart Study Activity 3–Critically Evaluating an Article Activity 4–Interpreting the Data: A Medicine Trial to Compare Artesunate with Mefloquine to Treat Malaria Activities

45. Critical evaluation of the clinical literature is important but difficult and time consuming. Best source is systematic review but may have to review primary studies. Methodology needs careful evaluation because many studies are subject to bias. Use of clinical literature evaluation techniques will improve the DTC’s ability to assess, evaluate, and select the best medicines for the formulary. Summary