Causal inference: emulating a target trial when a randomized trial is not available

1. Causal inference: emulating a target trial when a randomized trial is not available Miguel Hernán departments of epidemiology and biostatistics

2. The situation:We need to make decisions NOW • Treat with A or with B? • Treat now or later? • When to switch to C? • A relevant randomized trial would, in principle, answer each comparative effectiveness and safety question • Interference/scaling up issues aside Hernán - Target trial

3. But we rarely have randomized trials • expensive, untimely, unethical, impractical • And deferring decisions is not an option • no decision is a decision: “Keep status quo” • Question: • What do we do? Hernán - Target trial

4. Answer: We analyze observational data(pre-existing or collected for research) • Epidemiologic studies • Electronic medical records • Administrative claims databases • National registers • Disease registries • Other Hernán - Target trial

5. We analyze observational data • but only because we cannot conduct a randomized trial • Observational studies are not our preferred choice • For each observational study, we can imagine a hypothetical randomized trial that we would prefer to conduct • If only it were possible Hernán - Target trial

6. The Target Trial • An analysis of observational data (e.g., large health care database) can be viewed as an attempt to emulate a hypothetical pragmatic randomized trial • That would answer our question • If the observational study succeeds at emulating the target trial, both studies would yield identical effect estimates • except for random variability Hernán - Target trial

7. Procedure to answer clinical/policy questions • Step #1 • Describe the protocol of the target trial • Step #2 • Option A: Conduct the target trial • Option B • Use observational data to explicitly emulate the target trial • Apply appropriate causal inference analytics to estimate the effects of interest Hernán - Target trial

8. Key elements of the protocol of the target trial • Eligibility criteria • Treatment strategies • randomly assigned at start of follow-up • Randomized assignment • Start/End of follow-up • Outcomes • Causal contrast(s) of interest • Analysis plan Hernán - Target trial

9. The observational study needs to emulate • Eligibility criteria • Treatment strategies • randomly assigned at start of follow-up • Randomized assignment • Start/End of follow-up • Outcomes • Causal contrast(s) of interest • Analysis plan Hernán - Target trial

10. Example • Suppose we use observational data • a large health care claims database • to emulate a target trial • of hormone therapy and heart disease • First we need to outline the protocol of the target trial Hernán - Target trial

11. Hernán - Target trial

12. Target trial emulation is not straightforward: data experts only • When observational data were not collected for research purposes • e.g., “coronary heart disease” may be recorded when a woman was diagnosed with it, or when her physician suspected it and ordered a diagnostic test • Must consult with knowledgeable data users • Time-varying clinical workflows, idiosyncratic coding practices, software versions… Hernán - Target trial

13. Besides expert knowledge of the data • Validation studies to quantify data accuracy • Internal consistency checks to detect problems • Cross-datasets comparisons to better understand coding differences • Let’s say we have consulted with experts and done the above before attempting to emulate the target trial Hernán - Target trial

14. Eligibility criteriaEmulation • Apply same criteria as the target trial to women who at baseline have been included in the database for at least 2 years • Potential problems • Insufficient data to characterize individuals eligible for the target trial • Example: If target trial required baseline screening to exclude prevalent cases, emulation may be hard if database records the performance of a test (for billing purposes) but not its findings Hernán - Target trial

15. Treatment strategiesEmulation • Eligible individuals assigned to the strategy consistent with their baseline data • Strategy 1: women who start estrogen plus progestin therapy • Strategy 2: women who do not start hormone therapy • Excluded: women who start a different hormone therapy • Target trial is typically a pragmatic trial • observational data cannot be used to emulate trials with tight monitoring and enforcement of adherence to the study protocol • cannot emulate a placebo-controlled trial • at most a trial with a “usual care” group Hernán - Target trial

16. Assignment proceduresEmulation of blinding • Generally impossible • individuals in the dataset, and their health care workers, are usually aware of the treatment they receive • Observational data can only emulate target trials without blind assignment • standard for pragmatic trials • not a limitation if the goal is comparing real-world treatment strategies Hernán - Target trial

17. Randomized assignmentEmulation of randomization • Generally requires adjustment for all confounding factors • via matching, stratification or regression, standardization or inverse probability (IP) weighting, g-estimation… • If insufficient information on baseline confounders or we fail to identify them, then successful emulation of the target trial’s random assignment is not possible • Confounding bias Hernán - Target trial

18. Start of follow-upWhen is time zero (baseline)? • In true trials • the time of eligibility and randomization • In emulated trials • the time of eligibility and treatment assignment • Failure to assign time zero correctly may lead to misunderstandings • e.g., immortal time bias, others to be discussed Hernán - Target trial

19. OutcomeEmulation • Use the database to identify women with a diagnosis of coronary heart disease during the follow-up • Potential problem: observational data cannot be generally used to emulate a target trial with systematic and blind outcome ascertainment • Except if outcome ascertainment cannot be affected by treatment history, e.g., if the outcome is mortality independently ascertained from a death registry Hernán - Target trial

20. The observational analysisneeds to emulate • Eligibility criteria • Treatment strategies • randomly assigned at start of follow-up • Randomized assignment • Start/End of follow-up • Outcomes • Causal contrast(s) of interest • Analysis plan Hernán - Target trial

21. Causal contrastEmulation • Truly randomized trials can be used to estimate different types of causal effects • We first define these effects in randomized trials, then their analogs in observational studies Hernán - Target trial

22. Causal contrasts1. Intention-to-treat effect • The effect of being assigned to a strategy, regardless of strategy received • In our target trial, contrast of the outcome distribution under • Assignment to initiation of estrogen plus progestin therapy at baseline vs. • Assignment to no hormone therapy • at baseline Hernán - Target trial

23. Causal contrasts1. Intention-to-treat effect • In some trials, assignment to and initiation of the treatment strategies occur simultaneously • ITT effect is effect of initiation of the treatment strategies • Effect magnitude depends on treatment decisions made after baseline • discontinuation or initiation of the treatments of interest, use of concomitant therapies… • Two trials with the same protocol in similar settings may have different ITT effect estimates with neither of them being biased Hernán - Target trial

24. Causal contrasts2. Per-protocol effect • The effect of receiving the treatment strategies specified in the study protocol • In our target trial, contrast of the outcome distribution under • receiving estrogen plus progestin therapy continuously (unless toxicity or contraindications arise) vs. • receiving no hormone therapy • between baseline and study end Hernán - Target trial

25. Causal contrasts2. Per-protocol effect • Often the implicit target of inference • When investigators say that the ITT effect estimate is biased, they mean biased for the per-protocol effect • Cannot be biased by treatment changes that are consistent with the protocol • If protocol lets physicians make their own treatment decisions when toxicity or contraindications arise, then treatment discontinuation because of toxicity is not a deviation from protocol and should not be adjusted for Hernán - Target trial

26. Causal contrasts3. Other • The effect of receiving strategies other than the ones specified in the study protocol • The per-protocol effect in an alternative trial • In our target trial, contrast of the outcome distribution under • receiving statin and estrogen plus progestin therapy continuously (unless toxicity or contraindications arise) vs. • receiving neither statin nor hormone therapy • between baseline and study end Hernán - Target trial

27. Observational analogIntention-to-treat effect • Contrast of the outcome distribution under • Initiation of estrogen plus progestin hormone therapy vs. • No initiation of hormone therapy • at time zero • This contrast preserves the key feature of the intention-to-treat analysis: groups are defined solely by baseline information • When using prescription data, analog of ITT effect in a target trial • When using dispensing data, analog of ITT effect in a target trial in which everybody initiates their assigned strategy Hernán - Target trial

28. Observational analogPer-protocol effect • Defined identically as that for the target trial • Contrast of the outcome distribution under • receiving estrogen plus progestin therapy continuously (unless toxicity or contraindications arise) vs. • receiving no hormone therapy • between time zero and study end Hernán - Target trial

29. The observational analysis needs to emulate • Eligibility criteria • Treatment strategies • randomly assigned at start of follow-up • Start/End of follow-up • Outcomes • Causal contrast(s) of interest • Analysis plan Hernán - Target trial

30. Analysis plan • Identical for true and emulated trials • Except that no adjustment for baseline confounding is expected in intention-to-treat analysis of true trials • Both true and emulated trials require adjustment for post-baseline confounding and selection bias • Possibly using Robins’s g-methods • Which means longitudinal data on treatment, confounders, and outcomes are required Hernán - Target trial

31. The target trial will be a compromise • between the ideal trial we would really like to conduct and the trial we may reasonably emulate using the available data • The drafting of the protocol of the target trial is typically an iterative process • That requires detailed knowledge of the database Hernán - Target trial

32. Examples of trial emulation using Big Data • Classic cohort study • Nurses’ Health Study • Postmenopausal hormone therapy and coronary heart disease • Electronic medical records - THIN • Statins and coronary heart disease • Static strategies • treat vs.no treat • Claims database - USRDS Medicare (not today) • Epoetin and mortality • Static and dynamic strategies • intervention depends on response to previous intervention Hernán - Target trial

33. EXAMPLE #1Hormone therapy and heart disease • Question • What is the intention-to-treat effect of hormone therapy on the risk of coronary heart disease in postmenopausal women? Hernán - Target trial

34. Answers(shocking discrepancy) • Observational studies • >30% lower risk in current users compared with never users • e.g., HR 0.68 in Nurses Health Study • Grodstein et al. J Women’s Health 2006 • Randomized trial • >20% higher risk in initiators compared with noninitiators • HR 1.24 in Women’s Health Initiative • Manson et al. NEJM 2003 Hernán - Target trial

35. The WHI randomized trialManson et al, NEJM 2003 • Double-blind • Placebo-controlled • Large • >16,000 U.S. women aged 50-79 yrs • Randomly assigned to estrogen plus progestin therapy or placebo • Women followed approximately every year like in many large observational studies • No intervention after baseline Hernán - Target trial

36. WHI: Effect estimatesIntention-to-treat hazard ratio (95% CI) of CHD • Overall 1.23 (0.99, 1.53) • Years of follow-up • 0-2 1.51 (1.06, 2.14) • >2-5 1.31 (0.93, 1.83) • >5 0.67 (0.41, 1.09) • Years since menopause • <10 0.89 (0.54, 1.44) • 10-20 1.24 (0.86, 1.80) • >20 1.65 (1.14, 2.40) Hernán - Target trial

37. Why did observational studies get it “wrong”? • Popular theory: residual confounding • insufficient adjustment for lifestyle and socioeconomic indicators • Corollary: causal inference from observational data is a hopeless undertaking • An alternative theory: Observational and randomized studies asked different questions Hernán - Target trial

38. Randomized trial estimated the intention-to-treat effect • What is the CHD risk in women who initiate hormone therapy compared with women who do not? • Design and analysis: • Women randomly assigned to initiation of hormone therapy or placebo • Analytic approach • Compare risk between incident users and nonusers of hormone therapy Hernán - Target trial

39. Observational studies did not estimate intention-to-treat effect • What is the CHD risk in women who are currently taking hormone therapy compared with women who are not? • Design and analysis: • Women are asked about therapy use • Analytic approach • Compare risk between prevalent users and nonusers of hormone therapy (current users vs. never users) Hernán - Target trial

40. “Current vs. never users” contrast is not clinically relevant • Consider a woman wondering whether to start hormone therapy • The current vs. never contrast does not provide the information she needs • Consider a woman wondering whether to stop hormone therapy • The current vs. never contrast does not provide the information she needs Hernán - Target trial

41. What if we re-analyze the observational study… • … to compare the risk in incident users vs. nonusers? • That is, what if we use the observational data to answer same question as randomized trial? • estimate the observational analog of the intention-to-treat effect • Hernán et al. Biometrics 2005 • Hernán et al. Epidemiology 2008 Hernán - Target trial

42. Effect estimates (ITT hazard ratios) Randomized Observational Women’s Health Initiative Nurses’ Health Study • Overall 1.23 (0.99, 1.53) 1.05 (0.82, 1.34) • Years of follow-up • 0-2 1.51 (1.06, 2.14) 1.43 (0.92, 2.23) • >2 1.07 (0.81, 1.41) 0.91 (0.72, 1.16) • Years since menopause • <10 0.89 (0.54, 1.44) 0.88 (0.63, 1.21) • 10-20 1.24 (0.86, 1.80) 1.13 (0.85, 1.49) • >20 1.65 (1.14, 2.40) -- Hernán - Target trial

43. When same question is asked • No shocking observational-randomized discrepancies for ITT estimates • though wide CIs in both studies • What about the popular hypothesis? Any residual confounding? • Probably, but insufficient to explain the original discrepancy Hernán - Target trial

44. Aside: Analysis can/should be extended in two ways • Causal contrast • Estimate per-protocol effect • Rather than intention-to-treat effect only • Effect measure • Estimate survival (or cumulative risk) curves • Rather than hazard ratios only Hernán - Target trial

45. ITT effect is problematicHernán, Hernández-Díaz. Clinical Trials 2012 • Depends on adherence patterns • Substantial non-adherence in both randomized trial and observational study • Inappropriate for safety outcomes • Not patient-centered • We also estimated per-protocol effect • via IP weighting (more later) • Again no randomized-observational discrepancies • Toh et al. Ann Intern Med 2010 Hernán - Target trial

46. Hazard ratios are problematicHernán. Epidemiology 2010 • Overall hazard ratio is a weighted average of the time-varying hazard ratios • Effect size depends on follow-up length • Hazard ratio may be vary over time because of built-in selection bias Hernán - Target trial

47. End of aside Hernán - Target trial

48. Conclusion: CER based on epidemiologic studies is possible • If high-quality observational data on treatment, outcome, and confounders are available • e.g., the Nurses Health Study • But most observational CER relies on large databases (big, pre-existing data) • Health claims, electronic medical records • Can emulation of a target trial work in that setting? • See next case studies Hernán - Target trial

49. EXAMPLE #2Statins and coronary heart disease • Question • What is the effect of statin therapy on the risk of coronary heart disease? • Extreme example of confounding Hernán - Target trial

50. Hernán - Target trial