Alternatives to Randomized Trials for Determining Treatment Efficacy (or Harm)

1. Alternatives to Randomized Trials for Determining Treatment Efficacy (or Harm) Thomas B. Newman, MD, MPH \Clinepi2004\Alt to RCTs for ATCR 3Nov04

2. Outline • Background • Instrumental variables and natural experiments • Measuring additional unrelated variables to estimate bias • Propensity scores • Additional examples from TN’s research (time permitting) • Phototherapy for neonatal jaundice • Natural history of UTI

3. Background • Why do RCTs? • Randomize to assemble comparable groups at baseline (avoid confounding/selection bias) • Blinding to avoid placebo effect, cointerventions, and bias in measuring outcome variable • Observational studies • May be able to assemble comparable groups or use statistical adjustment • Won’t be blinded

4. Why is it hard to assemble comparable groups without randomizing? • By comparable, we mean comparable with respect to prognosis/risk of outcome being studied. • Treated people will often be at higher risk of bad outcome (confounding by indication for treatment). • For some screening tests, screened people may be at lower risk (volunteer bias)

5. When it’s easy • For outcomes not related to indications for treatment there can’t be confounding by indication for treatment • Example: even without an RCT, easy to attribute rhabdomyolysis to statins. But harder to tell effects of statins on stroke without an RCT.

6. Confounding by Indication and Selection Bias • Learning disabilities in children treated with anticonvulsants • Suicide in users of antidepressants • High mortality after surgery for gastroesophageal reflux in children

7. Why cover material here? • Used for clinical, not just etiologic research, so appropriate for Clinical epidemiology as well as classical epidemiology course • We like these designs

8. Natural Experiments and Instrumental Variables • Find a time or place where receipt of treatment was more-or-less random (at least unlikely to be related to prognosis) • E.g., time-series analyses where something changed (e.g. new intervention became available) • Instrumental variables (IV): measurable factors that influence likelihood of treatment that are not otherwise associated with outcome

9. Use of large databases • Allows use of (weak) surrogate measures for actual predictor • Biased towards null • Achieve statistical significance with large sample size • Algebraically reverse bias towards null (with various assumptions)

10. I.V. Example 1: Delayed Effects of the Military Draft on Mortality* • Origin of study: Agent Orange concern • Design: “Randomized natural experiment” using the draft lottery • Data source: computerized death certificate registries, CA and PA • Predictor variable of interest: military service *Hearst N, Newman TB, Hulley SB. NEJM 1986; 314:620-24

11. Why not compare outcomes according to the predictor variable of interest? • Biased comparison – those who serve in the military start out healthier • “Healthy warrior effect”

12. Delayed Effects of the Military Draft on Mortality • The instrumental variablemeasured: draft lottery number below cutoff (based on date of birth) • IV associated with predictor variable of interest, not independently associated with outcome

13. BUT: Having an eligible number was a poor measure of military service:

14. Algebraic Correction 1: • Assume death rates in eligible (RI) and ineligible (RC) men are weighted averages of rates among those serving (A) and not serving (B) • Then if p1 and p2 are proportions serving in the eligible and not eligible groups, Ri and Rc are:RI = p1A + (1-p1)B RC= p2A + (1-p2)B

15. Algebraic correction 2: • What we want to know is the relative risk for military service (A/B) • What we have is the relative risk for draft eligibility (RI/RC) • Then with algebra it can be shown that :A/B = 1-RI/RC + 1 p2RI/RC -p1

16. Results

17. Example 2: Health effects of breast feeding • Can’t do RCT of breast-feeding • Can do RCT of breast-feeding PROMOTION • Need VERY large sample size • Algebraic correction *Kramer MS et al., JAMA 2001

18. Promotion of Breastfeeding Intervention Trial (PROBIT)* • Cluster-randomized trial at 31 sites in Belarus • Subjects 17,046 term singleton infants >2500g initially breastfed • Intervention: WHO/UNICEF Baby Friendly Hospital Initiative • Outcomes: BF @ 3,6,9,12 months and allergic, gastrointestinal and respiratory disease • F/U to 12 months on 16,491 (96.7%)

19. PROBIT, cont’d • RQ#1: Does a breastfeeding promotion program increase exclusive breastfeeding? • Predictor = Group assignment • Outcome = Exclusive breast feeding • ITT analysis is fine • RQ#2: How much does exclusive breastfeeding reduce the risk of atopic eczema in the infant? (What is NNEBF*? ) • Predictor = Exclusive breast feeding • Outcome = Atopic eczema • ITT won’t work -- too much misclassification *Number Needed to Exclusively Breast Feed

20. Results • Exclusive BF at 3 months 40% vs 5%* • Eczema 3.3% vs 6.3%; OR = 0.55 (95% CI 0.31-.95 based on GLIMMIX) • Question: if the risk difference and risk ratio in this study are 3% and 0.55, what can we say about the values for exclusive breast feeding (as opposed to treatment allocation)?*(Rounded)

21. Question: • What is the true effect of breast feeding, undiluted by misclassification bias? • That is, what would the risk ratio and risk difference for eczema have been if everyone in the intervention group and no one in the control group had exclusively breast fed? • Might be relevant for helping a working mother make an informed decision about whether to breastfeed exclusively. (NNEBF)

22. Algebraic correction (simplified) 1 • Assume, as in draft lottery study • There is a rate of eczema for breast fed infants (A) and a different rate for bottle fed infants (B) • These rates are not dependent on group assignment • Then if p1 and p2 are proportions breastfed in the intervention and control groups, the observed rates of eczema in the two groups, Ri and Rc are:RI = p1A + (1-p1)B RC= p2A + (1-p2)B

23. Algebraic correction (simplified) 2 • To obtain the risk difference, we first subtract the two equations: RI = p1A + (1-p1)B RC= p2A + (1-p2)B RC -RI =(p2-p1)A - (p2-p1)B RC -RI =(p2-p1)(A-B) B-A = (RC-RI)/(p1-p2) • So difference in risk of eczema for exclusive BF is: (6.3%-3.3%)/(40%-5%)= 8.6%

24. NNEBF and caveat • Since estimated risk difference is 8.6%, NNEBF to prevent 1 case of eczema is about 12 • Caveat: The algebraic correction only tells us about the effect of breastfeeding in response to the breast feeding promotion intervention. Similarly, effects of draft lottery only apply to those who served as a result of the lottery.

25. Clipping vs Coiling for Cerebral aneurysms Next few slides from Clay Johnston

26. Unadjusted OR= 0.92 (Clipping better), P = .68 Partially adjusted 1.01 BUT: residual confounding still likely -- prognosis of coil patients worse Instrumental variable: proportion of patients treated with coiling at hospital of admission Result: OR = 0.89 per 10% increase in patients treated by coiling Clipping vs Coiling

27. Grouped Treatment Analysis

28. Summary/other examples • If variables known NOT to be associated with outcome are associated with treatment of interest, consider this approach. • Generalizes to many”natural experiments.” • E.g., an intervention is intermittently available, or only available to certain groups. -- different outcome by day of the week, etc.

29. More natural experiments: • Costs of discontinuity of care: increased laboratory test ordering in patients transferred to a different team the next morning • Effect of ER Copay: rates of appendicitis perforation before and after increase in co-pay unchanged.* • Aircraft cabin air recirculation and symptoms of the common cold.** * Hsu J, et al. Presented at Bay Area Clinical Research Symposium 10/17/03 ** Zitter JN et al. JAMA 2002;288:483-6

30. Unrelated variables to estimate bias or confounding • One approach to selection bias: estimate rather than eliminate it • Measure an outcome that WOULD be affected by bias, but not by intervention or • Measure a predictor that WOULD cause the same bias as the predictor of interest (and see if it does)

31. Case control study of screening sigmoidoscopy • Possible bias: patients who agree to sigmoidoscopy are likely to be different • Solution: measure an outcome that would be similarly affected by bias: cancers beyond the reach of the sigmoidoscope • Result: No apparent benefit for cancers beyond the reach of the sigmoidoscope Selby et al, NEJM 1992;326:653-7

32. Effect of British “breathalyser” crackdown • Abrupt drop in accidents occurring during weekend nights (when pubs are open) • Measure an outcome that would be affected by bias: drop in accidents during other times • Result: No difference in accidents occurring during other hours See Cook and Campbell: Quasi-Experimentation.Boston:Houghton Mifflin, p. 219

33. Fenoterol and death from asthma • Case-control studies suggest increase risk of death in users of fenoterol • Risk of confounding by intention to treat -- worse asthma, more frequent beta-agonist use • Measure a predictor that would be affected by bias of concern: other asthma medicines • Result: no association with use of oral or inhaled steroids and smaller association with albuterol Spitzer et al., NEJM 1992;326:501-6; Crane et al., Lancet 1989;1:917-22

34. Calcium Channel Blockers (CCB) and AMI • Population based case-control study at Group Health • Progressive increase in risk of AMI with higher doses of CCB (P <0.01) • Measure a predictor that would be affected by bias of concern: beta-blockers: • Result: progressive decrease in risk associated with higher doses of beta-blockers (P =0.04) Psaty et al., JAMA 1995;274:620-25

35. Effect of Vitamin E on Coronary Heart Disease Risk • Nurses’ Health Study*: • Vitamin E > 100 IU/d x > 2 yr, adjusted RR= 0.57 (.36-.89) • Multivitamins, adjusted RR 0.92 (.73-1.15) • Health Professionals Study** • Vitamin E > 100 IU/ d x > 2 yr, adjusted RR=0.63 (0.47-0.84) • Vitamin C, adjusted RR 0.83 (0.64 to 1.08) *Stampfer et al. NEJM 1993;328:1444-49 **Rimm et al. NEJM 1993;328:1450-56

36. Propensity Scores -1 Big picture: want to know if association between treatment and outcome is CAUSAL Recall competing explanation = confounding by indication for treatment: Factor must be associated with outcome Factor must be associated with treatment Traditional approach: adjust for factors associated with outcome

37. Propensity Scores -2 Alternative approach: Create a new variable, propensity to be treated with the intervention Then either match on that variable or include it in multivariate analyses Advantage: many studies have relatively few outcomes, so less power to identify potential confounders. Receipt of the intervention is much more common, so better power to identify predictors of it. Propensity score is then entered as a single variable (or used for matching) in analysis

38. Example: Aspirin use and all-cause mortality among patients being evaluated for known or suspected Coronary Artery Disease* RQ: Does aspirin reduce all-cause mortality in patients with coronary disease Design: Cohort study Subjects: 6174 consecutive patients getting stress echocardiograms Predictor: ASA use Outcome: All-cause mortality JAMA 2001; 286: 187

39. Analysis using Propensity Scores Two multivariable analyses: Predictors of death Predictors of aspirin use Predictors of ASA use turned into a propensity score estimating probability of ASA use Users and non-users of ASA matched on ASA propensity score ASA propensity score also used in multivariate model

40. Survival in Propensity-Matched Patients

41. Efficacy of Phototherapy (PT) for Neonatal Jaundice • Background: • AAP recommends PT at Total Serum Bilirubin levels ~ 18-25 mg/dL • No RCTs of PT for bilirubin in this range. • Large interfacility practice variation

42. Group R: AAP RECOMMENDS phototherapy Group C: AAP says “CONSIDER” phototherapy Atkinson L, Escobar G, Takayama J, Newman TB. Pediatrics 2003;111:e555-61

43. Instrumental variable: for each hospital, calculate the proportion of newborns in group C who received phototherapy use this proportion as a predictor of TSB >= 20 mg/dL in individual level analyses Individual level analysis: estimate OR for timely phototherapy, by entering key confounders in the model (± propensity scores) Strategies

44. Switch to stata use atcr /*combination of ceph and inst var datasets*/ logistic tbge20 gest_wks asian ma_age black sex,cluster(facil) graph f_bige20 f_yptcon, xlab(0,.1,.2,.3,.4) ylab (0,.01,.02,.03,.04) logistic tbge20 gest_wks asian ma_age black sex f_yptcon, cluster(facil) /*Interpretation of the above: f_yptcon is the proportion of infants in the newborns birth hospital that received phototherapy for a TSB in the AAP "consider" range. So the OR for a unit change in this variable predicts the effect of such treatment (i.e, the effect of being born in a hospital where such treatment is done 100% rather than 0% of the time*/ logistic tbge25 ptlt8hrptbge20lt23 if tbge20lt23==1 & a1tbge20lt23 >=48 & a1tbge20lt23 ~=. & a1tbge20lt23 <= agetbmax & lasteqmax<1 logistic tbge25 ptlt8hrptbge20lt23 rorb4tbge20lt23 if tbge20lt23==1 & a1tbge20lt23 >=48 & a1tbge20lt23 ~=. & a1tbge20lt23 <= agetbmax & lasteqmax<1 logistic tbge25 ptlt8hrptbge20lt23 rorb4tbge20lt23 gest_wks if tbge20lt23==1 & a1tbge20lt23 >=48 & a1tbge20lt23 ~=. & a1tbge20lt23 <= agetbmax & lasteqmax<1 lroc xi: logistic ptlt8hrptbge20lt23 rorb4tbge20lt23 scalpinj gest_wks sex asian black ma_age i.facil if tbge20lt23==1 & a1tbge20lt23 >=48 & a1tbge20lt23 ~=. & a1tbge20lt23 <= agetbmax & lasteqmax<1 predict prop_pt logistic tbge25 ptlt8hrptbge20lt23 prop_pt if tbge20lt23==1 & a1tbge20lt23 >=48 & a1tbge20lt23 ~=. & a1tbge20lt23 <= agetbmax & lasteqmax<1

45. Statistical adjustment: Urine Testing and UTI in the PROS Febrile Infant Study • RQ: what tests should be done to evaluate infants < 3 months old with fever? • Study design: cohort study in the AAP’s Office-based research network • Subjects enrolled by 579 different pediatricians • About half (55%) of the infants had urine tested

46. Subjects • T > 38.0 in office or in previous 24 hr at home. • Age < 3 months • Initially seen by a PROS Practitioner • Enrollment March, 1995 to March 1998 • N = 3,066

47. Urine Testing by Age and Office Temp

48. Multivariate Predictors of UTI

50. Summary Main threat to observational studies of treatment is confounding Confounders are assoc. with both predictor and outcome Instrumental variables are associated with predictor, but not (independently) with outcome Propensity scores allow adjustment for association with predictor If you can’t avoid bias, measure it