Handling (and Preventing) Missing Data in RCTs

1. Handling (and Preventing) Missing Data in RCTs ASENTMarch 7, 2009 Janet WittesStatistics Collaborative

2. Topics • Missing values: What, me worry? • Methods of treatment • Methods of prevention • Moral: prevention is better than cure • Or – “The moral of this tale is ‘care’.”

3. Topics turned into learning objectives • Why are missing data important? • What is the distinction between • Ignorable and non-ignorable missing? • MAR and NMAR? • What are general approaches to analysis? • How can we prevent (or minimize) missing?

4. Extent of missing primary outcome data • Cardiovascular outcome trial: 1-2% • Short-term blood pressure trial: 5-10% • 12 week pain trial: 20-40% • 12 week antipsychotic drug: 30-50% • 1 year Alzheimer’s Disease: 20-50% Source: informal experience

5. Why people don’t care about missing data • Many people care about missing data in outcome trials • Many do not care about missing data in symptom trials • Who cares about those who don’t take drug? • “We know the drug won’t work if you don’t take it” • “I am not interested in what happens after people stop.” • No evidence that the two groups differ in Pr{missing} • We are interested in what we observe – “complete cases” • Too hard/expensive to bring back those who stop med

6. So let me convince you to care • What many assume (explicitly or implicitly)

7. What many assume

8. What many assume

9. What statisticians fear (and assume)

10. What statisticians fear (and assume) • Those still in study differ from those randomized • We can’t characterize who is missing • Who is missing differs by group

11. How papers typically report(Hard to ferret out extent and timing of missing) “Missing values imputed using LOCF” (hard to tell that 48% have missing 12 month data)

12. The fundamental dilemma • Want to know: Effect of intervention if everyone took it • Can learn: Effect of intervention among randomized

13. So what question should we ask • What would have been the effect if people were forced to continue? • What is the effect among people who can tolerate it? • What would the effect have been if it were measured?

14. Language – Little/Rubin • Ignorable • Missing completely at random (MCAR): random number selected the missing • Missing at random (MAR) • Given observed data, missingness mechanism does not depend on unobserved. • Rarely definitively determined from the data at hand • Nonignorable: Not missing at random (NMAR)

15. Examples of MAR (Probability of missing depends only on values of observed) • 2 measurements of same variable made at the same time. • If they differ by more than a given amount a third is taken. • 3rd measurement missing for those who do not differ by the given amount. • Subject removed from trial if condition is not sufficiently controlled (criteria pre-defined)

16. Examples of NMAR (Probability of missing depends on value of the missing data) • AD patient too demented to come for measurement • Chronic pain: VAS every two weeks • Measured at Week 4 – pain still bad • Patient feels better at week 5 so doesn’t come at Week 6

17. Rule of thumb: no benefit for missingness Method of imputation shouldn’t give us stronger results than what we would have seen from the complete cases Simple example: More uncertainty but results strengthen. 80 patients: 40/group 20 success in treated and 12 in control P-value (Fisher’s exact) = 0.11 --------------------------------------- 120 patients but 20 in each group missing Assume missing data share results in observed P-value now 0.040

18. Handling missing binary outcomes • Just ignore the missing observations • Impute missing on basis of • Proportion in own group • Best case – all pbo fail; all rx succeed • Worst case – all pbo success; all rx fail • Proportion in placebo group (“not unreasonable guess”) • Proportion in opposite group (“reasonable worst case”) • Multiple imputation

19. Problems with usual approach • Too many degrees of freedom • Some methods overstate effect • Some methods understate effect

20. Loss of 4 points in ADAS-Cog • Two groups – treated and control • 120 per group • 40% in placebo; 20% in treated • Look at relative risk (<1 is “good”) • Missing % equal in both groups

21. Example: loss of 4 points on ADAS-COG

22. Continuous, longitudinal, time-to-event outcomes • Just ignore the missing observations • Impute missing on basis of data in: • Own group • Combined group • Placebo group • Opposite group (“worst reasonable case”)* • Last Observation Carried Forward • Baseline Observation Carried Forward • Last rank carried forward# • Carry forward trajectories • Multiple imputation *Proschan et al (2001)., J Stat Planning 96: 155 # O’Brien, Zhang, Bailey (2005). Stat Med 24:34

23. Effect size and p-values for pain study

24. Message • Analyses produce very different results • Can affect • Direction of effect • Effect size

25. Informed consent documents unclear • Participation in this study is entirely voluntary. Your treatment and your doctor’s attitude toward you will not be affected should you decide not to participate in this study… • You will be asked to return for follow-up visits and to provide follow-up information. • If you agree to participate, you may withdraw from the study at any time without affecting any benefits to which you would otherwise be entitled.

26. Permissive protocols encourage missing data • “Drop-outs will not be replaced” • Suggests that it would be ok to replace them • Suggests that analysis will ignore them • “Expect 10% drop out, therefore increase sample size by 10%” • “The primary analysis will use the intent-to-treat pop” • “The ITT pop is defined as all those randomized who…” • The ITT pop is defined as the evaluable group

27. Language about withdrawal: an outcome trial The reason that a subject discontinues from the study will be recorded in the Case Report Form. A discontinuation occurs when an enrolled subject ceases participation in the study, regardless of the circumstances, prior to completion of the protocol. … The final evaluation required by the protocol will be performed at the time of study discontinuation.

28. Outcome: continuous measure at week 48 • Subjects must be withdrawn from the study (i.e., from any further study medication or study procedure) for the following reasons: • At their own or their legally authorized representative’s request • If, in the investigator’s opinion, continuation in the study would be detrimental to the subject's well-being • Occurrence of an intolerable treatment-emergent adverse event as determined by the investigator and/or the subject • Failure of the subject to return to the study site for scheduled visits • Persistent noncompliance • Pregnancy

29. Prevention of missing values • Revise informed consent forms • Make protocols less permissive • Define outcome measures that don’t allow “success” for missing • E.g., Define measures as success or failure and missing = failure

30. Improved informed consent document • Participation in this study is entirely voluntary. Your treatment and your doctor’s attitude toward you will not be affected should you decided not to participate in this study… • If you agree to participate, you may withdraw from the study at any time without affecting any benefits to which you would otherwise be entitled. • You will be asked to return for follow-up visits and to provide follow-up information even if you are not taking study medication.

31. Protocols • Be vigilant about permissive language • Distinguish between • Stopping meds • Stopping active visits • Withdrawing consent to be followed passively • Understand the importance of full follow-up • (even for those who stop study medication)`

32. Typical language about withdrawal in protocols The reason that a subject discontinues from the study medication will be recorded in the Case Report Form. A discontinuation from the study occurs when an enrolled subject ceases participation a participant in the study dies, is permanently lost to follow-up, or withdraws consent, regardless of the circumstances, prior to completion of the protocol. … An final evaluation required by the protocol will be performed at the time of study discontinuation of study medication.

33. But, if there will be missing data • Choose analytic methods that • Do not add false precision • Are reasonably conservative • Are interpretable • Recognize need for big increase in sample size