Missing Inaction: Why Do So Many People Ignore Missing Data in RCTs

1. Missing Inaction:Why Do So Many People Ignore Missing Data in RCTs? Temple-Merck Conference 17-Oct-08 Janet Turk WittesStatistics Collaborative

2. 2 Extent of missing primary outcome data Cardiovascular outcome trial: 1-2% Cancer progression-free survival: 2-20% Short-term blood pressure trial: 5-10% 12 week pain trial: 20-40% 12 week antipsychotic drug: 30-50% 12 week anti-infective: 20-50% Source: informal experience

3. 3 What others assume

4. 4 What others assume

5. 5 What others assume

6. 6 What we fear (and assume)

7. 7 What we fear (and assume) What we have left is different from what was there at first We can�t characterize what is missing What is missing differs by group

8. 8 Evidence of inaction: hard to ferret out extent and timing

9. 9 Rarely apparent in survival curves

10. 10 Time to event

11. 11 What I am not going to talk about MCAR, MAR, not MAR Ignorable/non-ignorable The effect of missing data on inference In sample surveys In experiments In randomized clinical trials Detailed methods of dealing with missing data

12. 12 What I will discuss Once over lightly of the methods at hand Why others don�t care about missing values Why our protocols encourage missing data What we can do to prevent missing data Even though prevention is boring I also assume that the participants in the conference are familiar with a variety of techniques for missing data, some simple and some quite sophisticated. The talk will start with a brief review of these methods and will present some calculations showing the uncertainty in inference that missing data induce. Rather than focussing on approaches for handling missing data, however, most of the talk will address a different set of questions stemming from my observation that many investigators do not feel angst when they see even substantial amounts of missing data in their trial. The talk will summarize the types of missing data we often encounter in trials. Examples include missing outcome data in trials of long-term outcomes; missing partial outcome data in the same type of trials; missing data on symptoms and measurements for trials that study outcomes like pain or blood pressure; missing items when the outcome is a score from a questionnaire with several parts; and structured missing data that arise in trials of vaccine where outcomes are not counted until several months after the last immunization. I will hazard some guesses about the reasons for the apparent lack of concern among many experienced investigators and sponsors. The talk will then discuss suggestions for communicating to sponsors, investigators, study participants, and IRBs the importance of collecting full data even when a participant stops active study medication. I also assume that the participants in the conference are familiar with a variety of techniques for missing data, some simple and some quite sophisticated. The talk will start with a brief review of these methods and will present some calculations showing the uncertainty in inference that missing data induce. Rather than focussing on approaches for handling missing data, however, most of the talk will address a different set of questions stemming from my observation that many investigators do not feel angst when they see even substantial amounts of missing data in their trial. The talk will summarize the types of missing data we often encounter in trials. Examples include missing outcome data in trials of long-term outcomes; missing partial outcome data in the same type of trials; missing data on symptoms and measurements for trials that study outcomes like pain or blood pressure; missing items when the outcome is a score from a questionnaire with several parts; and structured missing data that arise in trials of vaccine where outcomes are not counted until several months after the last immunization. I will hazard some guesses about the reasons for the apparent lack of concern among many experienced investigators and sponsors. The talk will then discuss suggestions for communicating to sponsors, investigators, study participants, and IRBs the importance of collecting full data even when a participant stops active study medication.

13. 13 Underlying principle Our method of imputation shouldn�t give us better results than what we would have seen from the complete cases

14. 14 The cards in our deck: binary outcomes Just ignore the missing observations Impute missing on basis of Proportion in own group Best case � all pbo fail; all rx succeed Worst case � all pbo success; all rx fail Proportion in placebo group (�not unreasonable guess�) Proportion in opposite group (�reasonable worst case�) Multiple imputation

15. 15 Problems with the binary cards Too many degrees of freedom Some methods overstate effect Some methods understate effect Some methods are unreasonably pessimistic

16. 16 Loss of 3 lines of vision Two groups � treated and control 120 eyes per group (one per person) 40% in placebo; 20% in treated Look at relative risk (<1 is �good�) Missing % equal in both groups

17. 17 Loss of 3 lines of vision � impute own group

18. 18 Binary example � loss of 3 lines of vision

19. 19 Binary example � loss of 3 lines of vision

20. 20 Binary example � loss of 3 lines of vision

21. 21 What do binary cards do for us Bad Too many degrees of freedom Some methods overstate effect Some methods understate effect Good Sensible cases provide bounds Multiple imputation (if we have a good model)

22. 22 The cards in our deck: continuous outcomes Just ignore the missing observations Impute missing on basis of mean in: Own group Combined group Placebo group Opposite group (�worst reasonable case�) Last Observation Carried Forward Baseline Observation Carried Forward Last rank carried forward Multiple imputation

23. 23 The cards in our deck: longitudinal Just ignore the missing observations Impute missing by carrying forward Last observation Baseline observation Own group trajectory Placebo trajectory Opposite group trajectory* Last rank# Longitudinal model Multiple imputation *Proschan et al (2001)., J Stat Planning 96: 155 # O�brien, Zhang, Bailey (2005). Stat Med 24:34

24. 24 Longitudinal outcome Pain at Day 4 325 patients per group 250 per group completed 7 point scale Placebo Treated Baseline 5.0 5.0

25. 25

26. 26

27. 27 Continuous outcome Placebo Treated Baseline 5.0 5.0 Day 1 3.7 3.2

28. 28

29. 29 The cards in our deck: survival Censor when missing Assume missing have event At same proportion as own, placebo, or opposite group Need to decide when the imputed event occurs At time of censoring At rate in assigned group

30. 30 Message Analyses produce very different results Can affect Direction of effect Effect size

31. 31 Why people don�t care about missing data in outcome trials In outcome trials we can censor � doesn�t matter what happens after people stop drug

32. 32 Why people don�t care about missing data Outcome trials are different from symptom trials Who cares about those who don�t take drug? �We know the drug won�t work if you don�t take it� �I am not interested in what happens after people stop.� No evidence that the two groups differ in Pr{missing} We are interested in what we observe � �complete cases� Too hard/expensive to bring back those who stop med

33. 33 Informed consent documents unclear Participation in this study is entirely voluntary. Your treatment and your doctor�s attitude toward you will not be affected should you decide not to participate in this study� You will be asked to return for follow-up visits and to provide follow-up information. If you agree to participate, you may withdraw from the study at any time without affecting any benefits to which you would otherwise be entitled.

34. 34 Permissive protocols encourage missing data �Drop-outs will not be replaced� Suggests that it would be ok to replace them Suggests that analysis will ignore them �Expect 10% drop out, therefore increase sample size by 10%� �The primary analysis will use the intent-to-treat pop� �The ITT pop is defined as all those randomized who�� The ITT pop is defined as the evaluable group

35. 35 Language about withdrawal: an outcome trial The reason that a subject discontinues from the study will be recorded in the Case Report Form. A discontinuation occurs when an enrolled subject ceases participation in the study, regardless of the circumstances, prior to completion of the protocol. � The final evaluation required by the protocol will be performed at the time of study discontinuation.

36. 36 Outcome: continuous measure at week 48 Subjects must be withdrawn from the study (i.e., from any further study medication or study procedure) for the following reasons: At their own or their legally authorized representative�s request If, in the investigator�s opinion, continuation in the study would be detrimental to the subject's well-being Occurrence of an intolerable treatment-emergent adverse event as determined by the investigator and/or the subject Failure of the subject to return to the study site for scheduled visits Persistent noncompliance Pregnancy

37. 37 Prevention of missing values Education What is the effect of various analytic methods Why is missing important Revise informed consent forms Make protocols less permissive

38. 38 Education of investigators Important to explain to investigators need for follow-up consequences to the study of failure to follow-up

39. 39 Improved informed consent document Participation in this study is entirely voluntary. Your treatment and your doctor�s attitude toward you will not be affected should you decided not to participate in this study� If you agree to participate, you may withdraw from the study at any time without affecting any benefits to which you would otherwise be entitled. You will be asked to return for follow-up visits and to provide follow-up information even if you are not taking study medication.

40. 40 Protocols Be vigilant about permissive language Distinguish between Stopping meds Stopping active visits Withdrawing consent to be followed passively Explain to investigators the importance of follow-up (even for those who stop study medication)`

41. 41 Typical language about withdrawal in protocols The reason that a subject discontinues from the study medication will be recorded in the Case Report Form. A discontinuation from the study occurs when an enrolled subject ceases participation a participant in the study dies, is permanently lost to follow-up, or withdraws consent, regardless of the circumstances, prior to completion of the protocol. � An final evaluation required by the protocol will be performed at the time of study discontinuation of study medication.

42. 42 But, if there are missing data Choose analytic methods that Don�t add false precision Are reasonably conservative Are interpretable Recognize that need for big increase in sample size Phil Lavori�s rule: 1 missing observation needs three additional So, if you expect 10% missing, inflate sample size by 1/3

43. 43 Conclusion: homework assignment Look at all your protocols Look at all your model informed consent forms Prevent permissive language in the future