Missing Data in Research Studies

1. Missing Data in Research Studies Joseph A. Olsen

4. What do I do about missing data?

5. Introduction • What is certain in life? • Death • Taxes • What is certain in research? • Measurement error • Missing data • Missing data can be: • Due to preventable errors, mistakes, or lack of foresight by the researcher • Due to problems outside the control of the researcher • Deliberate, intended, or planned by the researcher to reduce cost or respondent burden • Due to differential applicability of some items to subsets of respondents • Etc.

6. Some Characteristics of Missing Data • Facets of missing data • Persons • Variables • Occasions • Type of non-response • Unit non-response • Block non-response • Wave non-response • Item non-response • Special non-response problems in longitudinal and clustered data • Attrition/drop-out • Group (e. g. family) member non-response

7. Missing Data Mechanisms (1) • Preliminaries: • Yobs: The non-missing or observed data • Ymiss: The missing or unobserved data • M: Whether the data on a given item for a given case is missing (1) or not (0) • Missing Completely at Random (MCAR) • The probability that an item is missing (M) is unrelated to either the observed (Yobs) or the unobserved (Ymiss) data • Missing at Random (MAR) • The probability that an item is missing (M) may be related to the observed data (Yobs) but is unrelated to the unobserved data (Ymiss) • Missing Not at Random (MNAR) • The probability that an item is missing (M) is related to the (unknown) value of the unobserved data (Ymiss), even after conditioning on the observed data (Yobs)

8. Missing Data Mechanisms (2) • The appropriateness of different missing data treatments depends (among other things) on the underlying missing data mechanism • “Real” missing data can seldom be classified into just one of the three (MCAR, MAR, MNAR) • Because we don’t have access to the missing data (Ymiss), we can not empirically test whether or not the data is MNAR • If we know (or can convincingly argue) that the data is not MNAR, a test of whether the data is MCAR is available (e. g. in SPSS Missing Values Analysis).

9. Missing Data in Research Studies • Missing data mechanism • Missing completely at random (MCAR)—Ignorable • Missing at random (MAR)—Conditionally ignorable • Missing not at random (MNAR)—Nonignorable • Amount of missing data • Percent of cases with missing data • Percent of variables having missing data • Percent of data values that are missing • Pattern of missing data • Missing by design • Missing data patterns • Univariate • Monotonic • File matching • General

10. Goals of a Missing Data Treatment • Preserve the essential characteristics of the data • Distributions of the variables • Relationships among the variables • Maintain the representativeness of the analyzed data • Provide valid statistical inference (control Type I error) • Maximize the statistical power of the study and its statistical analyses (minimize Type II error) • Avoid bias and instability in the parameter estimates and standard errors for statistical models

11. Older Missing Data Treatments (1) • Deletion methods • Listwise deletion (complete case analysis) • Pairwise deletion (available case analysis) • Cold deck imputation • Deterministic, logical, or rule-based imputation • Treat missing data for nominal predictors as an additional category • Hot deck (donor case) imputation • Cluster based methods • Distance based (e. g. nearest neighbor) methods • Mean substitution • (Variable) mean substitution • Mean substitution with added random error • Predictor mean substitution with missing data dichotomy

12. Older Missing Data Treatments (2) • Regression imputation • Regression predicted value imputation • Regression imputation with added random error • Special methods for longitudinal studies and randomized controlled trials • Endpoint only analysis • Last observation carried forward (LOCF) • Intent to treat worst (best) case imputation • Summary growth parameters • Special methods for multi-item scales • Available item method of scale construction • Person mean imputation • Two-way imputation • Two-way imputation with added random error

13. Newer Missing Data Treatments • Modern state-of-the-art missing data treatments for MAR data • Maximum likelihood • Multiple imputation • Cutting edge investigational missing data treatments for MNAR data • Pattern mixture models • Selection models • Shared parameter models • Inverse probability weighting

14. Statistical Analysis with Missing Data • What do you get when you don’t specify what you want? What choices do you have within a given analysis procedure? • Often, listwise deletion is the default (and only) option (SPSS Reliability and GLM) • Listwise default with pairwise and mean substitution as options (SPSS Factor and Regression Analysis) • Pairwise default with listwise option (SPSS Correlation) • Modeling approaches that incorporate missing data handling • Survival models • Mixed effects models • Structural equation models • Missing data treatments carried out prior to analysis • Ad hoc methods (Listwise, pairwise, single imputation, etc.) • Modern methods(Maximum Likelihood, Multiple Imputation)

15. Modern Missing Data Treatments • Maximum likelihood (ML) • Estimates summary statistics or statistical models using all available data • Available in modern structural equation modeling software (Amos, EQS, Lisrel, Mplus, Mx, etc.) • The ML covariance matrix and mean vector can also be obtained from SPSS MVA, and used for standard Regression, Factor analysis, Reliability, and other procedures • There are also freeware and open source programs that can produce the ML covariance matrix and mean vector, usually by using the Expectation Maximization (EM) algorithm (e.g. EMCOV) • Multiple imputation • Imputes individual data values in multiple complete datasets, averaging the results of the statistical analyses across these datasets • Available in the current versions of certain SEM software (Amos, Mplus). • Also available in SPSS (MVA), SAS (Proc MI and MIANALYZE), Stata (mi impute and mi estimate), and stand-alone missing data packages such as SOLAS

16. Why do social scientists use modern missing data treatments so infrequently? • Lack of awareness or familiarity • They are not convinced of the problems with older methods • The statistical literature on missing data is technically daunting • The techniques aren’t incorporated into the standard statistical analysis procedures used by social scientists • Journal reviewers and editors have not required it