Total Survey Error: Past, Present, and Future

1. Total Survey Error: Past, Present, and Future Robert M. Groves U.S. Census Bureau Lars Lyberg Statistics Sweden and Stockholm University

2. Outline • Evolution of “total survey error” (TSE) • Weaknesses of the TSE framework • Strengths of the TSE framework • Future of surveys and research agenda ideas

3. Deming (1944) “On Errors in Surveys” • American Sociological Review! • First listing of sources of problems, beyond sampling, facing surveys • The 13 factors

4. Deming’s 13 factors -The 13 factors that affect the usefulness of a survey -To point out the need for directing effort toward all of them in the planning process with a view to usefulness and funds available -To point out the futility of concentrating on only one or two of them -To point out the need for theories of bias and variability that correlate accumulated experience

6. Comments on Deming (1944) • Does include nonresponse, sampling, interviewer effects, mode effects, various other measurement errors, and processing errors • Omits coverage errors • Includes nonstatistical notions (auspices) • Includes estimation step errors (wrong weighting) • “Total survey error” not used as a term

7. Sampling Text Treatment of Total Survey Error • Kish, Survey Sampling, 1965 • Graphic on biases • 65 of 643 pages on various errors, with specified relationship among errors

8. Frame biases “Consistent” Sampling Bias Sampling Biases Constant Statistical Bias Noncoverage Nonobservation Nonresponse Field: data collection Nonsampling Biases Observation Office: processing

9. Sampling Text Treatment of Total Survey Error • Särndal, Swensson, Wretman, Model Assisted Survey Sampling, 1992 • Part IV, 124 pp. of 694, coverage, nonresponse, measurement error; omits processing error • Lohr, Sampling Design and Analysis, 1999 • 34 of 500 pages, nonresponse, randomized response

10. Other textbooks • Cochran (1953). Sampling Techniques. 40 pages in concluding chapter on “sources of error in surveys” • Deming (1950). Some Theory of Sampling. Starts with the 1944 factors but then continues with pure sampling • Hansen, Hurwitz and Madow (1953). Sample Survey Methods and Theory, Vol 1. Nine pages on survey errors. • Zarkovich 1966. Quality of Statistical Data.

11. Total Survey Error (1979)Anderson, Kasper, Frankel, and Associates • Empirical studies on nonresponse, measurement, and processing errors for health survey data • Initial total survey error framework in more elaborated nested structure

12. Sampling Variable Error Field Nonsampling Processing Frame Total Error Sampling Consistent Noncoverage Bias Nonobservation Nonresponse Nonsampling Field Observation Processing

13. Survey Errors and Survey Costs (1989), Groves • Attempts conceptual linkages between total survey error framework and • psychometric true score theories • econometric measurement error and selection bias notions • Ignores processing error • Highest conceptual break on variance vs. bias • Second conceptual break on errors of nonobservation vs. errors of observation

14. Mean Square Error construct validity theoretical validity empirical validity reliability Variance Errors of Nonobservation Observational Errors Coverage Nonresponse Sampling Interviewer Respondent Instrument Mode criterion validity - predictive validity - concurrent validity Bias Observational Errors Errors of Nonobservation Coverage Nonresponse Sampling Interviewer Respondent Instrument Mode

15. Nonsampling Error in Surveys (1992), Lessler and Kalsbeek • Evokes “total survey design” more than total survey error • Omits processing error

17. Introduction to Survey Quality, (2003), Biemer and Lyberg • Major division of sampling and nonsampling error • Adds “specification error” (a la “construct validity”) or relevance error • Formally discusses process quality • Discusses “fitness for use” as quality definition

19. Survey Methodology, (2004) Groves, Fowler, Couper, Lepkowski, Singer, Tourangeau • Notes twin inferential processes in surveys • from a datum reported to the given construct of a sampled unit • from estimate based on respondents to the target population parameter • Links inferential steps to error sources

20. Representation Measurement Inferential Population Construct Validity Target Population Coverage Error Measurement Sampling Frame Measurement Error Sampling Error Response Sample Processing Error Nonresponse Error Edited Data Respondents Survey Statistic

21. Key Statistical Developments in Total Survey Error 1 • Errors of observers can be correlated (1902), Karl Pearson • Interpenetrating samples (1946), Mahalanobis • Criteria for true values (1951), Hansen, Hurwitz, Marks and Mauldin • Essential survey conditions, correlated response variance (1959), H-H-Bershad • BC survey model “mixed-error model”(1961), H-H-B

22. Key Statistical Developments in Total Survey Error 2 • Interviewer effects using ANOVA (Kish 1962) • Simple response variance via reinterviews (1964), H-H-Pritzker • Relaxed assumptions of zero covariance of true values and response deviations (1964, 1974), Fellegi • Errors of Measurement (1968), Cochran • Estimating model components via basic study schemes using replication, interpenetreation and combinations of the two (1969), Bailar and Dalenius • Estimating nonsampling variance using mixed linerar models (1978), Hartley and Rao • “Error Profile” of Current Population Survey (1978), Brooks and Bailar • Multi-method multi-trait models on survey measures (1984), Wothke…Browne

23. Key Statistical Developments in Total Survey Error 3 • Measurement of imputation error variance through multiple imputation (1987), Rubin • Total error model for PES (1991), Mulry and Spencer • Measurement Errors in Surveys (1991) Section (Biemer and Stokes..Colm..Fuller and others • attempts to juxtapose psychometric notions with survey statistical notions of measurement error • Latent class model applications to survey errors (late 1990’s), Biemer, Tucker and others • Measurement error effects on analysis (1997) Biemer and Trewin

24. Key Statistical Developments in Total Survey Error 4 • Certain components understudied • coverage error variance • nonresponse error variance • all processing errors • biases in general • The movement from measurement of MSE to CBM, SOP thereby getting a situation where Var becomes an approximation of MSE (error-free processes) • Simultaneous treatment of more than one error source (ITSEW)

25. Summary of the Evolution of “Total Survey Error” • Roots in cautioning against sole attention to sampling error • Framework contains statistical and nonstatistical notions • Most statistical attention on variance components, most on measurement error variance • Late 1970’s attention to “total survey design” (methods and theory for resource allocation to the control of various sources of error • 1980’s-1990’s attempt to import psychometric notions, CASM • Key omissions in research

26. Weaknesses of the Common Usage of “Total Survey Error” • Notably a user perspective is missing • Key quality dimensions are missing in the TSE paradigm • User often cannot or prefers not to question accuracy • The complexity does not invite outside scrutiny of accuracy • Users not really informed about real levels of error or uncertainty • We don’t really know how users perceive information on errors

27. Other Weaknesses of the Total Survey Error Paradigm 1 1. Lack of routine measurements No agency does this Error/quality profiles are useful but rare 2. Ineffective influence on professional standards Little expansion beyond sampling error in practice Press releases on Federal statistics rarely contain even sampling errors Survey error research compartmentalized rather than integrated Methodologists tend to specialize Root causes of error often still missing How about OMB’s requirement of NR bias studies if NR expects to exceed 20%?

28. Other Weaknesses of the Total Survey Error Paradigm 2 3.Large burden on design of some estimators Interpenetration, reinterviews for variance estimation complicated and costly Intractable expressions for some components 4. Some assumptions unrealistic

29. Strengths of the Total Survey Error Framework • Taxonomic decomposition of errors • nomenclature for different components • Separation of phenomena affecting statistics in different ways • variance vs. bias; observation vs. nonobservation; respondent/interviewer/measurement task; processing • Conceptual foundation of the field of survey methodology • subfields defined by errors • Tool to identifying gaps in the research literature • e.g., where are the error evaluation papers on processing?

30. Needed Steps in a Research Agenda for Total Survey Error 1 • Integrating causal models of survey errors • cognitive psychological mechanisms (anchoring, recall decay) • Research on interplay of two or more error sources jointly • e.g., nonresponse and measurement error • Research on the interplay of biases and variances • e.g., does simple response variance increase accompany some response bias reductions (self-administration effects)?

31. Needed Steps in a Research Agenda for Total Survey Error 2 4. Guidance on tradeoffs between quality measurement and quality maximization and between measures and developing error-free processes - how much should we spend on quality enhancement vs. measurement of quality (Spencer, 1985)? 5. Integrating other notions of quality into the total survey error paradigm - if “fitness for use” predominates as a conceptual base, how can we launch research that incorporates error variation associated with different uses? Australian Bureau of Statistics

32. Needed Steps in a Research Agenda for Total Survey Error 3 6. Exploiting a multiple-mode, multiple frame, multiple phase survey world 7. Need for methodological studies to assist the user 8. Costs and risks 9. Develop theories for optimal design of specific operations, design principles 10. More standards?