Report on the 2008 International Total Survey Error Workshop (ITSEW)

1. Report on the 2008 International Total Survey Error Workshop (ITSEW) Presented at Q2008 Conference Rome, ItalyJuly 11, 2008 Presented by Paul Biemer

2. Acknowledge Other Organization Committee members: Roeland Beerten, ONS Lilli Japec, Statistics Sweden Mary Mulry, Census Bureau Alan Karr, National Institute for Statistical Science Jerry Reiter, Duke University Clyde Tucker, BLS Brian Meekins, BLS

3. Outline • Definition of Total Survey Error • Highlights of the Recent Workshop on Total Survey Error Workshop • Other Emerging Areas of Research • Future Directions of Research on TSE

4. Purpose of the ITSEWs • To stimulate new research on total survey error (TSE) • Provide a forum for researchers to exchange ideas on current research projects • Initiate new projects to advance the TSE field • To generate interest in the field among students and new researchers

5. Organization of the Workshops • Attendance by invitation only • although no one has ever been refused a seat at the workshop • Focus is on on-going research and undeveloped ideas • Midcourse research • Ideas still in incubation • Exploratory research • ITSEW series began 2005 in Washington, DC • ITSEW 2009 will be held in Tälberg, Sweden • National Institute of Statistical Science (NISS) provides funding and staff support • Support from other organizations is welcome

6. 2008 Total Survey Error WorkshopTheme: Multiple Error Sources and their Interactions • Held in Research Triangle Park, NC on June 2-4, 2008 • Organizing committee: • Paul Biemer, RTI and UNC-CH • Roeland Beerten, ONS • Lilli Japec, Statistics Sweden • Mary Mulry, Census Bureau • Alan Karr, NISS • Jerry Reiter, Duke University • Clyde Tucker, BLS • Brian Meekins, BLS • Sponsored by the National Institute for Statistical Science, RTI International and the Survey Research Methods Section of ASA

7. Joop Hox Annica Isaksson Peter Lundquist Noelle Molinari Donsig Jang Scott Fricker Ting Yan Steven Cohen Wendy Hicks Barry Schouten Emilia Peycheva Kristen Olson John Dixon Paul Biemer Catharine Burt Andy Peytchev Brian Meekins Mary Mulry Roger Tourangeau Jerry Reiter Lars Lyberg Rita Thissen Oztas Ayhan, Mojca Bavdaz, Steven Machlin Eric Slud 27 Presenters at the 2008 ITSEW

8. Total Survey Error • Sampling Error • Nonsampling Error • Specification • Nonresponse • Noncoverage • Measurement • Data Processing Mean Squared Error (MSE)

9. TSE: the Concept • Includes both measurement, analysis and reporting of multiple sources of error • Quantify the major components of survey error for a survey • Assess contributions of each source to TSE • Combine these to estimate total MSE • Does not include error reduction methods • E.g., questionnaire design, nonresponse reduction methods, etc.

10. Arguments for Estimating TSE • To correct inferences based upon faulty theory • Biased estimates • Biased measures of uncertainty • To optimize the design of future surveys • To provide quality declarations to users

11. Arguments Against Estimating TSE • Cost • Intractability • Inaccessible methodology • Lack of motivation • Fear

12. Topic Coverage

13. Highlights • Interaction between nonresponse and measurement error • Effect on measurement error of nonresponse followup • Statistical models for simulating the measurement error and nonresponse interaction • Meta-analysis of literature on nonresponse-measurement error interaction • Empirical studies of the relationship between nonresponse and measurement error • Causal factors influencing both response propensity and measurement error • Nonresponse propensity and under-reporting of sensitive topics

14. Other common themes • Frame stratification errors and sampling errors • Frame coverage errors and measurement errors (e.g., cell phone samples and measurement error) • Interviewer effects on nonresponse and measurement errors

15. Gaps in the Research • Data processing errors • Questionnaire design and editing errors • Information content and coding errors • Establishment surveys • Specification error vs. measurement error • i.e., measuring the wrong concept accurately vs. measuring the right concept inaccurately • Need for meta-analysis of existing studies • Better indicators of data quality • Schouten, Bethlehem, et al • More attention to costs of error reduction methods

16. Gaps in the Research (cont’d) • Methods for developing countries • TSE issues in multi-cultural surveys • Methods for estimating variances that reflect multiple error sources, not just sampling errors • Standardized indicators for comparing the quality of survey data • A common language for TSE

17. Need for Data • How can “true values” be obtained? • Paradata for TSE analysis • E.g. level of effort to obtain a response in the broadest sense, condition or value of housing, encounters with gatekeepers • need a systematic / theory-based framework • quality of paradata / “metadata for paradata” • Data sets for TSE analysis are scarce

18. Emerging Research Areas • Interviewer effects in surveys and their effects on TSE • Methods for predicting poor reporters (i.e., bad respondents) • Language translation issues for TSE • Completing the feed-back loop between data processors and survey designers • Methods for estimating the nonresponse-response error interaction

19. Some Initiatives Underway for ITSEW 2009 • Lars Lyberg is organizing ITSEW 2009 in Talberg, Sweden for June 14-17, 2009 • Biemer and Lyberg will edit a special issue of Public Opinion Quarterly in 2010 on TSE. • Special issue on TSE is also being planned for the Journal of Official Statistics in 2010. • Alan Karr has offered NISS services to host a moderated Wikipedia on Total Survey Error

20. Some Initiatives Underway for ITSEW 2009 (cont’d) • Special Topic Sessions for ITSEW 2009 • Nonresponse and measurement error interaction and practical implications (Paul Biemer) • Shared data sets for TSE research (Barry Schouten) • Coverage error (Stephanie Eckman) • POQ and JOS have offered special issues in TSE

21. Some Issues in Research on Nonresponse-Response Error Interaction • Is there an interaction between the propensity to respond and the propensity to respond correctly? • Are reluctant respondents more likely to be misclassified by survey questions? • What can be done to obtain accurate data from reluctant respondents? • If so, what are some strategies for nonresponse followup that minimize total survey error? • What research designs allow analyst to separate nonresponse error and measurement error? • What models are helpful for understanding the linkage between measurement error and nonresponse error?

22. Example: Kristen Olsen’s “Causal” Models T Z T Zε Zp T Y P ε ε Y P P ε Y Measurement Process Model True Value Model Common Cause Model

23. Summary • Papers from the ITSEW 2008 will be available by August 15 on the NISS website • Call for papers for ITSEW 2009 will be announced in September • ITSEW 2009 will be held June 14-17 in Sweden • Papers that examine the interaction between two or more error sources are encouraged.