Practical Survey Design Strategies for Minimizing MSE

1. Practical Survey Design Strategies for Minimizing MSE Lars Lyberg and Bo Sundgren Statistics Sweden lars.lyberg@scb.se bo.sundgren@scb.se

2. The Survey Process Research Objectives Concepts Population revise revise Mode of Administration SamplingDesign QuestionsQuestionnaire Data Collection Data Processing Analysis/Interpretation

3. New goals and organization at Statistics Sweden • Standardized processes, methods and tools • Centralized methodological and IT staff • User orientation • World class vision • Subject-matter departments responsible for survey design

4. The typical survey at Stats Sweden • Used to have extensive freedom regarding systems, methods and tools • Suboptimization due to methodologists and culture • Now a very clear responsibility is given to the survey managers • Use standard methods and tools • Decide a proper resource allocation

5. The idea: A survey design handbook for survey managers • Simple language • Educational • Rules of the road

6. The iterative process Talking to the client Methods available General assessment of survey situation The planning criterion An ideal approach Trade-off situations The final compromise design Information needs Pilots and pretests QC based on paradata Evaluation Error and cost structures Responsive designs Documentation Creating a design team Contents

7. The typical survey textbook • Lack of a real planning theory • Emphasis on sampling design • Handling of specific sources of error • Reducing, weighting, estimating via survey models and latent class analysis • Design checklists and process design one by one • Handbooks by Eurostat, ABS, SNZ, UN FAO

8. Talking to the client • Research problem and analytical needs • Budget • Information available • Various briefings of design developments • What to do when the client situation is fuzzy

9. Characteristics of alternative methods Similar surveys done before? Is info available elsewhere? Information needs Constraints Error and cost structures Special features such as population distribution, sensitivity, lack of options, new research field, etc. Risk assessment To do or not to do General assessment of the survey situation

10. The planning criterion • Smallest possible MSE for a given budget • Smallest possible MSE for a given budget and further constraints such as timeliness, comparability, accessibility, coherence, response burden, etc.

11. Ideal seldom realistic One quality dimension vs another Cost vs errors Bias vs variance Quality vs cost Error in one process step vs another Mode switch vs new error structures New technology vs new error structures One method vs another The issue: How are multiple trade-offs handled? Trade-off situations

12. The final design the result of: • Continuing discussions between team and client • Adoption of certain rules of the road • Appreciation of the fact that optimum is often flat • Prioritizing between multiple purposes • No major steps or error sources ignored • Proper resource allocation

13. General thinking • Think upstream • Use expertise and prior experience • Analyse reduction of error as a function of cost • Those correlated variances • Develop a plan

14. Use known reliable methods Use information on errors and costs Allocate resources and assign responsibilities Collect information about quality as survey progresses Collect paradata for QC and responsive efforts Document and disseminate info on quality to users and producers Use experts and best practices Use guidelines and standards Rules of the road

15. References • Dalenius 1971 • Fellegi and Sunter 1974 • Groves 1989, 2006 • Linacre and Trewin 1993 • Holt and Jones 1998 • Weeks • Biemer and Lyberg 2003 • Campanelli 2006 • Heeringa and Groves 2007 • O’Muircheartaigh 2008 • Couper 1997, 1998