Mode Bias and Adjustment in Social Surveys: General Overview

1. ROME April 11th | 12th 2019 MIMOD Mixed-Mode Designs for Social Surveys FINAL WORKSHOP Mode bias/mode effect and its adjustment: General Overview WP2 Orietta Luzi Istat

2. Mode bias/mode effect and its adjustment: General Overview Session organization 1) General overview (Orietta Luzi) 2) Keyissues and maindecisions in mode effectdetecion and adjustment (Orietta Luzi) 3) Re-interviews – a case study (Barry Schouten) 4) Experimenting methods to assess and adjust mode effect when a single mode control survey is available as a benchmark: a case study (Claudia De Vitiis) 5) Discussant (PawelSZymankiewicz) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

3. Mode bias/mode effect and its adjustment: General Overview • InvolvedCountries • ISTAT Italy • Orietta Luzi (coordinator) • Claudia De Vitiis, Francesca Inglese, Roberta Varriale, Alessio Guandalini, Marco Terribili • CBS Netherlands • Barry Schouten, Bart Buelens, Jan van denBrakel MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

4. Mode bias/mode effect and its adjustment: General Overview Mainobjectives and performedactivities • 1) Updated review of methodologies to: • Assess mode effects and evaluate the comparability of data collected by different modes (assessing the measurement equivalence) • Deal with mode effectsto ensure accurate results by properly estimating mode bias/mode effect • 2) Evaluation of the suitability of selected methodologies to deal with selection errors and to adjust for measurement errors in current MM surveys • Re-interview designs • Selected methods at the estimation phase • 3) Production of evidence-based guidelines for the use of methods to deal with mode effects in MM surveys, with a discussion of assumptions, advantages and disadvantages of the most common approaches MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

5. Mode bias/mode effect and its adjustment: General Overview Keyoutputs/deliverables MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

6. Mode bias/mode effect and its adjustment: General Overview Overview on current methodologies to deal with mode effect in MM surveys • MIMOD survey • Specific section of the questionnaire devoted to collect information on methods/strategies currently adopted in the ESS Countries for mode effect assessment and adjustment • Recent literature review • Updated overview on methodologies for mode effects assessment and adjustment in MM designs MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

7. Key issues and main decisions in mode effect detection and adjustment MIMOD survey: main results on methodologies to deal with mode bias/mode effects (1) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

8. Key issues and main decisions in mode effect detection and adjustment MIMOD survey: main results on methodologies to deal with mode bias/mode effects (2) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

9. Mode bias/mode effect and its adjustment: General Overview • Overview on current methodologies to deal with mode effect in MM surveys: Main evidences from the MIMOD survey • 31 responding Countries • Assessment of mode effects: • Most of the activities aim to assess the total mode effect • About32% didnotconductanyassessmentactivityin MM social surveys • Adjustment for mode effects: • About60% didnotconductanyadjustmentactivityin MM social surveys • Future plans: • About 50% of Countries reported to have future plans for research in into mode effect • Most of theeseplansfocus on assessmentand to a lesserextent on adjustment MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

10. Mode bias/mode effect and its adjustment: General Overview • Overview on current methodologies to deal with mode effect in MM surveys: Main evidences from the Literature review • Mode assessment studies are often limited to quantifyingthe total mode effect • Accordingly with the MIMOD survey, literature review highlights that mode effect assessment methods are more widespread than mode effect adjustment techniques: • Adjustment methods require mode effects to be separable into selection and measurementeffects and this isnot always done in the analyzed literature • An important reason is that it is difficult to separate selection from measurement effects, but easy to assess their combined effect. The two effects are said to be confounded MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

11. ROME April 11th | 12th 2019 MIMOD Mixed-Mode Designs for Social Surveys FINAL WORKSHOP Key issues and main decisions in mode effect detection and adjustment WP2 Orietta Luzi Istat

12. Key issues and main decisions in mode effect detection and adjustment Methodological strategies to deal with mode effect: key issues • MM is used to contrastdeclining response rates and coverage and to reduce the total survey costs • Main methodological drawbacks of MM: • Difficulty of controlling over mode effects and their biasing effects on survey estimates • The confounding of selection and measurement effects • Mode effect refers strictly to measurement differences due to the mode of survey administration • The selection effect, due to the differences in the distributions of respondents to the different modes • Especially in repeated surveys, estimates must be consistent and comparable, for ensuring that changes in the time series are exclusively due to real changes of the observed phenomena Appropriate methodological strategies are necessary to properly assess and adjust mode effects to ensure accurate and consistent estimates MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

13. Key issues and main decisions in mode effect detection and adjustment • General “guidelines” • Methodological strategies to deal with mode bias/mode effect: key decisions • Methodological strategies to deal with mode bias/mode effect : key elements • Methods to deal with mode effects • Assessing and adjusting mode effects: some key aspects • Check-list for the design of a strategy to deal with mode bias/mode effects MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

14. Key issues and main decisions in mode effect detection and adjustment • Methodological strategies to deal with mode bias/mode effect: Key decisions • Deciding if and how to estimate mode effects and/or to adjust for their biasing effects • quality criterion (e.g. the MSE) against a cost limit: how to assess whether mode effect adjustment is beneficial? • multi-dimensionality of a survey: what key estimates and population parameters of interest need to be evaluated? • time perspective: is the survey repeated, can effects be assumed constant, which actions are planned for future survey repetitions? MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

15. Key issues and main decisions in mode effect detection and adjustment • Methodological strategies to deal with mode bias/mode effect: key elements • Main requirements to be defined: • A “design” to control for mode bias/mode effects • Appropriate auxiliary variables (from administrative data/frame data/paradata) referred to as covariates that are mode-insensitive and informative about • Mode selection • Mode measurement • A set of assumptions (depend on the type of auxiliary data and type of design) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

16. Key issues and main decisions in mode effect detection and adjustment The design: strategies to control for mode bias/mode effect Experimental designsallow controlling for selection effects, and hence the unbiased assessment of measurement differences between modes. Experimental designs are rather “rare” because of costs. Observational studiesrequire covariates that explain the selection mechanisms. If available, differences between mode groups are attributed to measurement differences, conditional on the (error free) covariates. MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

17. Key issues and main decisions in mode effect detection and adjustment The design: strategies to control for mode bias/mode effect - Ageneral scheme MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

18. Key issues and main decisions in mode effect detection and adjustment • Methods to deal with mode effects • Reference schemesthat outline the methods which can be applied for different: • Survey/experimental contexts • Objectivesof the study • Analysis of total mode effect (1) • Analysis to disentangle measurement and selection effects (2) • Adjustment for selection and measurement effects (3) • Where appropriate, additional elements about requirements, assumptions, advantages/drawbacks MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

19. Key issues and main decisions in mode effect detection and adjustment • Methods to deal with mode effects • Assessing total mode effect (1) • Schemes - examples MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

20. Key issues and main decisions in mode effect detection and adjustment • Methods to deal with mode effects • Disentangle measurement and selection effects (2) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

21. Key issues and main decisions in mode effect detection and adjustment • Methods to deal with mode effects • Adjust for selection and measurement effects (3) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

22. Key issues and main decisions in mode effect detection and adjustment • Assessing mode effects: key aspects • Assessments are most sensibly conducted with respect to a benchmark • In assessment studies, the representativity of respondents, the response rates etc. can provide insight into the selection mechanism. Selection effect is a desirable effect of MM strategies as it could reduce selection bias on survey estimates • Assessmentof total mode effectitisrelativelyeasy and sometimes can be sufficient, butwhendetected, undesired mode effectsneed to be properlyestimated and adjustedfor (measurementeffects) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

23. Key issues and main decisions in mode effect detection and adjustment • Adjusting for mode effects: key aspects • Adjustment methods are necessary to correct estimates for undesidered mode effects • Appropriate adjustment methods require the effective separation of selection and measurement effects. • Disentanglingand estimatingmode effectcomponents can be a difficulttask • It requires that covariates are available which explain the selectionmechanism and which are assumed mode-insensitive and informative on mode selection/mode measurement • Both assessment and adjustment strategies are most reliable and less dependent on assumptions when conducted in experimental settings • When applying adjustments, a reference mode has to be chosen as benchmark MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

24. Key issues and main decisions in mode effect detection and adjustment • Check-list for the design of a strategy to deal with mode bias/mode effect • Identify the main quality and cost criteria • Decide whether mode effect estimation serves explanation only, design choice or adjustment • Identify available auxiliary data that is informative about mode selection/mode measurement • Evaluate anticipated validity of assumptions for mode selection, mode measurement and absence of experimental influences • Decide whether an experimental design (such as re-interview or parallel run) is required and feasible to serve the purposes of the mode effect estimation • Conduct experimental designs if deemed feasible and necessary MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

25. Key issues and main decisions in mode effect detection and adjustment THANK YOU FOR YOUR ATTENTION MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019