What is meant by mode effect on measurement? A research study to identify causes of mode effects

1. What is meant by mode effect on measurement? A research study to identify causes of mode effects Gerry Nicolaas

2. Background Increasing use of mixed modes Falling response rates Rising costs of data collection Risk of reduced data comparability Coverage error Non-response error Measurement error Need for practical advice to inform decisions about when to mix modes and how

3. Mixed Modes and Measurement Error Funded under the ESRC Survey Design and Measurement Initiative 3-year contract starting 1 Oct 2007 Collaboration between NatCen, ISER and independent survey methods consultant

4. Research Team National Centre for Social Research (NatCen) Gerry Nicolaas Steven Hope David Hussey Institute for Social & Economic Research (ISER) Peter Lynn Annette Jäckle Alita Nandi Nayantara Dutt Independent Survey Methods Consultant Pam Campanelli

5. Main Objective Practical advice on how to improve portability of questions across modes Which mode combinations are likely to produce comparable responses? Which types of questions are more susceptible to mode effects?

6. Research Design A literature review & framework of mixed modes develop a conceptual framework identify gaps in evidence base and formulate hypotheses to address gaps Quantitative data analysis test hypotheses using existing datasets and new experimental data Cognitive interviewing explore how respondents process questions in different modes

7. Mode is defined by: Interviewer presence (face-to-face, phone, none) Oral and/or visual transmission of information Question delivery Delivery of response options Recording of response

8. Face-to-Face Interview Question Response Options Response Description Aural Aural Oral FTF (no card)‏ Aural Visual Oral FTF (card)‏ Visual (CAI)‏ Visual (CAI)‏ Written (CAI)‏ CASI Visual (paper)‏ Visual (paper)‏ Written (paper)‏ SAQ in-int’w Aural (CAI)‏ Aural (CAI)‏ Written (CAI)‏ ACASI Aural (CAI)‏ Visual (CAI)‏ Written (CAI)‏ ACASI

9. Telephone interview Question Response Options Response Description Aural Aural Oral Telephone Aural Aural Written (CAI)‏ TDE Aural Visual Oral Phone with showcards Aural (rec)‏ Aural (rec)‏ Oral (CAI)‏ IVR

10. Self-completion Question Response Options Response Description Visual (paper)‏ Visual (paper)‏ Written (paper)‏ SAQ (e.g. mail)‏ Visual (CAI)‏ Visual (CAI)‏ Written (CAI)‏ Web / email Aur & Vis (CAI)‏ Aur & Vis (CAI)‏ Written (CAI)‏ A-Web

11. Causality • How, when and why to mix/choose modes • Need to better understand causal mechanisms • Development and testing of behavioural theory

12. Interviewer presence Privacy/legitimacy I-R interaction Socialdesirability bias Social norms: Willingness to disclose? Comprehension Retrieval Judgement Response Context information: Influence on processing? Depth of cognitive processing: Sufficient Effort? Satisficing Sequential/ Simultaneous Qs Add. explanations I characteristics Time pressure R distraction R motivation Cognitive demands Control Qaire Interviewerpresence Aural/visual

13. Hypotheses • Short versus long response lists • Effects of interviewer presence on satisficing • Agree-Disagree scales • Is acquiescence caused by satisficing, cognitive ability, social desirability? • Ranking versus Rating • Effects of interviewer presence • Fully-labelled versus End-labelled scales • Effects of visual stimulus and interviewer presence • Showcards versus No Showcards (face-to-face interview) • Effects of visual stimulus • “Branched” versus “Non-branched” questions • Effects of item design (no mode differences expected) • “Yes/No” versus “Code all that apply” • Effects of item design (no mode differences expected)

14. Question selection Questions were designed to vary by: • Task difficulty • Sensitivity • Question type • Satisfaction • Other attitudinal • Behavioural • Other factual

15. Mixed Modes experiment Follow-up surveys to NatCen Omnibus (& BHPS) Face-to-face, telephone and web comparisons Experimental design with random allocation Limitation Restricted to respondents with web access

16. Acquiescence (1) • Hypothesis • Acquiescence is a form of satisficing • More satisficing in web than f2f & tel (no interviewer to motivate, explain, probe, etc) • 12 Agree/Disagree questions • 5-point agree/disagree scale • Use of opposite statements

17. Acquiescence (2) • Initial results from the experiment • More acquiescence in f2f & tel compared to web • Results from the cognitive interviews • Only 2 out of 23 cases of agreeing to opposite statements due to acquiescence • Justifiable explanations given for other 21 cases

18. Acquiescence (3) • Example of justifiable agreement with opposite statements • N36: Compared to other neighbourhoods, this neighbourhood has more properties that are in a poor statement of repair. • N38: Compared to other neighbourhoods, this neighbourhood has more properties that are well kept. • Respondent: In this village, … it’s like half and half…”

19. Acquiescence (4) • Interim conclusions • Use of opposite statements to detect acquiescence bias brought into question • Why higher rate of acquiescence in f2f & tel compared to web?

20. Other forms of satisficing (1) • Hypotheses: • More satisficing in web than f2f & tel (no interviewer to motivate, explain, probe, etc) • More satisficing in tel than f2f (lack of physical presence of interviewer, lack of non-verbal communication, distractions, etc) • Indicators of satisficing • Primacy effects in visual modes, recency effects in aural modes, middle category effects, item non-response

21. Other forms of satisficing (2) • Initial results from the experiment • Primacy & recency effects: inconsistent patterns • Item non-response: no mode differences • Middle category effects: web respondents more likely to select middle categories than f2f & telresps (also for agree/disagree scales – see acquiescence results) • Results from cognitive interviews • Results suggest more satisficing in web & tel than in f2f interview

22. Mode effect or question design effect? • Example: • “Code all that apply” frequently used in f2f interviews and self-completion surveys • Alternative format for tel interviews tends to be a series of Y/N questions • Hypothesis • No mode effect • i.e. No differences if series of Y/N questions used across all modes

23. Mode effect or question design effect? • Initial results from the experiment • F2F & tel resps more likely than web resps to say “Yes” in series of Y/N questions • Similar mode effect not found for f2f and web respondents in “code all that apply” format (nb this format not used in tel mode) • Results from cognitive interviews • Questions raised about validity of “Yes” answers in Y/N series that may differ by mode

24. Continue with analysis of experimental data Papers Causes of mode effects on survey measurement The role of the interviewer in producing mode effects The role of visual/aural stimuli in producing mode effects The role of question format in producing ‘mode’ effects Using cognitive interviews to explore mode effects Training course Questionnaire design for mixed mode surveys What next?

25. For more details on this project, contact: Gerry.Nicolaas@natcen.ac.uk