1 / 17

Ian Brunton-Smith, University of Surrey Patrick Sturgis, University of Southampton

Detecting and understanding interviewer effects on survey data using a cross-classified mixed-effects location scale model. Ian Brunton-Smith, University of Surrey Patrick Sturgis, University of Southampton George Leckie, University of Bristol 7 th ESRC Methods Festival, 5 th July, Bath, UK.

healyb
Download Presentation

Ian Brunton-Smith, University of Surrey Patrick Sturgis, University of Southampton

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Detecting and understanding interviewer effects on survey data using a cross-classified mixed-effects location scale model Ian Brunton-Smith, University of Surrey Patrick Sturgis, University of Southampton George Leckie, University of Bristol 7th ESRC Methods Festival, 5th July, Bath, UK

  2. Motivation • Interviewers can substantially inflate variance of estimators • idiosyncrasies in administration of questions • interaction of personal characteristics with those of respondents • Induce within-interviewer dependency • Akin to clustering • Standard approach: mixed effects model with random intercept

  3. Another source of interviewer error?

  4. Mixed effects approach to interviewer effects • Two-level (e.g. individuals nested in interviewers) random intercept model: where , • Enables estimation of interviewer ICC

  5. Location-scale model • Extend the standard model by modelling the level-1 variance as a log-linear function of covariates and a further random effect (Hedeker et al., 2008) Mean function: Level-1 variance function: where ,

  6. Location-scale model • Extend the standard model by modelling the level-1 variance as a log-linear function of covariates and a further random effect (Hedeker et al., 2008) Mean function: Level-1 variance function: where , , • Extended to the cross-classified case to adjust for area confounding (e.g. Vassallo et al., 2016; Durrant et al., 2010)

  7. Data and measures • Wave 3 of UKHLS general population sample (2011-12) n = 17,471 • Respondent age and gender • matched to ‘understanding society interviewer survey’ n = 303 • Interviewer gender, age, experience, beliefs about the value of surveys, and personality (big 5) • Adjusted for area clustering (MSOA) n = 3,473 • Ethnic diversity, socio-economic disadvantage, urbanicity, population mobility, age/housing structure (census)

  8. Results “People in this neighbourhood generally don’t get along with each other” (5-point likert scale)

  9. Identifying differences between interviewers • Design effect between 2.5 and 4.9 across middle 95%

  10. Quantifying the effect • Interviewer scoring 1SD below mean on extraversion, had worked on other surveys, but does not believe surveys conducted responsibly: DEFF = 3.2 • As above but 1SD above mean on extraversion: DEFF = 2.4

  11. Sense check: A self completion item “The friendships and associations I have with other people in my neighbourhood mean a lot to me”

  12. Setting up the model – a template for Stat-JR http://www.bristol.ac.uk/cmm/software/statjr/

  13. Discussion • Interviewer can have effect on variability of responses as well as mean • Location-scale model provides a flexible approach to identify these effects (implemented in Stat-JR) • Can be linked to specific interviewer characteristics to better understand effects Full paper available: Brunton-Smith, I., Sturgis, P., and Leckie, G. (online) ‘Detecting and understanding interviewer effects on survey data by using a cross-classified mixed effects location-scale model.’Journal of the Royal Statistical Society Series A. • http://onlinelibrary.wiley.com/doi/10.1111/rssa.12205/epdf

More Related