1 / 25

Interviewers, nonresponse bias and measurement error

Interviewers, nonresponse bias and measurement error. Patrick Sturgis University of Southampton. Research Methods Festival, Oxford, 2-5 July 2012. Co-authors. Ian Brunton -Smith (University of Surrey) Joel Williams (TNS-BMRB ). Background and Motivation. Common Causes of Survey Error.

derora
Download Presentation

Interviewers, nonresponse bias and measurement error

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Interviewers, nonresponse bias and measurement error Patrick Sturgis University of Southampton Research Methods Festival, Oxford, 2-5 July 2012

  2. Co-authors • Ian Brunton-Smith (University of Surrey) • Joel Williams (TNS-BMRB)

  3. Background and Motivation

  4. Common Causes of Survey Error 4 Recent attention has focused on common causes of nonresponse and measurement error (cf. 2010 special issue of POQ on Total Survey Error) Agencies often target field resources at persuading reluctant respondents to meet response rate targets These respondents are less motivated and potentially less able to complete questionnaire accurately

  5. Error trade-offs • So potential reduction in nonresponse bias my be offset by increase in measurement error • Growing evidence that this does happen in practice (Kreuter et al 2010; Sakshaug et al 2010) • This work has focused on respondents so far • What about interviewers?

  6. Interviewers as common cause? • Interviewers can cause nonresponse bias and measurement error • Success in obtaining contact and cooperation related to interviewer characteristics: • Tailoring • Maintaining an interaction • Personality • Attitudes and beliefs • Some interviewers don’t get interviews, where ‘better’ interviewers would • If these ‘lost’ respondents are different on survey variables, result is biased population estimates

  7. Interviewer variance • Interviewers cause measurement error through the way they administer the questionnaire • At the individual level, these can be considered biases (response different to true value) • But across respondents and interviewers the result is larger variances

  8. Interviewer Variance • E.g. interviewer always reads same question incorrectly • Across interviews, this creates within-interviewer correlation – same as geographical clustering • Interviewer contribution to variance of estimator denoted ρInt

  9. How is ρInt related to nonresponse? • Anecdotal evidence that more successful interviewers on the doorstep are less diligent at sticking to questionnaire wording and instructions • Alternatively, some interviewers are good at what they do, others are not so good • Either way, we should anticipate a correlation between response rate and ρInt?

  10. Conceptual model 1 - Nonresponse bias + Response rate + Tailoring Agreeable-ness + - VarianceInt + Deviation from questionnaire wording and instructions + ρInterviewer Implications for total survey error MSEint = bias2 + varianceint

  11. Conceptual model 2 - Nonresponse bias + Response rate + Tailoring Conscientiou-sness + + VarianceInt + Deviation from questionnaire wording and instructions + ρInterviewer Implications for total survey error MSEint = bias2 + varianceint

  12. Analytical approach Fit cross-classified multilevel models to face to face interview data Partition ρinto area and interviewer components Examine variation in ρIntacross distribution of measures of interviewer success in obtaining contact and cooperation seperately

  13. Measuring interviewer success • Average response rate problematic as indicator of interviewer success on the doorstep • Our measure of interviewer success • Calculate ‘expected’ response propensity for all original issue cases based on geodemographic characteristics and paradata • Take mean of ratio of expected to observed rate across all cases for each interviewer • Do this separately for contact and cooperation • Group into ‘success quantiles’

  14. Modelling strategy • Cross-classified multilevel models with a complex interviewer error structure • Allows simultaneous estimation of separate ρInt for each interviewer success quintile Adjusts estimates for clustering of interviews within sample points Models also include individual, interviewer and area controls to account for non-random allocation of respondents to interviewers

  15. Data and analysis • British Crime Survey (2005/06) • 43,465 respondents , 472 interviewers, 3,782 areas • 36 items asked of all respondents which were non-factual and included probes and/or show-cards • Cross-classified multilevel model with complex error term at interviewer level • Controls • Individual - Gender, age, ethnicity, education • Interviewer - Gender, age, ethnicity, experience level (months worked) • Area - Socio-economic disadvantage, urbanisation, ethnic diversity, housing structure, age profile, population turnover

  16. Results I: Example Model

  17. Results II: Overall estimates • For contact measure bottom quintile has largest variance on 25/36 items (15 at p<0.05) • For cooperation measure bottom quintile has largest variance on 20/36 items (13 at p<0.05) • For contact, bottom group had 74% higher variance across all items compared to top group • For cooperation, bottom group had 34% higher variance across all items compared to top group

  18. Mean Interviewer variance components across 36 items by contact and cooperation success quintiles

  19. ITEMS WITH A DOWNWARD TREND ASSOCIATION BETWEEN CONTACT SUCCESS AND INTERVIEWER VARIANCE

  20. ITEMS WITH A U-SHAPED ASSOCIATION BETWEEN COOPERATION SUCCESS AND INTERVIEWER VARIANCE

  21. Analternative explanation? • Are these differences really due to interviewers deviating from the script? • Could also arise due to differential nonresponse bias • We find no differences across quintile groups on a range of background variables • If due to nonresponse bias, should observe uniform gradients over different question types that vary the degree of interviewer involvement

  22. Results III: by question type

  23. Discussion Historical measure of contact and cooperation success negatively correlated with interviewer variance Different pattern for contact and cooperation – contact broadly linear, cooperation evidence of u-shaped distribution Pattern of findings across question types suggest effect is due to interviewer behaviour in questionnaire administration Relatively small group of ‘poor’ interviewers make disproportionate contribution to total survey error Suggests response rate may be used as indicator of potential problems with interviewer behaviour/ training

  24. Some extensions

  25. Conceptual model - Nonresponse bias + + Response rate Tailoring + Agreeableness VarianceInt - + Deviation from questionnaire wording and instructions + ρInterviewer

More Related