1 / 28

Roger Tourangeau, SRC, JPSM Robert M. Groves, SRC, JPSM Stanley Presser, JPSM

Experiments Exploring the Relationship (or Lack Thereof) Between Nonresponse Error and Measurement Error. Roger Tourangeau, SRC, JPSM Robert M. Groves, SRC, JPSM Stanley Presser, JPSM Chris Toppe, Points of Light Foundation Courtney Kennedy, MPSM Ting Yan, NORC. Acknowledgement.

tanner-farley
Download Presentation

Roger Tourangeau, SRC, JPSM Robert M. Groves, SRC, JPSM Stanley Presser, JPSM

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Experiments Exploring the Relationship (or Lack Thereof) Between Nonresponse Error and Measurement Error Roger Tourangeau, SRC, JPSM Robert M. Groves, SRC, JPSM Stanley Presser, JPSM Chris Toppe, Points of Light Foundation Courtney Kennedy, MPSM Ting Yan, NORC

  2. Acknowledgement • The work reported here was supported by a grant from the National Science Foundation (SES 550385) to Roger Tourangeau and Robert Groves. We are grateful to NSF for its support and especially to Cheryl Eavey. • The National Science Foundation is not responsible for the conclusions presented here. • We thank Duston Pope for his contributions to the design, John LaFrance, and Mirta Galesic for their contributions to the studies discussed here study design. Nonresponse Error and Measurement Error

  3. Outline of Talk • Models of the relationship between nonresponse and measurement error • Experiment 1: A study of volunteers and volunteering • Experiment 2: Topic and sponsorship in a two phase web surveys • Two additional studies • Gain-loss framing and response propensities • An experiment with voters and nonvoters • Conclusions Nonresponse Error and Measurement Error

  4. Models of the Nonresponse-Measurement Error Relationship • Underlying model Nonresponse Error and Measurement Error

  5. Model 1: Independence Model • Nonresponse essentially motivational (interest in topic, altruism, low opportunity costs) • Error largely cognitive (poor comprehensive of the items, memory difficulties, poor estimation strategies) M C Up p e Ue

  6. Model 2: Reluctance • Difficult to persuade respondents answer sloppily (Bollinger and David, 2001; Fricker 2006)—Common cause model M p Up e Ue Nonresponse Error and Measurement Error

  7. Model 3: Topic Sensitivity • Drug users and reports about drug use X p e Up Ue Nonresponse Error and Measurement Error

  8. Model 4: Topic, Sponsor Effects • Presentation of the survey request affects both who responds and how they respond Topic Interest Topic p e Up Ue Nonresponse Error and Measurement Error

  9. “Framing” Effects • Why should the description of the topic or sponsor effect the answers Rs give? • Three mechanisms • Construal: Description of the survey may affect how they interpret the questions (National Crime Survey vs. National Survey of Violence Against Women—the latter may suggest a broader interpretation of “sexual assault” than the former) • Priming: Topic, sponsor may serve as a retrieval cue, making some memories more accessible • Perception of what is relevant or useful—Norenzayan and Schwarz (1999): Institute for Social Research and Institute for Personality Research • Evidence for all three (Galesic and Tourangeau, 2007) Nonresponse Error and Measurement Error

  10. Experiment 1: The March of Dimes Study • Survey topics • Survey of Giving and Volunteering • Survey of Jobs and Labor Force Activity • Target populations • March of Dimes mailing list • Donors, volunteers, other nominated by donors • General adult sample from vendor list • Sponsor • March of Dimes • University of Michigan Nonresponse Error and Measurement Error

  11. Recruitment Protocol • Mailed paper questionnaire • Advance letter • Initial questionnaire packet • Reminder postcard • Followup questionnaire packet All materials emphasized sponsor and topic Nonresponse Error and Measurement Error 10

  12. Response Rate by Topic and Population Concern about sugging? Nonresponse Error and Measurement Error 11

  13. Commitment to MoD Predicts Preference for the MoD Survey 12

  14. Mixed Results on Commitment and Poor Match to Records Nonresponse Error and Measurement Error 13

  15. Conclusions from Preliminary Analysis Volunteers and donors to March of Dimes tend to participate in all survey topics/sponsor combinations Those MoD donors with greater commitment more positive to the MoD sponorship than those with lesser commitment Mixed results on whether those with greater commitment tend to misreport their activities in support of MoD Nonresponse Error and Measurement Error 14

  16. Experiment 2: Two-Wave Web Survey • Collected data from members of two web panels in on-line survey • Wave 1 collected predictors of response propensities and answers to second wave • Wave 1 respondents (n=3000) invited to take part in second survey about a month later • Ostensible topic and sponsorship of second survey varied • Examined how response rates varied as a function of attitudes toward topic and sponsor • Also looked at effects on answers Nonresponse Error and Measurement Error

  17. Experiment 2: First Wave • n=47,078 asked to complete University of Michigan survey on “attitudes and lifestyles” • Members of two web panels: SSI (4.1% responded), e-rewards (14.8%); 3,000 respondents in all • Wave 1 items included: • Questions on how often the respondent had responded to web survey requests in the past; • Items assessing the respondents’ views about the importance of a number of social issues, including issues related to the topic of the second questionnaire (crime and violence, gun control); • Items assessing support for or opposition to gun control policies; • Relevant factual items (such as the presence of a gun in the household and experiences using guns); • Rating scales on the prestige of various organizations, including the purported sponsors of the second Nonresponse Error and Measurement Error

  18. Experiment 2: Second Wave • n=2,020 (67.3 response rate conditional on wave 1) • Varied three factors—sponsor, topic, and purported length • Sponsor: • National Coalition of Gun Owners (pro-Gun) • National Coalition for Victims of Gun Violence (anti-gun) • National Center for the Study of Crime • Topic • Crime prevention • Gun control policies • Length (20 minutes, 5 minutes, no mention) Nonresponse Error and Measurement Error

  19. Our Hypotheses • Attitudes toward the sponsor and topic to affect both the likelihood of cooperation and response behavior on questions relevant to the sponsors’ likely point of view. • Similarly, we expected topic interest to affect response propensities and answers to questions in the survey • Expected burden to lower response rates Nonresponse Error and Measurement Error

  20. Results—Response Rates The experimental variables seem to have little impact on either the response rates or nonresponse bias in estimates in the second survey.

  21. Results — Answers • Created three composites • Personal safety (4 items) — no effects • Gun control attitudes (11 items) • Main effect for topic: More support for gun control when the topic was described as crime prevention than when it was described as gun control—context effect? • Purported sponsor interacted with wave 1 att’s (next slide) • Involvement in issue of gun control (3 items)—topic effect among pro-gun types: those below the first survey median in their support for gun control measures reported greater involvement in the issue when the topic was described as crime prevention (mean of 2.47) than when it was described as gun control (mean of 2.36) Nonresponse Error and Measurement Error

  22. Results — Gun Control Attitudes

  23. Results — Liking for the Sponsor and Sloppiness Nonresponse Error and Measurement Error

  24. Two New Studies: Gain-Loss Framing • Gain-Loss Framing: Two-wave telephone study; request to do second interview at end of first interview We really appreciate the help you’ve given us today. We are interested in how people’s views about the issues we discussed today change over time so it is important we talk to you again in a couple of weeks. Then get either The information you’ve already provided to us will be a lot more valuable if you complete the second interview (77.9%) Or Unfortunately, the information you’ve already provided to us will be much less valuable unless you complete the second interview (87.5%) Nonresponse Error and Measurement Error

  25. Voting Study • Obtained sample of Maryland voters and non-voters • Vary mode of data collection (mail vs. telephone), incentive ($5 vs. none), and topic (Health and Lifestyles vs. Politics, Elections, and Voting) • Attempt to test “topic sensitivity” model (Model 3) • Reasons to think that dosage could be higher or lower in mail Nonresponse Error and Measurement Error

  26. Voting Study—Mail Questionnaire

  27. Voting Study—Mail Questionnaires

  28. Conclusions • Every study is framed somehow • Framing can effect response rates, though not necessarily nonresponse error • Topic effects on response rates found in Groves, Presser, and Dipko (2004), Experiments 1 and 2 here, “Birding study” in Groves et al. (2006) • But not always—three other studies in Groves et al. (2006) don’t find topic effects • Sponsor can effect responses, but again not always • Framing can effect answers, though effects may be subtle as in Experiment 2 here and in earlier experiment by Galesic and Tourangeau (2007) • Only weak evidence that reluctant R’s, or those who dislike sponsor or topic more prone to measurement error Nonresponse Error and Measurement Error

More Related