1 / 32

An Evaluation of Nonresponse Bias in the Resident Relocation and Making Connections Surveys

An Evaluation of Nonresponse Bias in the Resident Relocation and Making Connections Surveys. Lisa Lee & Catherine Haggerty NORC SLLS, September 2010. Nonresponse Bias. Nonresponse can bias survey estimates High response rates are not a guarantee against nonresponse bias.

liam
Download Presentation

An Evaluation of Nonresponse Bias in the Resident Relocation and Making Connections Surveys

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An Evaluation of Nonresponse Bias in the Resident Relocation and Making Connections Surveys Lisa Lee & Catherine Haggerty NORC SLLS, September 2010

  2. Nonresponse Bias • Nonresponse can bias survey estimates • High response rates are not a guarantee against nonresponse bias.

  3. Nonresponse in Panel Surveys • Analyses on changes over time are conducted on the panel respondents • Characteristics of the panel may not represent those of the nonrespondents

  4. Study of Nonresponse Bias • Examine issues of nonresponse bias in two longitudinal surveys conducted by NORC • Resident Relocation Survey • Making Connections Survey

  5. Resident Relocation Survey • Public housing leaseholders • Four rounds of interviews between 2002 and 2009 • Support from John D. and Catherine T. MacArthur Foundation

  6. RRS Response Rates

  7. Making Connections Survey • Ten urban communities • Cross-sectional study of neighborhoods combined with a panel survey of households with children • Three waves of data collection since 2002 • Funded by the Annie E. Casey Foundation

  8. MC Response Rates

  9. Addressing Nonresponse Bias • Weighting • RRS data weighted to account for slightly varying response rates by public housing development and by phase • MC Wave 1 data are weighted to account for varying response rates by neighborhood and by sample frame from which cases were drawn (Zimowski, Tourangeau, Ghadialy, & Pedlow, 1997; Ellis, 1970; Smith, 1983)

  10. Addressing Nonresponse Bias • Nonrespondent Follow-up • Conduct follow-up effort to locate and interview nonrespondents to the study • Best evidence for assessing reasons for nonresponse, presence and magnitude of nonresponse bias in a survey • However, if the surveyed nonrespondents are not representative, data will be of limited use in evaluating non-response bias Srinath, 1971; Zimowski, Tourangeau, Ghadialy, & Pedlow, 1997; Lasek, Barkely, Harper, & Rosenthal, 1997; Klein, 1981, Hill, Roberts, Ewings, & Gunnell, 1997

  11. Impact of Relocation on Leaseholders • Concern that CHA leaseholders being relocated would experience problems • Lease compliance • Unsubsidized housing

  12. RRS Wave 3 Nonrespondents

  13. RRS Nonrespondent Survey • Sample of 106 Wave 3 nonrespondents • 62 Wave 3 nonrespondents interviewed • 58.5% response rate

  14. Comparison of Respondents and Nonrespondents

  15. Another Approach to Assessing Nonresponse Bias • A nonresponse follow-up survey can affect budget and schedule • Is nonresponse survey necessary?

  16. Using Respondent Data to Examine Nonresponse Bias • “Continuum of resistance” approach • No contact was primary reason for nonresponse in RRS • “Continuum of locatability” Voigt, Koepsell, & Daling, 2003; Paganini-Hill, Hsu, Chao, & Ross, 1993; Filion, 1970. But see Lin & Schaeffer, 1995; Ellis et al., 1970.

  17. Time to Completion • Case management data on days to survey completion • Divide respondents into two groups • Fast = 25 days or less • Slow = more than 25 days

  18. Comparison of Wave 3 Fast and Slow Cases

  19. Findings from Fast vs. Slow Respondents • Nonrespondent survey provided evidence of presence and magnitude of bias • Differences in survey estimates found for Fast vs. Slow respondents (“locatability”)

  20. Using Data Across Survey Waves to Examine Nonresponse Bias • Examining Early Characteristics of Later Nonrespondents • Characteristics of respondents in earlier waves of a survey may be related to later nonresponse. • Are nonrespondents to later surveys different from those who continue to be respondents?

  21. RRS Wave 4 Dispositions of Wave 3 Respondents • 665 Wave 3 Respondents • 572 (86%) participated in Wave 4 • 93 (14%) did not participate in Wave 4

  22. Differences Between RRS Wave 4 Respondents and Nonrespondents

  23. Differences Between RRS Wave 4 Respondents and Nonrespondents

  24. Making Connections • Examining Wave 1 Characteristics of Wave 2 Respondents and Nonrespondents • Panel sample of 2029 children • 1432 (71%) participated in Wave 2 • 597 (29%) did not participate in Wave 2 • Cross-sectional sample of 1692 adult-only households • 1468 (87%) participated in Wave 2 • 224 (13%) did not participate in Wave 2

  25. MC Wave 1 Households with Children

  26. MC Wave 1 Households with Children

  27. MC Wave 1 Adult-Only Households

  28. MC Wave 1 Adult-Only Households

  29. Conclusions • Nonresponse bias can be present despite high response rates • Nonrespondent follow-up survey useful for understanding nonresponse bias • Differences between fast and slow to complete may provide clues to bias • Early characteristics of respondents may be associated with later nonresponse

  30. Next Steps • Expand research to • Look at additional waves of data • Compare respondents and nonrespondents on additional variables • Look at additional paradata

  31. Contact Information lee-lisa@norc.org haggerty@norc.uchicago.edu

More Related