The Use of Random Digit Dialing in Household Surveys: Challenges and Changes

1. The Use of Random Digit Dialing in Household Surveys: Challenges and Changes Chris Chapman 2008 IES Research Conference Washington, DC June 11, 2008 chris.chapman@ed.gov

2. Overview Two primary challenges in the use of traditional RDD collections for household surveys • Declining response rates in traditional Random Digit Dialing (RDD) sample methodology • Declining coverage rates in traditional RDD sample design studies Analyses of alternative approaches for government RDD surveys • Use of pre-phone contact mailed information materials • Use of prepaid monetary incentives • Study of staged, multimode collection methods • Expanding coverage

3. National Household Education Surveys (NHES) The analyses discussed today will be in context of work primarily done with NHES NHES is a repeating household survey conducted by NCES NHES has relied on RDD sampling approaches since its inception in 1991 through the latest collection in 2007 Sample is drawn from existing 100 banks starting with 130,000 to 475,000 numbers Sample is restricted to non-business numbers and excludes cells phones

4. Topics Addressed Early childhood education and nonparental care for infants, toddlers, preschoolers and students in early elementary grades Parent and family involvement in the education of K-12 students After school programs and activities for elementary and middle school children Adult education for all civilian, non-institutionalized adults Civic education and participation for K-12 students, parents and adults School safety and discipline for K-12 students

5. National Household Education Surveys (NHES) Typically relies on two stages of interviews 1. Screener interview to determine household and within household eligibility – generally need 45,000- 60,000 completed screener interviews per collection 2. Extended interviews collects detailed information about person(s) sampled within the household Significant nonresponse problem in the Screener 1. Approximately 80% in the early 1990s 2. Approximately 50% now

6. Increasing Response Rates with RDD Use of premailing Conducted experiments showing that sending letters to households before calling helped improve first stage response rates Use of monetary incentives Use of small monetary incentives mailed to households before calling helped improve first stage response rates In-person follow-up During recent bias study, found that in-person contact with phone interview refusal cases significantly boosted response rates

7. Premailing with RDD Letters about the purpose of the study and where to get information about it help boost first stage response rates In the 2001 NHES, the response rate for matched households sent a letter was 75% response rate for matched households not sent a letter was 70% response rate for households that could not be matched was 55% Challenge is matching randomly generated phone sample to mailing addresses – approximately 50 percent See http://nces.ed.gov/pubs2005/2005071_2.pdf

8. Prepaid Incentives with RDD In 2003, we tested monetary incentive options Required that we had a mailing address – again had a 50% match rate Tested no incentives against combinations of $2 and $5 incentives sent before the interview and as a refusal conversion technique http://nces.ed.gov/pubs2006/2006066.pdf

9. adfafasfas adfafasfas Prepaid Incentives with RDD

10. Prepaid Incentives with RDD Optimal approach for cost v. response rate effect was no initial incentive with a $5 incentive for refusal conversion Next best approach was $2 initial incentive with a $2 conversion incentive Both boosted first stage response rates over no incentives by approximately 5 percentage points Tested use of priority and regular mail for initial and refusal conversion mailing - mailing did not have an independent effect beyond incentives for the first stage of data collection

11. Study of Potential Bias and Multimode Approach Conducted an extensive bias test in 2007 Key components included: 1. In person follow-up for telephone nonrespondents – used to study potential nonresponse bias 2. In person contact for households not in the RDD frame – important to understand potential coverage bias

12. Multimode Work Results not released yet but some results are generally clear • If budget permits, in-person follow up can significantly increase response rates if used to help with telephone refusal conversion • If not corrected, surveys based on RDD samples of land line phones alone can incure coverage bias in studies of prekindergarten through high school students and their families

13. Frame Adjustments • NHIS indicates that in 2007 16% of households were cell only • Approximately 14% of children live cell phone only households • Approximately 15% of adults live in cell phone only households • Can correct for some of coverage bias through weighting • Preferable to incorporate cell phones into frames though • This may reduce overall response rates • Challenging for calculating proper weights • http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless200805.pdf