1 / 33

Cell Phones for Data Collection: Costs and Challenges

This article discusses the increasing reliance on cell phones for surveys, the operational challenges, and potential biases and measurement errors. It also compares data from cell phone surveys to landline surveys and examines differences in demographic subgroups.

jessiebennett
Download Presentation

Cell Phones for Data Collection: Costs and Challenges

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Cell Phones for Data Collection: Costs and Challenges Michael Link1, Michael Battaglia2, Martin Frankel3, Larry Osborn4, and Ali Mokdad5 1 Nielsen Media Research 2 Abt Associates Inc. 3 Baruch College, City University of New York 4 Knowledge Networks 5 Centers for Disease Control and Prevention

  2. The Plague of Cell Phones!!!

  3. Cell phones and telephone surveys • Reliance on cell phones increasing (July-December NHIS): • 57.1% of households have a working cell phone • 11.6% of households (11.8% of adults) are cell phone only • 25.4% for age 18-24, 29.1% for age 25-29, 54.0% of unrelated adult households w/o children, 26.4% for renters • Result: increased potential for noncoverage bias • Cell Phone Summit 2005, TSM II 2006, AAPOR 2007 – special issue of POQ

  4. Cell phones and telephone surveys • Conducting surveys via cell phones can be operationally challenging • Cell phone frame may not be that efficient • Geographic specificity is a problem • Cannot use autodialers/predictive dialers • Charges for incoming calls/minutes used • Safety concerns • Potential mode effects / measurement errors • Level of cognitive engagement

  5. Behavioral Risk Factor Surveillance System (BRFSS) • Monthly state-based landline RDD survey of health issues and related risk factors • 50 states, District of Columbia, Puerto Rico, Guam, and Virgin Islands • 350,000+ adult interviews conducted in 2006 • Significant declines in participation overall, particularly among younger adults and males

  6. 2007 BRFSS cell phone pilot • Conducted in Georgia, New Mexico, & Pennsylvania • Target: 200 cell & landline / 200 cell-only adults (per state) • 1,200 total interviews • Abbreviated BRFSS core interview: • 66 questions • 15-17 minutes (on average) • Incentives: • $10 post-paid incentive for completing the detailed interview • $1 for completing the screener

  7. Sample design • Marketing Systems Group (MSG): • All designated cellular 1,000 banks • Implicit stratification by area code and exchange • Equal probability sample of telephone numbers • Survey Sampling Inc. (SSI): • All 100 series banks designated as cellular • “Mixed use” (landline / cell phone) banks containing zero residential directory-listed numbers • Implicit stratification by FIPS, carrier, & 100-block • Systematic random sample of 100 blocks • Randomly generate last 2 digits of number

  8. Screening questions • Introduction • Confirmed telephone number • Is this a cellular telephone? • Are you 18 years of age or older? • Are you a resident of (state)? • “Do you also have a landline telephone that is used to make and receive calls?” • Yes – took subsample of respondents • No – took all respondents

  9. Survey Participation Rates

  10. Calculation of rates • Used detailed disposition codes modeled after Callegaro et al (2007) with some modifications/additions • Included “ring, no answer” and “voice mail” as working residential numbers • Only cases confirmed by company message as being not in service were excluded • Used AAPOR response rate guidelines • Calculated separate rates for: • Screening for eligible respondent • Completion of interview

  11. Participation rates

  12. Interview rate by landline access 64.8 66.7 64.9 64.1 70.2 64.3 66.1 57.7

  13. Percent working cell numberby sample vendor 74.5 70.5 66.1 63.3 65.3 61.2

  14. Comparison of respondent demographics

  15. Landline and Cell phonepopulations and frames LANDLINE B A C CELL PHONE

  16. Percent male 51.1 46.6 37.9 38.2 Landline survey Cell phone survey State equalized design weight applied

  17. Percent 18-34 years 51.4 24.0 14.5 19.6 Landline survey Cell phone survey State equalized design weight applied

  18. Percent Hispanic 21.4 15.2 16.8 12.2 Landline survey Cell phone survey State equalized design weight applied

  19. Percent black 15.0 15.8 9.3 7.5 Landline survey Cell phone survey State equalized design weight applied

  20. Percent high school or less education 60.3 48.5 39.8 33.6 Landline survey Cell phone survey State equalized design weight applied

  21. Percent married 69.8 62.0 49.5 32.0 Landline survey Cell phone survey State equalized design weight applied

  22. Summary of significant differencesacross demographic subgroups • Cell only v.s. cell & landline adults (from cell phone survey): • Significant differences for 12 of 24 subgroups examined • Particularly age, employment status & marital status • Cell & landline adults (cell phone survey v.s. landline survey): • Significant differences for 11 of 24 subgroups examined • Particularly sex, race, marital status, and children in household

  23. Comparison of key survey estimates

  24. Percent any kind of health care coverage 89.0 86.0 78.7 70.1 Landline survey Cell phone survey State equalized design weight applied

  25. Percent not received care due to cost barrier 20.4 24.9 16.3 10.2 Landline survey Cell phone survey State equalized design weight applied

  26. Percent currently smoke cigarettes 31.1 24.8 19.7 17.3 Landline survey Cell phone survey State equalized design weight applied

  27. Percent ever tested for HIV 54.2 43.6 37.5 36.6 Landline survey Cell phone survey State equalized design weight applied

  28. Percent binge drink past 30 days 23.5 21.1 13.0 11.0 Landline survey Cell phone survey State equalized design weight applied

  29. Comparison of Survey Costs

  30. Cost per Interview • Data collection costs only • Level of effort: • RDD = 7.4 calls/case • Cell = 3.2 calls / case • Response rate: • RDD = 38% • Cell = 26% • Interview length: • RDD = 25 minutes • Cell = 12 minutes $196 $74 $60

  31. What have we learned? • Group with both landline & cell phone differ across landline and cell phone surveys • Mode effect? Response/nonresponse effect? Frame effect? • This is an important issue when we try to combine landline & cell phone surveys • Cell phone only group differs significantly from landline group on some health variables, but not others • Risk behaviors seem most problematic

  32. What have we learned? • Cell phone & landline usage varies significantly across states • Makes use of national estimates from the NHIS for post survey adjustment problematic • Compared to landline surveys, cell phone surveys: • Have lower rates of response at the screener stage • But similar rates at the interview stage • Working residential rates lower, but not as bad as expected • Are considerably more expensive, especially if we decide to screen for cell-only adults

  33. Future directions • Problem is not going away, but will continue to worsen • Focus on combining estimates from cell and landline frames (Frankel and Battaglia are working on this for the BRFSS) • Cell only households not the only problem • Need more focus on primary cell users • Best to reach by landline or cell frame? • Mode effects – what measurement issues are raised by interviewing via cell phones?

More Related