240 likes | 387 Views
Sample Design and Issues with Telephone and Multi-Mode Surveys. Meena Khare National Center for Health Statistics, COCHIS NCHS/DUC , Washington, D.C. July 12, 2006
E N D
Sample Design and Issues with Telephone and Multi-Mode Surveys Meena Khare National Center for Health Statistics, COCHIS NCHS/DUC , Washington, D.C. July 12, 2006 “The findings and conclusions in this paper are those of the author(s) and do not necessarily represent the views of the National Center for Health Statistics, Centers for Disease Control and Prevention.”
Outline • Telephone surveys • Sample design • Trends in response rates • Weighting and estimation • Issues with telephone and multi-mode surveys • Summary 3
Telephone Surveys • Lower cost and rapid estimates • Sampling Frame: listed or unlisted telephone numbers with landline phone connections • Wireless numbers are excluded for legal reasons • Stratified by area codes and exchanges within selected geographic areas • Telephone numbers selected from the banks of 100 consecutive numbers within an exchange • Multiple call attempts made to resolve numbers as residential, business, nonworking, ring-no-answer, etc. 4
Sample Design • Single stage cluster design • Sampling frame: Replicates of 100 banks [(xxx)-xxx-xx00 – (xxx)-xxx-xx99] with at least one residential number (1+Banks) • List assisted Random-Digit-Dialing (matching with ‘white pages’) • Independent random samples within geographic areas • Simultaneous screening and interviewing of households with landline telephone • Advance letters (and may be prepaid incentives) to telephone numbers with valid residential address (by reverse matching) • Multi-mode surveys: RDD surveys followed by a mail, Web or FTF survey of respondents and/or nonrespondents 5
Distribution of Adults With and Without Landline Telephones Inside House and Wireless Telephone Service by Age, 2004 NHIS 6
Nontelephone Households: Wireless-Only and Phoneless Households • Wireless only households (6-10%) • Men • Younger adults • Renting home • Uunrelated roommates • Income at or above poverty • Non-minority • Phoneless households (~2-5%) • Men • Older adults • Less educated • Lower income • Unemployed • Uninsured • Minority • References: Blumberg et al. (AJPH, 2006) 7
Survey Operations (an example) Replicate of landline telephones Numbers with business, nonworking and other non-residential numbers removed 8
Guidelines for Computing Response Rates Standards for computing response rates from AAPOR "Standard Definitions: Final Disposition of Case Codes and Outcome Rates for Surveys" , 4th edition, 2006. Definition of CASRO* Response Rates “On the Definition of Response Rates”, A Special Report of the CASRO Task Force on Completion Rates, 1982. Ezzati-Rice, T.M., Frankel, M.R., Hoaglin, D.C., Loft, J.D., Coronado, V.G., and Wright, R.A. “An alternative measure of response rate in random-digit-dialing surveys that screen for eligible subpopulations” Journal of Economic and Social Measurement, 26, 99-109, 2000. *CASRO: Council of American Survey Research Organizations 9
RDD Response Rate CASRO* Response Rate = Resolution Rate x Screener completion Rate x Household Interview Completion Rate where * CASRO: Council of American Survey Research Organizations 10
Trends in CASRO Response Rates from Selected RDD Surveys * A 1998 study of 35 RDD surveys reported: CASRO response rates ranged 22.2% - 70.1%, O’Rourke and Johnson, 1999 11
NIS 2004 Sample • 3,607,627 phone numbers selected • 3,023,174 (83.8%) phone numbers resolved as residential • 32,638 households screened in with eligible children aged 19-35 months • 30,019 (92.0%) of eligible households with completed interviews • 73.1% CASRO response rate • 30,987 age-eligible children with completed interviews (87.1% with consent to contact provider) • 21,998 (71.0%) children with completed interviews and ‘adequate’ provider data (including unvaccinated children) 12
Trends in National Immunization Survey Response Rates , 1995-2004 NIS 13
Trends in Refusal and Partial Interview Cases, 1995-2004 NIS 14
Trends in Rates of Refusal Conversion Among Children with Completed Interviews, 1995-2002 NIS 15
Trends in Percent of Households Mailed an Advance Letter, NIS 1996-2004 Range by 78 sampling areas, 2002 NIS : 31.4% - 72.0% 16
Sample Weights • RDD level interview weights are adjusted for • Probability of selection of telephone numbers • Unresolved numbers (using exchange information) • Nonresponse to screener and household interview • Multiple landline phone lines per household • Noncoverage of households with no landline telephone • Keeter’s Method (using information on interruptions in telephone service) • Noncoverage of population subgroups (post-stratification and raking) 17
Estimation • Sampling weights should be used for computing estimates • For variance and interval estimates: geographic areas are used as Strata and sampled persons or households within an area are used as PSUs. • Available software for analyzing data from complex surveys • SUDAAN • SAS • STATA • SPSS (Complex Sample Module) • WESVAR 18
Issues Related to Telephone Surveys • Noncoverage of nontelephone households • Changing telephone environment (e.g.,VOIP) • Wireless phone penetration • Undercoverage of households with landline telephone • Call-screening and call-blocking devices • Number portability: • mixed use of landline and wireless numbers • mixed-use telephone exchanges within a geographic area • Unknown residential status • Decline in Resolution rate 19
Issues Related to RDD Surveys (cont…) • Multiple landline telephones in a household; mixed usage • Government initiatives (legal issues) • FCC: Do-not-call registry • Advance letters: missing for >40%; no address for unlisted numbers • Increase in refusals to participate and partial interviews • ERB mandate of only one additional call to refusals Decline in overall survey response rate 20
Issues with Multi-mode Surveys • Incomplete sampling frames • Noncoverage and under-coverage of population sub-domains • Legal and financial issues • Increase in noncontact • Nonresponse adjustments and combining estimates • Availability of population controls • Potential for bias in estimates 21
Summary • Telephone surveys are the most effective ways to obtain rapid estimates • Most RDD surveys are experiencing decline in response rates and decline is primarily due to decline in resolution rates • Increase in refusal rates and partial interview cases • Refusal conversion, advance letters, and incentives have helped in improving response rates 22
Summary (cont…) • Use sampling weights for computing population estimates • To compensate for clustering, noncoverage and nonresponse • Refer to data user’s guide and other documents before analyzing survey data • Use software for analyzing complex sample data for estimating variance 23
Thank You Contact:Meena KhareEmail:mkhare@cdc.gov 24