From Dual-Frame to Triple Frame:

1. Presented at AAPOR 2011 Phoenix, AZ May 15, 2011 From Dual-Frame to Triple Frame: An Assessment of Coverage Bias in a Telephone Survey Design Combining RDD, Directory-Listed And Cell Phone Samples

2. Thomas M.Guterbock University of Virginia TomG@virginia.edu Abdoulaye Diop Qatar University ADiop@qu.edu.qa James M. Ellis University of Virginia jme2ce@virginia.edu John Lee Holmes University of Virginia jlh2r@virginia.edu Trung Kien Le Qatar University kienle@qu.edu.qa Authors

3. Overview • Why triple-frame? • The 2010 Behavioral Study of the NCR • Coverage and distribution of phone service (5 segments) • Contrast of RDD, EWP and Cell Phone frames • Calling efficiencies • Comparing substantive results: • triple frame vs. RDD+Cell • Cost comparisons • Conclusion

4. Why triple-frame? From one frame, to two, to three

5. From RDD to Dual-frame • “Traditional” list-assisted landline Random Digit Dialing is beset with problems • Increasing under-coverage due to Cell Phone Only (CPO) households • Lower working-number rates • Declining response rates • Dual-frame telephone surveys are now standard for many survey organizations • Combining landline RDD and cell phone RDD frames • Most often using an ‘overlap’ or ‘all cell’ design • See AAPOR Task Force 2010 for full discussion • Adding cell phones covers the CPO’s and favorably alters reachability of dual-phone households

6. Consider: Electronic White Pages(directory listed) sample • Previous studies have compared EWP to landline RDD sample in statewide, regional, local studies. • Guterbock et al. 2003, Oldendick et al. 2004. • Guterbock, Diop & Holian 2007 explored race and other predictors of listedness in a survey of the National Capital Region • Caution: These studies pre-date the surge in CPO households

7. EWP vs. RDD • Substantive results are similar, in general • Advantages of EWP: • More efficient • Lower cost • Greater geographic specificity • Disadvantages: • Undercoverage of minorities, lower income, renters • African Americans less likely to have listed numbers • Blacks therefore under-represented in EWP samples

8. Who needs RDD? • There are notable similarities between characteristics of unlisted landline households and CPO households • Minorities, the young, renters, lower incomes • Could the gaps in the EWP frame (undercoverage) be filled in by inclusion of the cell phone frame? • We proposed in 2008: EWP+Cell as a dual-frame design alternative to RDD+Cell • Based on analysis of NHIS data through 2006 • Just published in Social Science Research • We presented a favorable comparison of the two designs in three county-based surveys in Virginia (2009)

9. From Dual Frame to Triple Frame • CSR has completed ten telephone surveys that use a triple-frame design • 4 metro-area based; 6 county-based • All cell phones included—no screening for CPO • All in Virginia or DC metro area • Two reasons for triple-frame design: • To allow comparison of RDD+Cell vs. EWP vs. Cell • Most of these studies needed comparison to earlier years that used landline RDD exclusively • We are at a transitional stage in telephone sampling • Triple-frame designs are a compromise between ‘standard’ dual-frame design and our proposed alternative of EWP+Cell

10. It’s all about. . . error cost

11. The 2009 Behavioral Survey of theNational Capital Region

12. 2009 Survey ofBehavioral Aspects ofSheltering and Evacuationin the National Capital Region Sponsor: VDEM Funding: U.S. DHS 12 12 12

13. Survey Goal Collect information from residents of the National Capital Region that would predict behavior in the area in the event of an emergency. Included factorial experiment that varied features of a “dirty bomb” attack by terrorists. Ask how residents would respond to specific “shelter-in-place” scenarios What variables have the most effect on behavior? What patterns of evacuation and shadow evacuation should be expected? Where would the evacuees try to go? The resulting data are being used to inform the decisions made by administrators in the region and beyond. Details in CSR’s report, 2010. 13 13 13

14. Features of the Survey In-depth survey: average interview length 28 minutes Fully supported Spanish language interviews as needed 2,609 interviews conducted by CSR, Sept-Dec 2009. Triple-frame sample design: 1269 Landline RDD completes 898 EWP (directory listed) completes 442 cell phone completes (no screening for CPO’s) RDD sample was backmatched to addresses Advance postcard sent to EWP and backmatched RDD cases Weighting by ownership, race, gender, geography, and type of telephone service Margin of error: +/- 2.3 percentage points After weighting 14 14 14

15. Coverage and distribution of phone service

16. What percentage of landline phones are unlisted? • Each respondent was asked whether their landline is listed in the directory. • Dual users reached by cell phone were asked to report on whether their landline is listed • Percent unlisted can be taken directly from the RDD frame. • 19.9% of landline RDD completes are unlisted. • As in our other studies, a small portion of those in the EWP frame report their phones to be unlisted.

17. What is the CPO percentage? • NHIS has been used as the ‘gold standard’ for weighting by phone service. • NHIS did not provide estimates for this geography. • We used the locally based method of estimation described by Guterbock 2009. • 31.4% of cell phone respondents were CPO’s. • Final estimate: 15.4% of telephone HH are CPO. • After excluding landlines with unknown listed status

18. 2009 NCR Telephone Universe 2 CELL + ULL 18.1% 3 ULL ONLY 1.3% 1 CELL ONLY 15.4% 4 CELL + LLL 61.0% 5 LLL ONLY 4.2%

19. What EWP+Cell would cover 2 CELL + ULL 18.1% 3 ULL ONLY 1.3% 1 CELL ONLY 15.4% 4 CELL + LLL 61.0% 5 LLL ONLY 4.2%

20. Little undercoverage • Households with no cell phone and an unlisted landline phone (ULL only), are but 1.3% of the region’s telephone households. • These are covered by the landline RDD frame, but not covered by the EWP frame. • EWP frame underestimates unlisted percentage. • We weight the unlisted percentage among all landlines to 19.9% (the unlisted percent in RDD).

21. Contrasting the three frames And the matched vs. unmatched portions of landline RDD frame

22. Comparing calling efficiency

23. Comparing key demographics

24. Comparing substantive results • How are survey results affected when a triple-frame sample is used? • We compare our triple-frame result with the results we would have obtained with an RDD+Cell design. • Both designs are post-weighted to the same control weights: • Ownership and race (joint distribution), gender, 8 counties, type of telephone service (CPO, LLO, dual user reached by landline, dual user reached by cell phone) and listed status

25. Comparing substantive results

26. Would leave scene in a dirty bomb attack Minimum hazard level Moderate Maximum Can trust most people Trust local government Has an emergency plan Has an emergency kit Has a meeting place Worry about attack 3 levels of hazard High perceived risk Property damage; injury 3 levels of hazard Agree/Strongly agree: Feel at home where I live I have a lot in common with neighbors Important for me to live in this area Let’s compare 20 variables

27. Differences for 20 variables Margin of error +/- 2.3%

28. Cost comparison

29. Cost factors and assumptions • Assume that a survey of like size had been carried out with a ‘traditional’ dual frame design • Same number of cell phone completions as in our triple-frame design • 442 cell phone completes • 2167 landline RDD completes • RDD uses more sampled numbers • RDD cost per sampled number is higher • Due to extra charge for backmatching • Assume postcard sent to backmatched RDD cases • Cell phone completes get $10 incentive

30. Triple frame costs

31. RDD+Cell costs (projected)

32. Triple frame vs.RDD+Cell costs

33. Cost summary RDD+Cell would have cost: $159,413 ($61.10 per completion) Triple frame design cost: $143,849 ($55.14 per completion) Substitution of EWP for some of the landline RDD frame saved $15,564 ($5.97 per completion) or 10.8% of the Triple Frame total. Greater savings could have been realized if EWP percentage were larger relative to landline RDD 33

34. Conclusion

35. It’s all about. . . error cost

36. Conclusion For representative general population results, we need to include cell phones in our telephone surveys When combined with the cell phone sample frame, EWP sample frames offer greater efficiency than landline RDD lower cost There are good reasons to retain some landline RDD sample in the mix at this transitional stage Allows direct measurement of unlisted percentage There is little or no loss of accuracy when EWP is substituted for some of the landline RDD frame and cell phones are included in the design Cost savings are considerable (over 10% in this study) 36

37. Conclusion Three frames are better than two! RDD EWP Cell

38. References AAPOR Cell Phone Task Force. 2010. New Considerations for Survey Researchers When Planning and Conducting RDD Telephone Surveys in the U.S. With Respondents Reached via Cell Phone Numbers. Available online at www.aapor.org. Guterbock TM, 2009. “Estimating Local Phone Service and Usage Percentages.”: How to Weight the Data from a Local, Dual Frame Sample Survey Of Cell Phone and Landline Telephone Users in the United States.” AAPOR paper. Guterbock TM, Diop A, Ellis JM, Le TK, & Holmes JLP, 2009. “Who Needs RDD–Part II: An Assessment of Coverage Bias in Dual-Frame Designs That Combine Directory-Listed And Cell Phone Samples.” AAPOR poster, Hollywood FL. Guterbock TM, Diop A, Ellis JM, Le TK, & Holmes JLP, 2011. "Who Needs RDD? Combining Directory Listings with Cell Phone Exchanges for an Alternative Telephone Sampling Frame". Social Science Research 40:3 (May): 860-872. Also presented as 2008 AAPOR paper, New Orleans. Guterbock TM, Diop A, & Holian L, 2007. “White pages, white people: Reasons for the low listed-phone rates of African-Americans.” AAPOR paper, Anaheim. Guterbock TM, Hartman DE & Hubbard RA, 2003. “RDD vs listed: An experimental study of coverage error, costs and non-response in a statewide telephone survey. AAPOR paper, Nashville. Guterbock TM, Lambert JH, Bebel RA, Ellis JM, & Kermer DA, 2010. Population Behaviors in Dirty Bomb Attack Scenarios: A Survey of the National Capital Region. Prepared for the Virginia Department of Emergency Management. University of Virginia Center for Survey Research, April. Oldendick, Robert W., et al. 2004. “Differences in an RDD and list sample: An experimental comparison.” AAPOR paper, Phoenix.

39. Presented at AAPOR 2011 Phoenix, AZ May 15, 2011 From Dual-Frame to Triple Frame: An Assessment of Coverage Bias in a Telephone Survey Design Combining RDD, Directory-Listed And Cell Phone Samples Contact: TomG@virginia.edu