A Sampler of Methodological Issues: Spanish Language, Cell Phones, and Address Based Sampling Sarah Cho PAPOR Mini-Confe

1. A Sampler of Methodological Issues: Spanish Language, Cell Phones, and Address Based Sampling Sarah Cho PAPOR Mini-Conference June 24, 2011

2. THE BENEFITS & CHALLENGES OF ADDRESS-BASED SAMPLING DESIGNS David Dutwin, PhD Vice President Social Science Research Solutions Michael W. Link, PhD Chief Methodologist The Nielsen Company 66th Annual Conference of the American Association for Public Opinion Research Phoenix, AZ May 2011

3. Address-Based Sampling • “Address-Based Sampling” is the sampling of addresses from a database with near universal coverage of residential homes • ABS helps solve coverage issues, but not response rate • While mail surveys have been around for decades, use of a residential mailing frame for sampling the general population is relatively new • Before 2000: U.S. Census begins developing a Master Address File for the 2000 Census from U.S. Post Office • 2003: First published evaluation of use of mailing addresses for in-person survey • 2008: First published use of the term “Address-Based Sampling” • ABS is now the basis or key component of many critical studies (National Election Survey, Knowledge Networks Panels, Nielsen Television Audience Measurements, some State/City-Level Health Interview Studies) • ABS is a sampling methodology, NOT a single survey design and can be used as a foundation for multimode data collection designs

4. ABS vs. RDD • 63% of the public has at least one listed LL telephone number • 10% are zero listed landline banks and typically excluded from most RDD samples, 25% are CPO, and 2% have no telephone • LL only RDD sample systematically excludes 37% of the public • With ABS, coverage is 98% if you include all addresses (including PO Boxes) • Note: there are regional variations in coverage

5. Sample Frame • Primary source: USPS Address Management System • Computerized Delivery Sequence File (CDSF) • Only licensed vendors can provide addresses from this product • Weekly or monthly updates from USPS available • Delivery Sequence File Second Generation (DSF2) • Not a product, but a process • Can only be applied to a mailer’s existing address list to clean the list of erroneous addresses and cannot be used to generate an address list

6. CDSF Frame Elements (Raw File) • House number • Apt number • Street name • Street Suffix (ex. Ave, Blvd) • Directional (ex. NE, W) • Zip • Zip+4 • City name • City code • State code • State Name • Tract • Block • County • Walk sequence number (Postal route) • Route type • Delivery type code (ex. door slot, curb) • Vacant code • Seasonal code • Drop count (# people) • PO Box

7. Appending Data to CDSF • A key benefit of having address as the base sample unit is the extensive amount of information that can be appended to the sample file • Geographic data • Demographic data • Names and telephone numbers • Percent name append on average is over 90% • Percent phone append on average is about 65% • All varies by geography and the company you use to append the CDSF file

8. After Appending Data to the Sample • Two key decisions after sampling • Mode of contact for recruitment (mail, in-person, telephone) • Mode of interviewing (in-person, mail, telephone, web, dairies) • A combination of multiple modes of contact or interviewing can be utilized

9. Summary • ABS can achieve near-universal coverage • Need to work with a licensed vendor • New terminology associated with ABS • Information can be appended to the sample file to help facilitative sampling stratification, specialized data collection, backend analyses • Can support an array of different survey designs • ABS is still in its infancy, but growing number of studies adopting and testing this approach will allow industry to rapidly optimize use of ABS

10. For more information… • Check out the first AAPOR webinar on June 30th, 10am-11:30am • Instructor: David Dutwin • Register on the AAPOR website to view the webinar live or to view a recorded version at your own pace • Discounts available for students

11. Factors Contributing to Differences in Reported Party Identification Across Polls in 2010

12. The Issue Prior to 2007, Kaiser Health Tracking polls generally reported a party ID distribution quite similar to that of other publicly released polls. In 2009, we began to notice that the party ID distribution in Kaiser tracking surveys was leaning more Democratic compared with most other publicly released polls.

13. More Details on The Issue: 2010

14. Hypothesis 1 HYPOTHESIS 1: Offering respondents the opportunity to be interviewed in Spanish leads to a higher Dem-Rep gap compared to surveys conducted in English only.

15. Hypothesis 1 (Spanish interviewing): Internal analysis of KFF data Party ID among Hispanics in 6 Kaiser tracking polls, June-Dec. 2010 Independent/other/DK Democrat Republican Hispanics interviewed in English Dem-Rep = 25 Hispanics interviewed in Spanish Dem-Rep = 41

16. Hypothesis 1 (Spanish interviewing): Internal analysis of KFF data Average party ID in Kaiser tracking polls July-Dec. 2010: Original weighting versus English-only weighting

17. Hypothesis 1 (Spanish interviewing): Does # of Spanish interviews matter?

18. Hypothesis 2 HYPOTHESIS 2: Using an “overlapping” dual frame design leads to a higher Dem-Rep gap compared with surveys that use a “segmented” dual frame design or a landline-only design.

19. Hypothesis 2 (Cell phone frame): Internal analysis of KFF data Average party ID in Kaiser tracking polls July-Dec. 2010: • Original weighting vs. “segmented” dual frame weighting

20. Hypothesis 2 (Cell phone frame): Comparison to other orgs. inconclusive 2010 Party ID averages across survey organizations, grouped by type of telephone frame

21. Other Hypotheses Tested HYPOTHESIS 3: Differences in question wording and rotation of the terms “Democrat” and “Republican” can explain the different results between organizations. MAYBE HYPOTHESIS 4: The topic of the questions asked in Kaiser surveys (health care and health policy) leads to a higher Dem-Rep gap compared to other polls. NO HYPOTHESIS 5: Something about the way Kaiser survey data is weighted to national parameters is contributing to the difference. NO HYPOTHESIS 6: “Field house effects” are contributing to the difference. UNKNOWN

22. Implications Different factors can work together in the same direction and lead to measurable differences in survey outcome measures Cell phone frame: Evidence suggests type of cell phone frame can have an impact. Important for researchers to continue investigating as survey industry tries to settle on “best practices” for including cell phones in GP samples Spanish language interviewing: Researchers should carefully consider question of whether to offer Spanish, and also questions of how many Spanish interviews, translation methods, etc.

23. Hispanic Attitudes Toward Immigration and the Language of the Interview Kate Kenski, Ph.D and Marisa Enriquez Department of Communication The University of Arizona

24. Why study Hispanics? • Latinos projected to be 24.4% of population by 2050 (US Census, 2006) • Growth rate • From 2000-2010 Latino population grew 43% (US Census, 2010) • Language • 6% Hispanic voters get news only in Spanish • 40% Hispanic voters get news in both English and Spanish (Suro, 2004) Kenski & Enriquez (2011) AAPOR

25. Data Source and Research Question • 2008 National Annenberg Election Survey • RDD telephone survey • AAPOR RR1: 19% • Examined differences between Hispanic respondents who completed the interview in Spanish versus those who completed the interview in English on three topics related to immigration • Path to citizenship • Border fence • Driver’s license Kenski & Enriquez (2011) AAPOR

26. Variables • Independent Variables • Language of Interview, demographic characteristics, citizenship, political ideology, party id • Dependent Variables • 5-point scale: 1=strongly oppose to 5=strongly favor • Provide a path to citizenship for some illegal aliens who agree to return to their home country for a period of time and pay substantial fines • Increase border security by building a fence along part of the U.S. border with Mexico. • Driver’s licenses to undocumented or illegal immigrants Kenski & Enriquez (2011) AAPOR

27. Hypotheses, part 1 COMPARED TO HISPANICS INTERVIEWED IN SPANISH, THOSE INTERVIEWED IN ENGLISH ARE - • H1a: more educated • H1b: have higher household incomes • H1c: more likely to be born in the U.S. • H1d: less likely to attend religious services • H1e: more likely to express a political party affiliation Kenski & Enriquez (2011) AAPOR

28. Hypotheses, part 2 COMPARED TO HISPANICS INTERVIEWED IN SPANISH, THOSE INTERVIEWED IN ENGLISH ARE - • H2a: less likely to support a path to citizenship • H2b: more likely to support a border fence • H2c: less likely to support allowing illegal immigrants to have a driver’s license Kenski & Enriquez (2011) AAPOR

29. Results • H1a (education): Supported, bivariate and multivariate • H1b (income): Supported, bivariate and multivariate • H1c (U.S born): Supported, bivariate and multivariate • H1d (religious attendance): Supported, bivariate • H1e (party affiliation): Supported, bivariate • H2a (path to citizenship): Not supported, significant in opposite direction from prediction • H2b (border fence): Supported, bivariate and multivariate • H2c (driver’s license): Supported, bivariate and multivariate Kenski & Enriquez (2011) AAPOR

30. Conclusion • Should not assume English speaking Hispanics represent entire group • No opportunity for Hispanics to answer in Spanish may systematically bias survey estimates Kenski & Enriquez (2011) AAPOR

31. Questions? Thanks for listening! SarahC@kff.org LizH@kff.org ddutwin@ssrs.com enriquem@email.arizona.edu