Presented at the 2005 Annual Meeting fo the American Association of Public Opinion Research

1. Data Collection Mode Effects Controlling for Sample Origins in a Panel Survey: Telephone and Internet Presented at the 2005 Annual Meeting fo the American Association of Public Opinion Research Mike Dennis, Knowledge Networks Cindy Chatt, University of Nebraska-Lincoln Alicia Motta-Stanko, Knowledge Networks Paul Pulliam, RTI International

2. Acknowledgements • The survey designed by: • RTI International • The Odum Institute at the University of North Carolina • The authors wish to thank RTI International for permission to analyze the data used in this paper and disseminate the findings.

3. The Purpose of the Research • What matters more in accounting for variance in survey responses? • Mode of Data Collection (Telephone versus Internet) • or • Sample Origin (KN panel sample versus KN panel rejecters/non-cooperators) • This is a contest of data collection mode versus sample quality. • The main purpose of this research is to help further our understanding of the respective contributions of mode of data collection (phone vs. web) and sample origin (cooperators vs. non-cooperators) in explaining variance in survey response data.

4. Background of Study • The analysis is based on the Survey on Civic Attitudes and Behaviors After 9/11. • Study was created with a non-response follow-up study and implemented by Knowledge Networks from January to March 2002. • The study is important because: It is the first mode-effects study using KN data that effectively controls for sample type (persons who joined the KN panel and participate in KN studies, versus those who refused to join the panel or who refused to participate in the panel study).

5. Questionnaire: Public Policy & Civic Attitudes & Behaviors • The survey questionnaire had several modules of interests: ratings of Bush and Gore (early 2002!), attitudes toward terrorism, the adequacy of governmental response to bioterrorism and terrorism more generally, attitudes toward what the government should be doing and would do in the event of a terrorist attack, civic participation and civic values questions, and background questions on religious faith and other aspects. • The survey was approximately 25 minutes in length.

6. Main Finding • The mode of data collection is more often a significant factor in accounting for variance in survey responses than is the origin of the sample. When controlling for panel experience and demographics, for every one question where sample origin was a significant factor, mode of data collection was significant for six questions. • MODE • Sample

7. Sample Design for the Study Random sample for each of three groups.

8. Sample Composition, Mode, & Sample Size 1Nonresponse sample consists of ‘panel rejecters’ and those panelists that did not respond to the Internet survey.

9. Sample Representativeness: Demographic Comparison • Each interviewed group was compared to the January 2002 Current Population Survey (Census) • Main Findings: • Mean Error (Unweighted): • Panel by Web: 2.8 percentage points • Panel by Phone: 4.1 percentage points • NRFUS by Phone: 3.6 percentage points

10. Summary of Multivariate Analyses: Count of Statistically Significant Predictors of Answers by Mode and NRFUS (p < .05)

11. Results of Multivariate Analyses • Phone respondents more likely to: • Disagree that bioterrorism is the most important problem • Seek info on anthrax from …web, hotlines, national TV, own doctor, local government, other • Rate Pres. Bush and Al Gore higher on feeling thermometer • Discuss politics • Discuss community issues • Help neighbors • Be happy about their neighborhood • Have pride in their neighborhood • Have a sense of belonging in neighborhood • Have a neighborhood that gives them pleasure • Rely on neighbors • Trust others • Enjoy mixing socially with others

12. Freq Distributions by Group (Unweighted) No large differences across the groups for grading political job performance.

13. Freq Distributions by Group (Unweighted) Telephone mode: Higher expectations for information from CDC.

14. Freq Distributions by Group (Unweighted) Telephone mode: Higher report of information-seeking behavior.

15. Freq Distributions by Group (Unweighted) Telephone mode: Higher report of ‘good neighbor’ behavior.

16. Freq Distributions by Group (Unweighted) Telephone mode: Higher report of positive feelings toward own neighborhood.

17. Freq Distributions by Group (No weight) Telephone mode: Higher report of positive sociability self-perceptions.

18. Discussion

19. Major Differences in Response Caused by Mode? • Major differences existing between Internet and telephone modes of data collection include: • Presence or absence of interviewer • Dependence on visual or aural communication • Interviewer or respondent control of interview pace or information sequence • Each difference can lead to multiple mode effects seen in the data. • (Dillman, Sangster, Tarnai, and Rockwood, 1996)

20. Presence or Absence of Interviewer • Can lead to: • Tendency to answer on extreme ends of response scales • Non-differentiation • Social desirability / self-presentation bias

21. Presence or Absence of Interviewer:Neighborhood Statements • 1. I am happy to live in this neighborhood. • 2. I really see myself as a part of this neighborhood • 3. I feel a sense of belonging to this neighborhood. • 4. Being in this neighborhood gives me a lot of pleasure. • 5. If there are things in my neighborhood that need to be fixed or improved, I would be able to get my neighbors to do something about it.

22. Very similar results for self-perception items (e.g., “I am trusting.”)

23. Evidence of Non-Differentiation and Recency Effects on Phone (Neighborhood Statements) *Significant differences at p<.05 between Internet and Telephone and Internet and NRFUS for Completely Agree

24. Conclusions • Sample origins do not appear to be having much of an effect on results compared to mode of data collection • Differences in the data are consistent with inherent differences between Internet and telephone data collection, leading to patterns in the phone-collected data reflecting: • Social desirability bias • Extreme positive responses on answer scales • Answer distribution differences due to communication type differences • Item Non-differentiation • Study limitation/Buyer Beware: Questionnaire potentially laden with more questions (compared to many surveys) susceptible to social desirability bias.

25. Thank you! mdennis@knowledgenetworks.com (650) 289-2160