Survey Non-response

1. Survey Non-response Survey Research and Design Spring 2006 Class #7

2. Today’s objectives • To understand causes and consequences of survey non-response • To re-visit ways to reduce non-response • To learn how to assess non-response bias Survey Research and Design (Umbach)

3. Quality perspective (Groves, et al., p. 48) Measurement Representation _ Y Target Population m1 Construct Coverage Error Validity _ yc Measurement Yi Sampling Frame Sampling Error Measurement Error _ ys Sample Response yi Non-response Error Processing Error _ yr Respondents Edited Response yip Adjustment Error Postsurvey Adjustments _ yrw _ yprw Survey Statistic Survey Research and Design (Umbach)

4. Non-response • Unit – person chosen refuses to cooperate entirely • Increasing over time • Item – person chosen does not respond to an item (or items) • Introduces non-response bias • Proportion of sample cases not responding • Differences in statistic between respondents and non-respondents Survey Research and Design (Umbach)

5. Types of unit non-response • Non-contact or failure to deliver as a result of access impediments • Refusal • Decision to participate is made very quickly • However, persons who refuse initially often accept when contacted later • Inability to participate Survey Research and Design (Umbach)

6. Why does non-response occur? • Opportunity cost • Cost of participation too high • Often associated with busy people • Social isolation • Those socially isolated less likely to respond • e.g., low SES, high SES • Topic of interest • Those interested in topic more likely to respond Survey Research and Design (Umbach)

7. Why does non-response occur? • Over-surveying or survey fatigue • Leverage-salience theory • Different people place varying levels of importance on features of the survey request (e.g., survey topic, burden, sponsor, incentive) • Not known to the researcher • Scale metaphor • Leverage=value people place on a specific attribute • Salience=How important attribute is in description of request through words or symbols Survey Research and Design (Umbach)

8. Groves, Cialdini, and Couper (1992) offer a complementary view In what ways do they complement and add to these ideas of non-response? Survey Research and Design (Umbach)

9. Culture • Orientations • Individualist versus collectivist • Power distance • Vertical versus horizontal relationships • Communication styles • Context requirements • Nonverbal behavior • Self-disclosure • Social participation • Minority oppression and opposition • Social distance • Helping behavior How do these affect survey response? Survey Research and Design (Umbach)

10. The effect of unit nonresponse • Amount of bias can be difficult to determine • Can attempt to correct • Proxy respondents • Statistical adjustments • Survey nonrespondents Survey Research and Design (Umbach)

11. Reducing non-response • Remember Dillman’s Tailored Design • Consider… • Number and timing of contacts • Data collection period • Sponsorship • Multiple invitations • Incentives • Burden • Introductory behavior • Types and content of contacts • Mode switches Survey Research and Design (Umbach)

12. How did Porter and Whitcomb assess non-response bias? What did they find? • In what other ways can we assess nonresponse bias? Survey Research and Design (Umbach)

13. I = Complete P = Partial R = Refusal NC = Noncontact O = Other eligible UH = Unknown if household/occupied HU UO = Unknown eligibility, other e = Estimated proportion of cases of unknown eligibility that are eligible Survey Research and Design (Umbach)

14. I = Complete P = Partial R = Refusal NC = Noncontact O = Other eligible UH = Unknown if household/occupied HU UO = Unknown eligibility, other e = Estimated proportion of cases of unknown eligibility that are eligible Survey Research and Design (Umbach)

15. I = Complete P = Partial R = Refusal NC = Noncontact O = Other eligible UH = Unknown if household/occupied HU UO = Unknown eligibility, other e = Estimated proportion of cases of unknown eligibility that are eligible Survey Research and Design (Umbach)

16. I = Complete P = Partial R = Refusal NC = Noncontact O = Other eligible UH = Unknown if household/occupied HU UO = Unknown eligibility, other e = Estimated proportion of cases of unknown eligibility that are eligible Survey Research and Design (Umbach)

17. I = Complete P = Partial R = Refusal NC = Noncontact O = Other eligible UH = Unknown if household/occupied HU UO = Unknown eligibility, other e = Estimated proportion of cases of unknown eligibility that are eligible Survey Research and Design (Umbach)

18. Causes of item non-response • Inadequate comprehension of the intent of the question • Judged failure to retrieve adequate information • Lack of willingness or motivation to provide information How then do you reduce item nonresponse? Survey Research and Design (Umbach)

19. What is the Wabash National Study of Liberal Arts Education?

20. Background • Purpose is to examine educational conditions, practices, and programs and the extent to which they foster specified outcomes associated with the liberal arts • Mixed method approach using cross-sectional quantitative measures, narrative interview data, and institutional case studies (through the use of document analysis, individual interviews, and focus groups) Survey Research and Design (Umbach)

21. Liberal Arts Outcomes • Moral reasoning (DIT-2) • Reflective judgment (RCI) • Psychological well-being (PWBS) • Leadership (SRLS) • Life-long learning (Need for Cognition) • Intercultural effectiveness (IDI) • Integration of learning (not quant) Survey Research and Design (Umbach)

22. Pilot Study Sample • Inst. sample: 4 convenience schools (1 research, regional, liberal arts, and community college) • Student quant. sample: 50 students from each institution in each of the four classes (CC had 100 students) • Student i-view sample: 25 students each from first & senior year Survey Research and Design (Umbach)

23. Pilot Study Quantitative Methods • Students were invited by mail to participate in October 2004. Registration materials were enclosed. Students registered by sending in a hard copy or by visiting secure website. • Registered students were then sent the college experience survey in the mail in Dec. 2004. They were allowed to choose to complete the hard copy version or to log in using their unique id and complete the survey online. • Students who completed a survey were then asked to attend an assessment session on their campus. Campuses held evening or weekend assessment sessions in late Jan. 2005 through Feb. 2005. Survey Research and Design (Umbach)

24. Registration: # Sent = 3820 # Returned = 909 Paper = 105 Web = 804 Response rate 23.8% Surveys # Sent = 907 # Completed = 723 Paper = 499 Web = 224 Response rate 18.9% (from total) Response rate 79.7% (from registration) Assessments # Completed = 601 Response rate 15.7% (from total) Response rate 66% (from registration) Response rate 83.1% (from survey) Attrition and Non-response Survey Research and Design (Umbach)

25. Quantitative Non-response Survey Research and Design (Umbach)

26. Design a non-response study (handout) • What would you do? What questions would you seek to address? • Why these questions? How does having a better understanding of these dimensions of non-response contribute to our overall understanding? • How would you do it? (implementation) • What are some key issues that you should think about? • What types of analyses would you run? Survey Research and Design (Umbach)

27. Group project • How will you implement your survey to reduce unit non-response? • What measures will you take to assess non-response bias? Survey Research and Design (Umbach)

28. For next week: Writing questions • Dillman, Ch. 2 • Groves et al. Ch. 7 • DeVellis Ch. 5 • Fowler Ch. 3 • Due: Application exercise 2 • Distributed: Take-home midterm distributed Survey Research and Design (Umbach)