1 / 51

The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates

The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates Diane K. Willimack U.S. Census Bureau 10 Economic Census 11 Outline Motivation Economic Census Conceptual framework for nonresponse in establishment surveys

niveditha
Download Presentation

The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates Diane K. Willimack U.S. Census Bureau 10 Economic Census 11

  2. Outline • Motivation • Economic Census • Conceptual framework for nonresponse in establishment surveys • Data analysis, results, and conclusions • Number of pages • Economic conditions

  3. U.S. Economic Census • Establishment level data • Collected every 5 years • Reference year ends in 2 or 7 • Collection year is the following year

  4. U.S. Economic Census continued • Self-administered • Paper mail-out/mail-back • 2002: Electronic via downloadable software • Tailored by industry • 2002: 520 different questionnaire versions

  5. Thank you for completing the 2002 Economic Census Motivation • Changes in format affecting the 2002 Economic Census • Paper size: Legal  Letter • Page layout: 2 columns  1 column • Number of pages increased • Response rates decline???

  6. The Data • 1992, 1997, and 2002 Economic Censuses • Check-in rates • Receipts + 2nd time Undeliverables Mail Counts • 1st week of June, 1993, 1998, 2003 • Unit of analysis = questionnaire version • n = 1459 questionnaire versions

  7. Check-In Rates by Year,1st week of June of the Collection Year

  8. Research Question • Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses?

  9. A Conceptual Framework forBusiness Survey Participation Out of Survey Organization Control Under Survey Organization Control External Environment Survey Design The Business The Respondent Business Goals Response Burden Survey Participation Decision Source: Willimack, D.K., Nichols, E., and Sudman, S., (2002), “Understanding Unit and Item Nonresponse in Business Surveys,” in Survey Nonresponse, Groves, et al. (eds.) New York: Wiley.

  10. Concept (External Environment; Characteristics of the Business; Characteristics of the Respondent; Survey Design) Measure Economic Conditions %GDP, %Employment Industrial Sector ---------------------- #pp for the form, availability of electronic mode Defining the Model Response Rate = f

  11. Concept (External Environment; Characteristics of the Business; Characteristics of the Respondent; Survey Design) Measure Economic Conditions %GDP, %Employment Industrial Sector ---------------------- #pp for the form, availability of electronic mode The Model Response Rate = f

  12. Main Effects Model:What is the relationship, regardless of the census year? Response Rate = a + b (#pages) + [effects of economic variables] + gi (Sector*) + h (Mode) + [error term] * The omitted variable among the Sector dummies is Mining.

  13. Care in Interpreting Results • Behavioral model • Explanatory, NOT predictive • Estimated coefficients • Relative magnitude • Direction (+ or -) • Significance

  14. Main Effects Model

  15. Research Question • “Does 2002 differ from previous censuses?” • H0: 1992 = 1997 = 2002 • Pairwise comparisons

  16. Interaction Effects Model:Does the effect of pages differ by census year? = a + b92(#pp) + c92(#pp)2 + d92 (#pp)3 + b97(#pp) + c97(#pp)2 + d97 (#pp)3 + b02(#pp) + c02(#pp)2 + d02 (#pp)3 + [effect of economic variables] + gi (Sector*) + h (Mode) + [error term] Response Rate * The omitted variable among the Sector dummies is Mining.

  17. Does the effect of pages differ by census years?Testing H0: 1992 = 1997 = 2002

  18. Interaction Effects Model:Does the effect of pages differ by census year?

  19. Interaction Effects Model:Does the effect of pages differ by census year?

  20. Interaction Effects Model:Does the effect of pages differ by census year?

  21. Interaction Effects Model:Does the effect of pages differ by census year?

  22. Estimated Model 1992 Response Rate Number of Pages

  23. Estimated Model 1997 Response Rate Number of Pages

  24. Estimated Model 2002 Response Rate Number of Pages

  25. Research Questions • Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses?

  26. Research Question #1 Conclusion • Negative effect • #pages   response rates  • Caution: Behavioral model vs. Controlled Experiment • Cannot identify causality • Cannot isolate factors

  27. Research Question #1 Conclusioncontinued • Reject H0: 1992 = 1997 = 2002 • 2002 differed from the previous censuses (per results from pairwise comparisons) • Negative effect of #pages on economic census response appears to have diminished over time

  28. A Conceptual Framework forBusiness Survey Participation Out of Survey Organization Control Under Survey Organization Control External Environment Survey Design The Business The Respondent Business Goals Response Burden Survey Participation Decision Source: Willimack, D.K., Nichols, E., and Sudman, S., (2002), “Understanding Unit and Item Nonresponse in Business Surveys,” in Survey Nonresponse, Groves, et al. (eds.) New York: Wiley.

  29. Household SurveyLiterature • Better economic times  refusal rates increased (in a household survey) • Harris-Kojetin and Tucker, 1999, “Exploring the Relation of Economic and Political Conditions with Refusal Rates to a Government Survey,” JOS, 15:167-184

  30. Research Questionscontinued • Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses? • Did economic conditions affect economic census response rates?

  31. Concept (External Environment; Characteristics of the Business; Characteristics of the Respondent; Survey Design) Measure Economic Conditions %GDP, %Employment Industrial Sector ---------------------- #pp for the form, availability of electronic mode The Model Response Rate = f

  32. Main Effects Model:What is the relationship, regardless of the census year? Response Rate = a + b (#pages) + e (%GDP) + f (%Employment) + gi (Sector*) +h (Mode) + [error term] * The omitted variable among the Sector dummies is Mining.

  33. Main Effects Model

  34. Interaction Effects Model:Do the effects of economic conditions differ by census year? = a + [effect of page variables] + e92 (%GDP)+ f92 (%Employment) + e97 (%GDP)+ f97 (%Employment) + e02 (%GDP)+ f02 (%Employment) + gi (Sector*) + h (Mode) + [error term] Response Rate * The omitted variable among the Sector dummies is Mining.

  35. Do the effects of economic conditions differ by census year?Testing H0: 1992 = 1997 = 2002

  36. Research Questions • Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses? • Did economic conditions affect economic census response rates?

  37. Research Question • Does the effect of number of pages on response rates vary with economic conditions?

  38. Mixed Effects Models For each census year (i = 1992, 1997, 2002): = a + bi(#pp) + ci (#pp)2 + di(#pp)3 + ei (% GDP) + fi (% Emply) + g (Sector*) + h (Mode) Response Rate + ji(#pp) * (% GDP) + ki (#pp)2 * (% GDP) + mi (#pp) * (% Emply) + ni (#pp)2 * (% Emply) +  * The omitted variable among the Sector dummies is Mining.

  39. Summary of Results Under the Mixed Effects Model • Variables that become non-significant (with p-values) • (#pp)2for 1992 (0.1159) • (#pp)3for 2002 (0.3753)

  40. %GDP (0.0282) (#pp) * (%GDP) (0.0103) (#pp)2 * (%GDP) (0.0087) %Emply (0.0322) (#pp) * (%Emply ) (0.0117) (#pp)2 * (%Emply ) (0.0082) Summary of Results Under the Mixed Effects Modelcontinued Economic variables that are significant(all are for 2002)(with p-values)

  41. What was happening in the U.S. economy in 2001-2003? • Economic recovery – beginning to emerge from a recession.

  42. What was happening in the U.S. economy in 2001-2003? • Economic recovery – beginning to emerge from a recession. •  GDP increasing sharply •  Employment declining, “bottoming out” • Unstable economy • Sharp opposite movements in GDP and Employment

  43. What is the nature of the interaction between economic conditions and #pages? • GDP and Employment effects tend to be offsetting. • GDP moderates the page effect on response rates. • Emply exacerbates the page effect on response rates. • Magnitude of Employment effect is larger.

  44. What is the nature of the interaction between economic conditions and #pages? continued Employment effect Response Rates Number of pages

  45. Economic Effects Postscript: Economic Theory of Lagged Expectations • In-depth examination of effects of quarterly employment change on response rates • Statistically significant impact of changes 4 quarters and 3 quarters prior to beginning of economic census data collection.

  46. Interpreting the Economic Effect • Unstable economy • Employment decline and GDP growth • Uncertainty • Riskier to divert resources to non-productive activities • Cost-benefit • Survey response results in a cost with no associated productive benefit.

  47. Interpreting the Economic Effect continued • The bottom line • In a risky economic environment… • Employees are too valuable to devote to filling out forms. • Forms that appear longer (#pages) are presumed to take longer and be more costly to complete… • Response rates are in jeopardy!

  48. Summary:Number of Pages • Negative effect • #pages   response rates  • Cubic functional relationship • Negative effect of #pages on economic census response appears to have diminished over time

More Related