Measuring the Effects of Unit Nonresponse in Establishment Surveys

1. Measuring the Effects of Unit Nonresponse in Establishment Surveys Clyde Tucker and John Dixon U.S. Bureau of Labor Statistics David Cantor Westat

2. Acknowledgement We would like to thank Bob Groves and Mike Brick for the use of their materials from their short course “Practical Tools for Nonresponse Bias Studies.” We also thank Bob for the use of materials from his 2006 POQ article “Nonresponse Rates and Nonresponse Bias in Household Surveys,” Public Opinion Quarterly, 70, 646-675.

3. Factors Affecting Nonresponse Outside Survey Organization Control • Three clusters of factors identified as being outside the control of the survey organization (Willimack et al., 2002) • External environmental attributes (“climate”) • Characteristics of the sample unit • Characteristics of the establishment employee(s) who decide whether to join a survey, priority of responding, and length of participation • Effects attributable to all three factors are widespread and substantially affect nonresponse • Even if nonresponse not increasing, these factors make it hard to maintain the status quo

4. Some Examples of These Factors Downsizing decreases staff available to provide data Increased firm size due to mergers and acquisitions increases complexity reporting burden “Gatekeeping” poses significant barriers Attitudes of owners or key managers toward government and data confidentiality Whether or not anyone in the firm actually uses data products from the survey Staff turnover The growing accounting practice of using third parties such as payroll processing or accounting firms

5. Nonresponse Error for Sample Mean In simplest terms OR Respondent Mean = Full Sample Mean + (Nonresponse Rate)*(Respondent Mean – Nonrespondent Mean)

6. Thinking Causally About Nonresponse Rates and Nonresponse Error • Key scientific question concerns mechanisms of response propensity that create covariance with survey variable where is the covariance between the survey variable, y, and the response propensity, p • What mechanisms produce the covariance?

7. Reporting Bias • The relative bias provides a measure of the magnitude of the bias. Interpreted similar to a percent, it is useful in comparing bias from survey measures which are in different scales. • : • Where: • Rel B ( ) = the relative bias with respect to the estimate, .

8. Reporting Bias • The bias ratio provides an indication of how confidence intervals are affected by bias: • Where: = the standard error.

9. What does the Stochastic View Imply? • Key issue is whether what influences survey participation also influences the survey variables • Increased nonresponse rates do not necessarily imply increased nonresponse error. Although lower propensity will tend to increase error. • Hence, investigations are necessary to discover whether the estimates of interest might be subject to nonresponse errors because of a correlation between p and y

10. Alternative Causal Models for Studies of Nonresponse Rates and Nonresponse Bias

11. A More Specific Theory Relating Nonresponse to Bias • Levels of bias will differ by subpopulations • Differences between estimates from the total sample and just respondents will be greatest on either end of the nonresponse continuum, but potential bias greatest when response rates are low • For example: Bias in a business survey may be greatest in the Services sector because it often has the lowest response rates

14. Nonresponse Bias Study Techniques • Comparison to other estimates (benchmarking) • Nonresponse bias for estimates based on variables available on sample • Studying variation within the respondent set • Altering the weighting adjustments

15. Weights and Response Rates • A base or selection weight is the inverse of the probability of selection of the unit. The sum of all the sampled units’ base weights estimates the population total. • When units are sampled using a complex sample design, suggest using (base) weights to compute response rates that reflect the percentage of the sampled population that respond. Unweighted rates are useful for other purposes, such as describing the effectiveness of the effort. • Weighted response rates are computed by summing the units’ base weights by disposition code rather than summing the unweighted counts of units. • In establishment surveys, it is useful to include a measure of size (e.g., number of employees or students) to account for the units relative importance. The weight for computing response rates is the base weight times the measure of size.

16. Weights and Nonresponse Analysis • A general rule is that weights should be used in nonresponse analysis studies so that relationships at the population level can be examined. Guides for choosing the specific weights to use are: • Use base weights for nonresponse bias studies that compare all sampled respondents and nonrespondents. Weights adjusted for nonresponse may be misleading in this situation. • Use fully adjusted weights for nonresponse bias studies that compare survey estimates with data from external sources. One important exception is when the survey weights are poststratified. In this case, weights prior to poststratification are generally more appropriate.

17. 1. Comparison to Other Estimates -- Benchmarking • Data or estimates from another source that are closely related to respondent estimates used to evaluate bias due to nonresponse in the survey estimates • Assume that alternative data source has different sources of measurement error and/or is a superior measure to target survey.

18. 1. Benchmarking Survey Estimates to those from Another Data Source • Another survey or administrative record system may contain estimates of variables similar to those being produced from the survey • Difference between estimates from survey and other data source is an indicator of bias (both nonresponse and other)

19. 1. How to Conduct a Nonresponse Bias Benchmark Study • Identify comparison estimates • surveys with very high response rates • administrative systems with different measurement error properties • Assess major reasons why the survey estimates and the estimates from the comparison sources differ • Compute estimates from the survey (using final weights) and from the comparison source to be as comparable as possible (often requires estimates for domains) • The difference is an estimate of the direction, or perhaps the magnitude, of the bias

20. Pro’s and Con’s of Benchmark Comparison to Estimate NR Bias • Pro’s • Relatively simple to do and often inexpensive • Estimates from survey use final weights and are thus relevant • Gives an estimate of bias that may be important to analysts • Con’s • Estimated bias contains errors from the comparison source as well as from the survey; this is why it is very important that the comparison source be highly accurate • Measurement properties are generally not consistent for survey and comparison source; often is largest source of error • Item nonresponse in both data sets reduces comparability • Hard to find comparable data for establishment surveys (IRS records?) • More common in household surveys

21. 2. Using Variables on Respondents and Non-respondents • Compare statistics available on both respondents and non-respondents • The extent there is a difference is an indication of the bias

22. Possible Sources of Data on Respondents and Non-respondents • Sampling frame variables • Matched variables from other data-sets • Screener information

23. Pro’s and Con’s of Using Data on both Respondents and non-respondents • Pro’s • Measurement properties for the variables are consistent for respondents and nonrespondents • Bias is strictly due to nonresponse • Provides data on correlation between propensity to respond and the variables • Con’s • Bias estimates are for the variables; only variables highly correlated with the key survey statistics are relevant • The method assumes no nonresponse adjustments are made in producing the survey estimates; if variables are highly correlated, then they could be used in adjustment

24. The CES Study • J. Dixon and C. Tucker (ICES3), “Assessing Bias in Estimates of Employment” • Collects employment, hours and earnings monthly from a current sample of over 300,000 establishments • Tracks the gains and losses in jobs in various sectors of the economy • In this paper, nonresponse bias work on this survey focuses on estimating bias for establishment subpopulations with different patterns of nonresponse using data from the 2003 CES and QCEW (state UI files)

25. Link relative estimate of employment (Y) Let Yt be the estimate for a primary cell for month t, then Yt = Rt,t-1 * Yt-1 • where Rt,t-1 is the ratio of the total sample employment in month t to the total sample employment in month t-1 for all sample units reporting data for both months.

26. Estimate of Bias • Using the most recent employment reports in the QCEW (not CES) for both responders and nonresponders • Compare the link relative for respondents to that for nonrespondents • Results presented are not weighted by probability of selection, but weighted results show similar patterns • At this point, not comparing the link relative of responders to the entire sample

27. Quantile Regression • Bias analysis performed at the establishment level on subpopulations defined by size and industry • Testing for the difference in employment between CES responders and nonresponders. Y=a+Bx+e where x is an indicator of nonresponse (essentially a t-test). • Since size of firm is theorized to relate to nonresponse, the coefficients relating nonresponse to employment is likely to be different for different size firms. • Quantile regression examines the coefficients for different quantiles of the distribution of the sizes of firms. • Since industries can be expected to have different patterns, the quantile regressions are done by industry group.

28. Distribution of size and the quantile regression curve

29. Quantile regression using the log of size.

30. MSA percent bias predicted by response rate for Mining

31. MSA percent bias predicted by response rate for Food Manufacturing

32. MSA percent bias predicted by response rate for Retail Trade

33. MSA percent bias predicted by response rate for Accommodation and Food Services

34. Hing (1987). Nonresponse bias in expense data from the 1985 national nursing home survey. Proceedings of the Survey Research Methods Section, American Statistical Association, 401-405. • Purpose: Estimate cost of care in nursing homes • Target population: Nursing home facilities in U.S. • Sample design: Stratified list sample of facilities, facilities sampled with probabilities proportionate to estimated number of beds, second stage sample of residents and staff • Mode of data collection: In-person interview of facility administrator, with drop-off self-administered Expense questionnaire for accountant • Response rate: Facility q’naire: 93%; Expense q’naire: 68% of those responding to Facility interview • Target estimate: Estimated cost of care • Nonresponse error measure: Comparison of Facility questionnaire items for respondents and nonrespondents of Expense questionnaire

35. Using the Facility Questionnaire to Estimate Nonresponse Bias based on Participation in Expense Questionnaire Tables 2 and 5 from Hing (1987)

36. Conclusions • Smaller nursing homes underrepresented; thus, respondent estimates overestimate averages on size-related attributes • Analysis suggested poststratification by ownership type would significantly reduce biases • Limitation: • Nonresponse bias estimate does not reflect nonresponse on Facility questionnaire

37. 3. Weighting Adjustments • Alter estimation weights and compare the estimates using the various weights to evaluate nonresponse bias. Weighting methods may include poststratification, raking, calibration, logistic regression, or even imputation.

38. Adjust Weights Using Model of Characteristics • Weighting can reduce nonresponse bias if the weights are correlated with the estimate. Auxiliary data in weighting that are good predictors of the characteristic may give alternative weights that have less bias. If the estimates using the alternative weights do not differ from the original estimates, then either the nonresponse is not resulting in bias or the auxiliary data does not reduce the bias. • If the estimates vary by the weighting scheme, then the weighting approach should be carefully examined and the one most likely to have lower nonresponse bias should be used.

39. How to Conduct Nonresponse Bias Analysis Using Weights From Modeling Characteristics • Using weighting method such as calibration estimation with these variables and produce alternative weights. • Compute the difference between the estimates using the alternative weights and the estimates from the regular weights as a measure of nonresponse bias for the estimate.

40. Pro’s and Con’s of Comparing Alternative Estimates Based on Modeling the Estimate • Pro’s • If good predictors are available, then it is likely that the use of these in the weighting will reduce the bias in the statistics being evaluated • If the differences in the estimates are small, it is evidence that nonresponse bias may not be large • Con’s • Recomputing weights may be expensive • If good correlates are not available then lack of differences may be indicator of poor relationships rather than the absence of bias • The approach is limited to statistics that have high correlation with auxiliary data

41. 4. STUDYING VARIATION WITHIN THE RESPONDENTS: Level of Effort • Some nonresponse models assume that those units that require more effort to respond (more callbacks, incentives, refusal conversion) are similar to the units that do not respond • Characteristics are estimated for respondents by level of effort (e.g., response propensity scores) • Models fitted to see if it fits and can be used to estimate characteristics of nonrespondents

42. Analyze Level of Effort • Associate level of effort data to respondents (e.g., number of callbacks, ever refused, early or late responder) • Compute statistics for each level of effort separately (usually unweighted or base weights only) • If there is a (linear) relationship between level of effort and the statistic, then may decide to extrapolate to estimate statistic for those that did not respond • Often more appropriate to do the analysis separately for major reasons for nonresponse

43. Pro’s and Con’s of Using Level of Effort Analysis to Estimate Bias • Pro’s • Simple to do, provided data collection systems capture the pertinent information • In some surveys may provide a reasonable indicator of the magnitude and direction of nonresponse bias • Con’s • Highly dependent on model assumptions that have not been validated in many applications • Difficult to extrapolate to produce estimates of nonresponse bias without other data

44. Brick et al (2007) “Relationship between length of data collection period, field costs, and data quality” Paper presented at ICES III, Montreal, Canada • Purpose: Estimate amount and condition of research space for science and engineering • Target population: Colleges and universities. • Sample design: Census • Mode of data collection: Web and mail • Response rate: 94% • Target estimate: Square feet • Nonresponse error measure: Relative bias when level of effort is cut by 25%

45. Relative bias for Facilities Survey estimates for academic institutions, by field and response level

46. Summary • The patterns are not consistent • The 75% response level generally exhibits larger bias than the other response levels, but generally not statistically significant • No significant bias if data collection were terminated at the 88% response level

47. 5. Followup of Nonrespondents • Use of respondent data obtained through extra-ordinary efforts as comparison to respondent data obtained with traditional efforts • “Effort” may include callbacks, incentives, change of mode, use of elite corps of interviewers

48. How to Do a Nonrespondent Followup Study • Define a set of recruitment techniques judged to be superior to those in the ongoing effort • Determine whether budget permits use of those techniques on all remaining active cases • If not, implement 2nd phase sample (described later) • Implement enhanced recruitment protocol • Compare respondents obtained in enhanced protocol with those in the initial protocol

49. Pro’s and Con’s of Nonresponse Followup Study • Pro’s • Direct measures are obtained from previously nonrespondent cases • Same measurements are used • Nonresponse bias on all variables can be estimated • Con’s • Rarely are followup response rates 100% • Requires extended data collection period

50. Marker, et al (2005).Terrorism Risk Insurance Program: Policyholders Survey. Final report prepared for the Department of Treasury. Westat: Rockville, MD • Purpose: Estimate use of Terrorism Insurance by Businesses • Target population: Businesses and state/local government offices in the U.S.. • Sample design: Stratified sample. • Mode of data collection: Web and mail • Response rate:17% • Target estimate: Estimated percent that have insurance • Nonresponse error measure: Indicators of use of terrorisminsurance