A Latent Class Call-back Model for Survey Nonresponse

1. A Latent Class Call-back Model for Survey Nonresponse Paul P. Biemer RTI International and UNC-CH Michael W. Link Centers for Disease Control and Prevention

2. Outline • Motivation for the study • Early cooperator effects (ECE) in the Behavior Risk Factor Surveillance System (BRFSS) • Call-back models • Manifest and latent • Model extensions • Application to the BRFSS • Results • Summary and conclusions

3. Terms and definitions • Cooperators = units that will eventually respond at some request or call-back • Non-cooperators (also calledhardcore nonrespondents) = units that will not respond to any call-back

4. Terms and definitions (cont’d) • Early cooperator = Cooperators that respond at early calls (say, 5 or less) • Later cooperators = Cooperators that respond at later calls (say, 6 or more) • Early cooperator effect (ECE) = expected difference in estimates based on early vs. early + later cooperators (say, )

5. Response rates as a function of number of call attempts Number of call attempts

6. Illustration 1-Have you ever been told by a doctor, nurse or other health professional that you had asthma? Small ECE maximum of 5 calls is adequate

7. Illustration 2- During the past 12 months, have you had a flu shot? Larger ECE max of 5 call attempts may be biasing Could consider other definitions of “early cooperator.”

8. Why study ECE? • Effort (and costs) could be saved if ECE is small • If ECE is not small, adjustments may be applied to reduce it • May need to adjust for HCNRs, not only later cooperators

9. What adjustments can be applied to reduce the ECE? • Nonresponse adjustments • Requires characteristics of nonrespondents • Lack of information a limitation for some surveys • Post-stratification adjustments • Requires known target population totals within adjustment cells • Variables limited to those available externally • Call-back model adjustments • Assumes response propensity is function of level of effort required to obtain a response and grouping variables • Related work of Drew and Fuller (1980), Politz and Simmons (1949), others

10. ECE in the BRFSS Survey details • One of the largest RDD surveys in the world • Estimates the prevalence of risk behaviors and preventive health practices • Monthly, state-based, cross-sectional survey • Target population is adults in telephone hh’s • Data source: 2004 survey with ~300,000 interviews

11. ECE in the BRFSS (cont’d) • Early cooperator defined as responding with 5 fewer call attempts • Examined differences in • demographic characteristics • 10 selected health characteristics overall and by demographic domain • ECE estimated by • Data weighted by base weights only

12. Typical Values of ECE

13. Typical Values of ECE (cont’d)

14. Summary of the Results • Early cooperators are different from later cooperators on many dimensions • For most characteristics ECE is relatively small • Less than 3 percentage points at aggregate level • Rarely more than 3 points for domains • For some characteristics, ECE may be important • Other definitions of ECE also considered

15. Hardcore Nonresponse Bias • Hardcore Nonrespondents = Units that will not respond under the current survey protocol no matter the number of call-backs • ECE does not include the bias due to hardcore nonrespondents • Total nonresponse bias = Bias due to cooperators who did not respond + bias due to hardcore nonrespondents • Adjusting for ECE may not remove bias due to HCNR

16. Call-back Models for Adjusting for ECE and HCNR Bias • General idea • Estimate the response propensity for subgroups of the population • Response propensity is modeled as a function level of effort (LOE) to obtain a response • Two models are considered • Manifest model (MM) – Ignores HCNR • Latent class model (LCM) –Includes HCNR • Includes a latent indicator variable to represent the HCNR’s in the population • Why latent?

17. Illustration for 5 Call-backs 1 = interview; 2 = noninterview; 3 = noncontact

18. Illustration for 5 Call-backs 1 = interview; 2 = noninterview; 3 = noncontact

19. Potential Advantages over Post-Stratification • Post-stratification adjustments (PSA’s) depend upon the availability of external benchmarks or auxiliary data • Selection of control variables is quite limited • Target populations also quite limited • Adjust for “ignorable” nonresponse only

20. Potential Advantages over Post-Stratification • Call-back model can rely only on internal variables • Weighting classes can be defined for any variables collected in the survey • Can be applied for any target population • Greater ability to selected variables that are highly correlated with response propensity • Adjust for “ignorable” and “nonignorable” nonresponse

21. Modeling Framework • Simple random sampling • Survey eligibility is known for all sample members • No right censoring • (i.e., all noncontacts received maximum LOE) Extensions to relax these assumptions are described in the paper

22. Incorporating the Model-based Weights Unadjustedestimator of the mean Based on the sample distribution Adjustedestimator of the mean Estimated from the model

23. Two Models for Estimating MM (Manifest Model) Assumes all nonrespondents would eventually respond at some LOE (i.e., all nonrespondents have a positive probability of response) LCM (Latent class model) Incorporates 0 probability of response for the hardcore nonrespondents (HNCR’s)

24. Technical Details

25. Notation Levels of effort (LOE) Outcome of LOE l where 1=interview, 2 = noninterview, 3=noncontact LOE associated with state S=1 or 2 Grouping variable (weighting class variable)

26. Notation Probability person in group g is interviewed at LOE l* Probability person in group g is noninterviewed at LOE l* Probability person in group g is never contacted Number of sample persons in group g interviewed at LOE l* Number of sample persons noninterviewed at LOE l* Number of sample persons never contacted after L (max LOE) attempts

27. General Idea –Outcome Patterns for 5 Call-backs Cooperator HCNR 11111 0 31111 0 33111 0 33311 0 33331 0 22222 32222 33222 33322 33332 33333

28. Likelihood for the Manifest Model • This model is appropriate when • Every sample member has a positive probability of responding at some LOE, or • Adjustment for ECE only is desired

29. Likelihood for the Latent Class Model Introduces a latent variable X where X = 1, if HCNR and X = 2, if otherwise Appropriate when some sample members have a 0 probability of responding and adjustment for total nonresponse (Later Cooperators + HCNR’s) is desired

30. Results

31. Four Estimators were Considered • Unadjusted estimator • Estimator using MM estimates of • Estimator using LCM estimates of • Estimator using CPS estimates of • i.e., usual PSA estimator • treated as the “gold standard”

32. Comparison of the ECE for a Maximum Five Callbacks Strategy Before and After MM Adjustment

33. Differences between PSA and Unadjusted and Adjusted Estimates for a Maximum Five Callbacks

34. Estimating the Potential Bias Reduction • BRFSS data do not exhibit very large nonresponse biases • Therefore, consider a variable, Y, that has maximum nonresponse bias given the BRFSS nonresponse rates • To do this, we form • Yg BRFSS response rate for group g • Compute the relative difference between unadjusted and adjusted estimates and the PSA estimate of the mean of Y

35. Absolute Relative Differences (|RDL|) for Unadjusted and Adjusted Estimators as a Function of Number of Call-backs

36. Conclusions • ECE for 5 call-backs is generally small, but can be moderately high for some characteristics • The Manifest Model can be employed to reduce ECE • The Latent Class Model can be employed to reduce total nonresponse bias (Later Cooperators + HCNR bias) • Future research should focus on • Variable selection • Comparisons of MSEs of the estimators • Small/medium size sample properties • Integration with other post-survey weight adjustments