How survey design affects analysis

1. How survey design affects analysis Susan Purdon Head of Survey Methods Unit National Centre for Social Research

2. Two general principles • Weighting affects both survey estimates and standard errors • Complex sample design (clustering, stratification) affects standard errors but not survey estimates (as long as survey isn’t weighted).

3. Why weights are used Two main reasons: • Because sample units are selected with non-equal probabilities of selection • To reduce non-response bias

4. Weighting for non-equal probabilities Non-equal probabilities of selection can be used: • For good statistical reasons: stratification with disproportionate allocation (e.g over-sampling by region) • For convenience: because sampling frame is of larger units than survey units, and need to sub-sample within units (e.g selection of one person per household). • Weights are calculated as the inverse of the probability of selection.

5. Example 1: WERS98

6. Effect of selecting one adult per household

7. Weighting for non-response • Many surveys use weights to reduce non-response bias. • Up-weight low responding groups; down-weight high responding groups.

8. Weighting for non-response

9. Effect of weights on estimates • Weighting changes almost all survey estimates (means, percentages, odds ratios, correlation coefficients, regression coefficients etc.)

10. Unweighted estimate= (25%x38 + 17%x102+15%x27+9%x8) /(38+102+27+8)=20% Weighted estimate= (25%x38x1 + 17%x102x2+15%x27x3+9%x8x4) /(38x1+102x2+27x3+8x4)=18% Effect of weights on estimates

11. Effect of weighting on standard errors • Standard errors for weighted and unweighted estimates are not the same. • Weighting because of non-equal probabilities of selection tends to increase standard errors. • Weighting for non-response sometimes increases/sometimes decreases ses. Impact tends to be smaller.

12. Impact of other design features - clustering • Most face-to-face surveys are clustered • Clustering doesn’t change estimates, but it does increase standard errors • Degree of increase depends on cluster size and cluster homogeneity • To account for clustering need to identify the primary sampling unit (psu) on dataset.

13. Impact of other design features - proportionate stratification • Most surveys use proportionate stratification (either overall or within regions) • Does not affect estimates. Tends to reduce standard errors. Degree depends on choice of stratifiers.

14. In summary: • To get unbiased estimates need to use survey weights. • To get correct standard errors need to take into account survey design, in particular weighting, clustering and stratification.