The Use of Test Scores in Secondary Analysis

1. The Use of Survey Weights in Regression Analysis(Wooldridge) Discussion The Use of Test Scores in Secondary Analysis PIAAC Methodological Seminar, June 2019, Paris Dr. Sabine Meinck

2. “Primary” vs “Secondary” Analysis? Concept note: • “… primary analysis is mainly about … efficient estimation of the main parameters … (the cognitive skills) in the population of interest. Secondary analysis … is mainly focused on the … parameters of a statistical model that aims at uncovering the causal relationship between different variables.” • Is there really a significant difference between “primary” and “secondary” analysis?

3. “Primary” vs “Secondary” Analysis? • I don’t fully agree on the arguments made in the concept note. • Reason to mostly utilize simple statistical indicators for “primary” reports is the wealth of the data • “Primary” analysis is still more broad than just measuring some domains. • Most of the papers with “secondary analysis” I read/reviewed, actually aim for making population inferences, even when using more advanced analysis types

4. “Primary” vs “Secondary” Analysis? • In which circumstance would we actually NOT be interested in inferring on the population? I actually have a hard time in thinking of any. • Hence, this difference alone clearly is no argument for altering recommendations on using weights.

5. Different Weighting Schemes Needed? • Perhaps. Perhaps not. • Using sampling weights for design-unbiased estimation of simple statistical indicators is out of question. • Weighting in simple regression analysis? • Weighting for propensity score modelling? • Weighting in MLM? • Weighting in SEM? • More evidence is needed to develop efficient weighting schemes for advanced analysis methods. • Technical doc’s/User Guides should cover related topics in more comprehensive ways.

6. Treatment effect estimation? • Randomized treatments can hardly ever be observed in LSA. • LSA are never RCT’s. • If external variables are imposed as “treatments” (e.g., some specific reform in an education sub-system), bias may arise from not having considered this variable in the conditioning model used for scaling. • Related discussion really relevant?

7. Estimating Causal Effects of LSA? • There is a large body of literature with a very critical view on the possibilities of uncovering causal effects with LSA data. • E.g., special issue of Large-scale Assessments in Education on “Quasi-causal methods” (May 2016). • Need to encourage further discussion among economists and educational researchers?

8. Weighting Reduces Precision of Estimates? • Using sampling weights can have an effect on sampling errors. • It can go in both directions (increasing or decreasing sampling errors), BUT it reflects the true sampling variance resulting from unequal sample allocation.

9. Weighting Reduces Precision of Estimates? • Think of a country in which all public schools follow the same curriculum • Each private school follows a different curriculum • If private schools are over-sampled, we overestimate the variety in the school system and also sampling variance. • Unless we use weights.

10. Weighting Reduces Precision of Estimates? • Neglecting weights will lead to biased sampling error estimates. • In other words: WLS may produce higher S.E.’s than OLS, but WLS produces unbiased S.E. estimates.

11. Trusting Sampling Weights? “To ensure consistency use the sampling weights – if they can be trusted.” • Generally, all procedures to derive sampling weights are methodologically sound and well established. • No doubt that design weights are correctly reflected in the final weights! • Sampling frames highly reliable; selection probabilities tracked for each sampling stage

12. Trusting Sampling Weights? • Critical issue: weight adjustments • Assumption: non-informative response model • It can be very well argued that this assumption is often violated • There is even evidence that nonresponse is not occurring at random • Why don’t we do something about it? • Information on the mechanisms of nonresponse is often not available (not at all or not in time) • Even if a thorough NRBA is possible, this is still no proof of unbiasedness • This is why LSA require such high participation rates

13. Summary • Enforce statements about using weights (e.g., Carstensand Hastedt, 2010; Braun and von Davier, 2017; Von Davier, Gonzalez and Mislevy, 2009, and now Wooldridge): • All recommend using sampling weights for most if not all statistical analysis. • More research is needed. 

14. Thank you! Sabine Meinck sabine.meinck@iea-hamburg.de