The Choice Between Fixed and Random Effects Models: Some Considerations For Educational Research

1. The Choice Between Fixed and Random Effects Models: Some Considerations For Educational Research Clarke, Crawford, Steele and Vignoles and funding from ESRC ALSPAC Large Grant

2. Motivation • Need evidence from different disciplines to answer the research question : how can we improve pupil achievement? • Contribute to multi-disciplinary understanding by comparing common alternative models used by different disciplines

3. Introduction • Pupils clustered within schools → hierarchical models • Two popular choices: fixed and random effects • Choice of model: • Often driven by discipline tradition – economists use fixed effects for example • May depend on whether primary interest is pupil or school characteristics

4. Illustrations • What is the impact of SEN status on pupil achievement? • What is the impact of FSM status on pupil achievement?

5. Why adjust for school effects? • Want to estimate causal effect of SEN on pupil attainment no matter what school they attend • Need to adjust for school differences in SEN labelling • e.g. children with moderate difficulties more likely to be labelled SEN in a high achieving school than in a low achieving school (Keslair et al, 2008; Ofsted, 2004) • May also be differences due to unobserved factors • Hierarchical models can account for such differences • Fixed or random school effects?

6. Basic model • FE: us is school dummy variable coefficient • RE: us is school level residual • Additional assumption required: E [us|Xis] = 0 • That is, no correlation between unobserved school characteristics and observed pupil characteristics • Both: both models assume: E [eis|Xis] = 0 • That is, no correlation between unobserved pupil characteristics and observed pupil characteristics

7. Relationship between FE, RE and OLS FE: RE: Where:

8. How to choose between FE and RE • Very important to consider sources of bias: • Is RE assumption (i.e. E [us|Xis] = 0) likely to hold? • Other issues: • Number of clusters • Sample size within clusters • Rich vs. sparse covariates • Whether variation is within or between clusters • What is the real world consequence of choosing the wrong model?

9. SEN: Sources of selection • Probability of being SEN may depend on: • Observed school characteristics • e.g. ability distribution, FSM distribution • Unobserved school characteristics • e.g. values/motivation of SEN coordinator • Observed pupil characteristics • e.g. prior ability, FSM status • Unobserved pupil characteristics • e.g. education values and/or motivation of parents

10. Intuition I • If probability of being labelled SEN depends ONLY on observed school characteristics: • e.g. schools with high FSM/low achieving intake are more or less likely to label a child SEN • Random effects appropriate as RE assumption holds (i.e. unobserved school effects are not correlated with probability of being SEN)

11. Intuition 2 • If probability of being labelled SEN also depends on unobserved school characteristics: • e.g. SEN coordinator tries to label as many kids SEN as possible, because they attract additional resources • Random effects inappropriate as RE assumption fails (i.e. unobserved school effects are correlated with probability of being SEN) • FE accounts for these unobserved school characteristics, so is more appropriate • Identifies impact of SEN on attainment within schools rather than between schools

12. Intuition 3 • If probability of being labelled SEN depends on unobserved pupil/parent characteristics: • e.g. some parents may push harder for the label and accompanying additional resources; • alternatively, some parents may not countenance the idea of their kid being labelled SEN • Neither FE nor RE will address the endogeneity problem: • Need to resort to other methods, e.g. IV

13. Data • Avon Longitudinal Study of Parents and Children (ALSPAC) • Children born in Avon between April 1991 and December 1992 • Rich data • Family background (including education, income, etc) • Medical and genetic information • Clinic testing of cognitive and non-cognitive skills • Linked to National Pupil Database

14. SEN • One in four pupils in England have SEN age 10 • Just under 4% have statement • In 2003-04, the period relevant to our data, approximately £1.3billion spent on primary school SEN (excluding special schools) • £1,600 per pupil with SEN

15. SEN • Substantial variation in %SEN across schools • Quarter of schools have fewer than 15% SEN • Quarter with more than 24% SEN • Key question is whether the factors driving differences in % SEN between schools are correlated with unmeasured school-level influences on academic progress

16. Estimated effects of SEN status on progress between KS1 and KS2

17. Results from this analysis • SEN negatively correlated with progress between KS1 and KS2 • Choice of model does not seem to matter here • OLS, FE and RE give qualitatively similar results • Correlation between being SEN and unobserved school characteristics not important • Regression and random effects assumptions are likely to hold in this example - prefer the random effects model

18. Conclusions • Often fixed effects approach is used because RE assumption is a strong one • Efficiency advantages to the RE approach • Failure of the regression assumption is major issue • Approach each problem with agnostic view on model/ may not make a difference • Should be determined by theory and data, not tradition