VALUE ADDED MODELS AND METHODS Differences between KS2-KS4 CVA and FFT SX DAVE THOMSON AND TREVOR KNIGHT

1. VALUE ADDED MODELS AND METHODSDifferences between KS2-KS4 CVA and FFT SX DAVE THOMSON AND TREVOR KNIGHT

2. Objectives of this session • Statistical philosophy and model building • The importance of ‘analysis of variance’ • The relative importance of explanatory factors • How DfES and FFT models differs • Statistical inference • Use and abuse of statistics from models • Removing the statistical night-terrors

3. CVA versus FFT SX • Differences at school level • 21% of schools have a different significance state for KS2-KS4 CVA compared to equivalent SX model • But just 4% (122 schools) have a significantly different result • Differences in variables used • Differences in statistical methodology • Multi-level modelling (CVA) and modified OLS (SX) • Differences in purpose • CVA a measure of whole school performance • FFT designed for school improvement; slicing and dicing by pupil groups • Bias in CVA to certain pupil groups • Developing criteria for establishing how good a model is • Statistical robustness • Fairness to all pupil groups and institution types • Interpretability

4. CVA versus FFT SX • Differences at school level (KS2 to KS4)

5. CVA versus FFT SX • Differences at school level (KS2 to KS4)

6. The importance of variance • Pupils’ KS4 capped points scores vary • Range from 0 to 464 • Mean in 2004 was 285.8 • Variance • Measure of how much pupils’ KS4 scores differ from the mean • Variance in 2004 was 11291 • Value added is about explaining variance in terms of prior attainment, pupil contexts and school contexts • If 100% of variance could be explained, every pupil and school in the country would have a value added score of 0 • In reality, we can account for about 60% based on KS2, pupil contexts and school contexts • The remaining variance is attributed to the differential effectiveness of schools

7. Accounting for variance in pupils’ KS4 scores Target: 11291 Base:0

8. Building the CVA model

9. Building the CVA model

10. Accounting for variance in pupils’ KS4 scores Target: 11291 PA only: 5351 Base:0

11. Building the CVA model • A simple model • The line has an equation Predicted KS4 capped points = Intercept + Co-efficient* KS2 APSIntercept = 60Co-efficient = 3.5Pupil Value Added = Actual - Predicted • 47% of variation in pupils’ capped KS4 points scores can be explained by this equation, i.e. 47% of 11291 = 5351 • Improving the model • Adding additional factors, each with their own co-efficient • This changes the co-efficients of factors already in the model and the intercept • Altering the shape of the line

12. Adding a quadratic line

13. Building the CVA model • A more elaborate model • Making the line curved and adding gender as a factor KS4 capped points = Intercept + Co-efficient1* KS2 APS+ Co-efficient2 * KS2 APS squared + Co-efficient3* femaleIntercept = -59Co-efficient1 = 6.6Co-efficient2 = 0.2Co-efficient3 = 23.8 • 48% of variation in pupils’ capped KS4 points scores can be explained by this equation

14. CVA Factors- www.standards.dfes.gov.uk/performance

15. Factors

16. Building the CVA model • Evaluating the choice of factors • Do they make sense educationally? • How much variation do they explain? • Are they significant? • What are their effect sizes? • Effect Sizes • Measure of the relative importance of a co-efficient in a statistical model • Effect sizes greater than 0.6 are considered large • Effect sizes smaller than 0.2 are considered small

17. CVA- the important effects

18. CVA- the less important effects

19. Multi-level modelling: Calculation of school lines

20. Variance between schools and between pupils • Multi-level modelling • To calculate amount of variance in KS4 points between schools (differences between school lines) • To calculate amount of variance in KS4 points within schools (scatter of pupils around school lines) • Responsible for the majority of difference between SX and CVA • Shrinkage Factor • Is a function of variance between schools, variance within schools and number of pupils in the cohort at a school • Varies from school to school depending on size of cohort • Is applied to the mean KS4 value added score for all pupils at a school to produce the school’s CVA score and confidence intervals

21. Variance in the CVA model • Calculating the shrinkage factors, confidence intervals and significance

22. Variance in the CVA model • Calculating the shrinkage factors, confidence intervals and significance

23. Calculating the shrinkage factor • Shrinkage factor • (342/ (342 + (4491/ number of pupils in cohort))). • School CVA score • SF* mean pupil value added score • Mean pupil CVA score at the school = -9.17 • 195 pupils, SF= 0.94 • School CVA score = 1000 + (0.94*-9.17) = 991.4 • If 50 pupils, school CVA score = 1000 + (0.79*-9.17)= 992.8

24. Calculating school significance • Confidence interval (95%) • 1.96* square root of (between school variance* within school variance) divided by (number of pupils * between school variance* within school variance) • 1.96 * sqrt ((341.8737 * 4490.57)/ (number of pupils*341.8737 + 4490.57)) • For 195 pupils = 9.1 • CVA score was -8.62 • Lower confidence limit = -17.72 • Upper confidence limit = 0.49 • Significance • If lower confidence limit >0 SIG+ • If upper confidence limit <0 SIG- • So the example school not significant (just!)- but would have been sig- without the shrinkage factor

25. Accounting for variance in pupils’ KS4 scores Target: 11291 CVA: 6458 PA only: 5351 Base:0

26. FFT SX model • SX models • Pupils divided into 96 bands and the mean capped KS4 points score calculated for each band • This is then used as the main explanatory factor • Some small adjustments are made to the line • School averages • Separate lines for schools not calculated, i.e. no shrinkage • School score is the simple average (mean) of pupils’ value added scores

27. PA lines in SX and CVA

28. FFT SX – the important effects

29. FFT SX – the less important effects (1)

30. FFT SX – the less important effects (2)

31. CVA and SX comparison of factors • Common to both models • Prior attainment, SEN School Action Plus, “Joined late” have similar effect sizes and are >0.6 in both models • EAL, School Action, Bangladeshi, Chinese, Black African, Gypsy/ Roma, Irish heritage Traveller have similar effect sizes and are >0.2 • Differences between models • Pupil FSM and school GDF rank relatively important in FFT • Interactions of pupil prior attainment with school FSM and school mean intake score also important • In care, Indian, Pakistani, Asian Other, Any other, Pupil IDACI relatively important in CVA

32. CVA and SX comparison • Example • Girl, August born, White British, not FSM, not EAL, not SEN, not mobile • IDACI score of postcode 22% • School FSM rank- 74; GDF (ACORN) rank 67

33. Impact of variables on KS4 points score estimates

34. Impact of variables on KS4 points score estimates

35. Impact of variables on KS4 points score estimates FFT PA only estimate 364.9

36. Impact of variables on KS4 points score estimates FFT PA only estimate 364.9 As produced by PAT

37. Issues with CVA • What are the issues? • Both “explain” roughly similar amounts of variation in KS4 points scores- 60% (SX) 57% (CVA) • CVA used for purposes for which it was not designed- i.e. pupil groups • Schools with ‘better’ CVA residuals tend to have: • Higher than average proportions of ethnic minority pupils • Higher than average prior-attainment • Beware ‘large’ differences in percentile ranks • Particularly in the 20th to 80th percentile range

38. Issues with CVA

39. PANDA- pupil groups

40. PANDA- pupil groups

41. PANDA- pupil groups

42. PANDA- pupil groups

43. PANDA- pupil groups

44. PANDA- pupil groups

45. Comparison between School CVA and FFT SX

46. Pupil residuals by ethnicity

47. Pupil residuals by SEN stage

48. Improving CVA and SX • Measurement error • SEN • IDACI at pupil level • Test marks • PLASC data (e.g. date of joining) • How can models be improved? • Use of interaction terms (e.g. ethnicity and FSM) • Use of random slopes for school lines • Use of ordinal models for individual subjects • Including CVA in FFT reports • Include CVA as well as or instead of SX? • Report showing schools with significantly different scores? • Report showing pupils with significantly different scores? • SX Ready Reckoner?

49. Improving CVA and SX • Development of criteria to establish model fitness • Clear statistical principles for potential improved models • Discussed implications of ‘educational’ significant residuals • Logs of FFT customers views for possible model enhancements • Basis for putting out possible improved models to FFT customers • Management of model change, rules and timescales/timetables

50. General conclusions on CVA and SX comparisons • CVA and SX VA differences at school-level are important for national accountability, but few schools are really judged different overall • CVA is a simple multi-level model – more complexity would help predictions (random slopes in prior attainment), and • CVA lacks ‘sophistication’ - no interactions (which might not be so important if slopes varied), and though • SX has more ‘sophistication’ but some have little impact and some are more difficult to ‘interpret’ • CVA has ‘issoos’ when sliced by pupil type for PANDAs and PAT, which FFT seems to avoid ….. but • Neither model MUST be used uncritically • FFT SX model has some desirable features for progress reflection which expanded information from CVA could provide • BOTH systems would gain from a coherent plan for model testing and discussion with customers – that is, with schools and LAs • Variety is the spice of life – provided your life doesn’t depend on the variety