PISA maths – England versus East Asia

1. The mathematics skills of school children: How does England compare to the high performing East Asian jurisdictions?

2. By age 15, children in East Asian counties are approximately 1.5 years of schooling ahead of children in England PISA maths – England versus East Asia

3. These regions and nations – from Alberta to Singapore, Finland to Hong Kong, Harlem to South Korea - should be our inspiration’ (Secretary of State for Education in England, Michael Gove) Huge policy interest in this…. • ‘we must learn from high-performing nations like Japan’ (ShadowSecretary of State for Education in England, Stephen Twigg) • ‘We have to see this as a wake-up call….. We can quibble, or we can face the brutal truth that we’re being out-educated’ (US Secretary of Education, Arne Duncan)

4. WHY!?....we don’t really know

5. Possible explanations • Biology / genetics • Cultural differences - Commitment and highly value education (‘Tiger mums’) • ‘Failing’ secondary schools • Teaching methods • Design of the curriculum • Many more!!

6. Possible explanations • Biology / genetics - Can’t do anything about • Cultural differences – Hard to change in short-run - Commitment and highly value education (‘Tiger mums’) • ‘Failing’ secondary schools • Teaching methods Things we can change • Design of the curriculum • Many more!!

7. ...but we do not know the relative importance of each of these factorsPolicy – hard to know what we need to do to catch up!

8. Current research Lots of research looking at the ‘high-performing’ systems (particularly East Asia) • E.g. Review of the curriculum https://www.education.gov.uk/publications/standard/publicationDetail/Page1/DFE-RR178 • E.g. Teaching observations in Shanghi Pretty much all qualitative This paper: Think what quantitative research can add to the debate

9. But there is a MAJOR problem for both qualitative and quantitative researchEast Asian countries have a lot of East Asian children in them!….Big confounding factor

10. Implications of this confounding Qualitative research • We observe differences in curriculum • We observe differences in teaching methods • BUT we don’t know if this has causalimpact • Really the reason why East Asian countries doing better? Quantitative research • Can’t easily ‘control away’ cross-country confounders • Therefore also difficult to establish causal impacts • PISA can tell us how far we are behind…… • …..But it can’t tell us why!

11. My paper….. What else can be done to inform the debate???? Starting point • How do we judge school performance? • Widely recognised it should not be on % 5 A*- C grades • Should be judged on ‘value-added’ (how much children improve) International application • Should think same way about countries / school systems • Don’t judge UK secondary schools by PISA test scores • Judge them by value-added • How much do UK children improve in secondary school (relative to East Asian children)

12. We try to start thinking about this….. • How do average test scores change between age 10, 14 and 16 in England compared to East Asia? • How does the distribution of test scores change between 10, 14 and 16 in England and East Asia? • How does the socio-economic gap in children’s test scores change over age in England compared to East Asia?

13. Data

14. Ideal data → Longitudinal → Follow same kids over time → Test scores designed to be comparable over time and across countries → E.g. A longitudinal PISA This type of very rich data does not exist Therefore have to think about an alternative

15. Data we do have…. → TIMSS 2003 4th grade (age 9/10) → TIMSS 2007 8th grade (age 13/14) → PISA 2009 (age 15 year 3 months – 16 years 2 months) • Think of this as ‘repeated cross-sectional data’ • Investigate change over time at the population level under certain assumptions: • Samples drawn from same population at each time point • The same skill is being measured at each time point Argument: Do these assumptions hold for TIMSS / PISA data? Argue they provide an OK approximation….

16. An aside – how similar are PISA / TIMSS? Often argued that PISA / TIMSS measure ‘different skills’ • PISA = ‘Functional ability’ • TIMSS = ‘Curriculum’ based skills Qualitative research: Analysis of questions suggests there are differences.. …but unable to say of this has major impact Other statistical differences Item-response model used Target populations etc But really how different are PISA and TIMSS??

17. PISA 2009 vs TIMSS 2011 (maths) Strong correlation between x-national results (r = 0.88) Actually tell us pretty similar things about performance

18. PISA 2003 vs Key stage 3 test scores (England) Correlation PISA maths with Key Stage 3 average scores = 0.83 Correlation PISA maths with Key Stage 3 math scores = 0.85

19. Previous graphs suggest that key assumptions of the method used may approximately hold

20. International z-scores Nevertheless …….. • PISA and TIMSS raw test scores are not directly comparable – based on a different array of countries. • Convert into international z-scores. • Each country’s mean test score (for each wave of the survey) is adjusted by subtracting the mean score achieved amongst all childrenin the countries for that particular survey and dividing by the standard deviation • Estimates refer to English pupils’ test performance relative to other countries

21. Countries included in this study • Include all countries that took part in TIMSS 4th grade (2003), TIMSS 8th grade (2007) and PISA 2009 • Leaves 13 countries: • England (Focus) • Asian Tigers (Hong Kong, Japan, Singapore, Taiwan) • Other developed (e.g. Italy, Norway, US)

22. Measures ‘Average’ = Mean z-score ‘Inequality’ = Standard deviation of test scores ‘Low achievers’ = 10th percentile of test distribution ‘High achievers’ = 90th percentile of test distribution ‘Low SES’ = 0 – 25 books in the home ‘High SES’ = More than 200 books in home Measuring construct of interest? Measurement error?

23. Methods ‘Average’ = OLS regression (allowing for clustering). Two-sample t-test for statistical significance ‘Low achievers’ = Quantile regression at 10th percentile. Bootstrapped standard errors to allow for clustering. ‘High achievers’ = Quantile regression at 90th percentile. Bootstrapped standard errors to allow for clustering. SES differences in achievement = OLS regression. Maths test scores as response. Books in home as key explanatory variable. Controls for gender and immigrant status.

24. Results

25. …. we are just as far behind at age 9/10 as at 15/16 (on average) Gap between England and East Asia does not widen between 9/10 and 15/16 (in terms of our relative position)

26. ….(some evidence) low achievers catch-up (age 10 – age 16) Low achievers in England may improve relative to low achievers in East Asian countries between primary and secondary school

27. …. But the high achievers may lose some ground High achievers in England may fall backrelative to those in East Asian countries between primary and secondary school

28. SES gap in math test scores Big gap in England…. …seems to increase between 10 and 16 ….but this holds true in all countries Caution needed Difficulties with measurement of SES (and with books in the home)

29. Conclusions

30. Suggests (but does not conclusively prove) that.... • ‘Poor’ PISA ranking not about failing secondary schools... - Just as far behind East Asian countries before children each secondary school (on average) • .......though still possible some changes are needed - E.g. Curriculum / incentives to stretch the highest achievers more? • Focus should perhaps be earlier years??? - Primary school? Early childhood? • .... but still can’t rule out ‘culture’ as the dominant force? - Likely to take hold early in life. - Hence consistent with the large cross-national differences at age 10.