Presentation to AERA Servaas van der Berg & Janeli Viljoen (University of Stellenbosch)

1. Investigating cognitive performance differentials by socio-economic status across international assessments – Towards a new methodology Presentation to AERA Servaas van der Berg & Janeli Viljoen (University of Stellenbosch) Email: svdb@sun.ac.za 6 April 2014

2. Reading performance in PISA is associated with per capita GDP for countries below GDP per capita of $20 000 PPP

3. … but not for richer countries (above $20 000 per capita) Source: OECD, Technical Committee for Pisa for Development, 2013

4. Background on SES and asset indices • Within countries, SES advantage is universally associated with more learning – “social gradients” (Willms2006; Ross & Zuze 2004) • Asset (wealth) indices are popular proxies for SES in analysing social gradients (in widespread use since Filmer & Pritchett 2001) • Convenient alternative to per capita income or consumption for ranking or classifying population into SES groups (deciles or quintiles) • Large literature on their appropriateness, construction, application • Assume wealth is latent trait underlying possession of assets • Weights given to each possession based on its contribution to information regarding the latent variable • Commonly first principal component in PCA or MCA

6. SA social gradient even much steeper than for SACMEQ • SA children from poor schools perform half a stdev (1 year of learning) below their SACMEQ counterparts • SA children from richer schools have a 5-6 year advantage over children from poor schools • SACMEQ mean SES = 0, SA mean SES = ±1

7. Asset indices: Accuracy within country vs. comparability across countries • Weakness is comparability across diverse communities • Does possession of a radio or bicycle, or even a donkey cart, mean the same in urban Johannesburg as in rural Malawi? • Accuracy within country vs. comparability across countries • If within country accuracy is the prime concern, indices should be constructed for each country separately, i.e. unique weights extracted for each country • If comparability across countries is the concern, the same weights need to be used in all countries, implicitly assuming that the same possession reveals the same information in every country about household wealth

8. Matching an asset index and consumption data – households containing 12 year olds from a South African dataset (NIDS 2008) (z-scores) Rank correlation: 0.601

9. What difference does it make? R2 from quadratic regressions using alternative weighting systems

10. How does it affect social gradients?

11. Our methodology Uses country-specific asset indices, but converts these into money-metric terms for comparability across countries and across international assessments This requires linking information on income distribution from household surveys to data in international assessments

12. Linking the household survey data to international assessment data • Use the international assessment data to construct an asset index for each country separately and rank all children in the sample accordingly • Use household survey data to rank children atttending school in the grade tested by the per capita income of their households, converted to PPP dollars • Assuming approximately similar rankings, allocate a per capita income value to each child in the education dataset, based on these rankings

13. Data for SACMEQ III • SACMEQ (Southern and Eastern African Consortium for Monitoring Educational Quality) tested Gr 6 children in Literacy and Numeracy in 2007 • Our focus is numeracy • Household survey data exist for 6 of 15 participating countries:

14. SES gradient for SACMEQ countries using common asset weights

15. SES gradient for SACMEQ countries using country-specific asset weights Note: Country-specific weights imply that all countries are ranged around mean SES of zero, thus ignoring country wealth differences

16. SES gradient for SACMEQ countries using money-metric SES values • Per capita income differences now fully considered • Thus, e.g., Kenya’s gradient shifts left, South Africa’s right • Now clear that Kenya considerably outperforms South Africa at every level of per capita income/consumption

17. Comparing across SACMEQ and SERCE • The same methodology can be applied to other international assessments, thus allowing comparisons of gradients across assessments • This is impossible using asset indices alone, unless the same assets are captured in both surveys • Comparing SACMEQ and SERCE illustrates value of the methodology • SERCE (Second Regional Comparative and Explanatory Study) undertaken in Grades 3 and 6 in 16 Latin American countries and Mexican state of Nuevo León in 2006 • Household surveys available for all 16 countries: Argentina, Brazil, Costa Rica, Chile, Dominican Republic, Ecuador, El Salvador, Guatemala, México, Nicaragua, Panama, Paragauy, Perúu, Uruguay • Equating of scores across surveys also needs separate attention –Gustafsson’s estimates are used here for illustration purposes

18. Comparing six weakest performing systems, given SES (i.e. lowest-lying gradients) Poverty line: $1.25 per person per day Poverty line: $2.00 per person per day

19. A money metric SES makes it possible to identify the absolutely poor (below $1.25 per person per day) across assessments and to analyse their performance by SACMEQ performance category

20. A money metric SES makes it possible to identify the absolutely poor (below $1.25 per person per day) and to analyse their performance by SACMEQ performance category

21. Conclusion • The value of international assessments is enhanced by controlling for SES when investigating relationships • But SES comparisons have drawbacks that are exaggerated in comparisons across countries where assets have very different values, or across assessments that did not capture the same assets • The methodology described here of converting country-specific asset indices into money-metric SES values holds promise to improve comparability, but needs good and comparable household surveys • It is also very data intensive • Equating of assessments and PPP conversions remain challenging

22. Thank you! Servaas van der Berg Research on Socio-Economic Policy (ReSEP) Stellenbosch University svdb@sun.ac.za http://resep.sun.ac.za/