Approaches to evidence based policy making in education Professor Geeta Kingdon

1. Approaches to evidence based policy making in education Professor GeetaKingdon

2. Congratulations CBSE • CAER commitment to evidence based decision-making • Centre will help to improve learning by promoting reliable and valid assessments • also help to create capacity do robust evaluations • Thus bring assessment-led reform in Indian education • Splendid example of a PPP

3. Talk is on evidence based policy, focus on macro • Current landscape • Pitiably poor learning levels (PISA, EI, NAS, ASER, NFHS, SchoolTELLS; NAS showed that <50% of class V kids could answer “how much greater is 555 than 198”? ) • Measuring learning can be disheartening, when much effort & large resources expended (SSA reforms, RTE, NCF, 6 Pay Comm, TET, NCFTE, NCSL) & still outcomes are poor; denial • Encouraging that Planning Comm’s 12th FYP (GOI, 2013, p49) “will place the greatest emphasis on improving learning outcomes at all levels”

5. Steps for evidence based decision-making • Obtain the evidence (e.g. measure learning levels) • Evidence has to be credible (e.g. measure learning with validity and reliability, with proper sampling) • Embrace & analyse results to reach diagnoses • Formulating policies to take remedial action • Pilot test new policies, to see their impacts on learning • Scale up policies that have strong impact at lowest cost

6. Evidence from learning assessments – can empower parents • An imp potential use of evidence from A – is parental info • Has potential to increase school accountability • World university rankings, but not school rankings • Fear they may reflect socio-econ background but there are ways of reducing that • In India a debate needed

7. Evidence from assessment of teachers • One potential reason for poor learning - teachers themselves lack competence • Teachers rarely tested in large scale way • Evidence from T tests in India showed poor competence • Led to the decision to bring in the TET

8. SchoolTELLS survey Assessment tasks for teachers aligned with standard teaching tasks that teachers in primary school would be required to do in the classroom routinely. Language tool helped to understand teacher’s ability in the following: Do you know: e.g. meaning of difficult words in a grade 4 level text Can you explain : e.g. explain difficult words in simple language or summarize a Std 4 story text effectively Can you spot common mistakes: e.g spelling and grammar mistakes Maths tool also helped to understand the teacher’s ability in the following : Do you know: e.g. solve problems Std 4 or 5 level Can you explain: e.g. explain problem solving in simple steps Can you spot and analyze common mistakes : e.g in arithmetic operations The teacher tests were graded by SCERT Bihar staff

9. TASK 1: VOCABULARY RELATED TASKS From Std 3 onwards, the vocabulary in language textbooks becomes difficult. So, teachers need to be able to explain difficult words in simple language. • Grading done on 3 criteria: • Was the word meaning “meaningful” ? • Was language used “easy to understand” ? • Were there any spelling mistakes ?

10. TASK 2 : SUMMARIZING TEXTS

11. TASK 3: SPOTTING SPELLING & GRAMMATICAL MISTAKES

12. Deficits in teachers literacy skills • 43% of word meanings correct; 57% wrong • 45% of summaries were meaningful; 55% wrong • 40% teachers did not have spelling mistakes (in a 2-sentence write up) • 35% had 1-2 mistakes; 25% had >=3 mistakes • Only 50% of teachers could spot >3 mistakes in a write up in which we had deliberately introduced 6 mistakes.

13. TASK 6 : SPOTTING & UNDERSTANDING COMMON MISTAKES IN ARITHMETIC Samples of children’s work shown. Teachers asked to choose…

14. Deficits in teachers’ numeracy skills • 78% could spot correct one when presented a sample of three simple division sums • 24.5% could do a percentage sum; 75% not • 27.9% could do an area sum; 72% not • About 20% said they never had problems in addressing the maths queries of their pupils

15. How this evidence and TET evidence helps • School-TELLS (2008) & ‘Inside Classrooms’ study (2011) highlighted deficits in T knowledge & ability to teach • This evidence contributed to decision to introduce the TET • TET objective to vet applicants, ensure competence • Abysmal pass rate - 0.4 to 3.7% pass rate

16. Evidence in this T assessment is extremely valuable • The evidence in this T assessment is extremely valuable It helps to identify the training needs of teachers. • It can inform policy makers who decide training curricula • But have states used this evidence in this way? States need the desire to analyse; make use of this evidence; a transparent approach • Congratulate the CAER for analysing C-TET – this is evidence based policy making (the policy maker CBSE sought evidence – gave data)

17. Is this evidence relevant only for govt and rural private schools? • Is this problem of low cognitive skills of Ts confined to government primary teachers ? • Clearly more generic problem – • Even highly paid govt T have major deficits in skills • TET evidence • Private schools cannot be complacent on this • Testing T can help to assess the training needs of each T

18. The importance of evidence • There are many initiatives to improve education • NFE (1982) OBB (1986) • TLC (1988) MDM (1982) • SK (1987) LJ (1988) • DPEP (1993) SSA (2003) • Aadhar, ABL, MLE, Nallikali, NaiDisha, Read India, RTE • Have these programs had impact? – little evidence to judge Efforts to improve more successful, when based on evidence

19. Culture of seeking evidence • Why base decisions on evidence? • While poor quality schooling does not threaten lives, it seriously affects people’s quality of life, and even longevity. • In medicine, its unthinkable without thorough testing by experts, & the use of most robust, expensive randomised control trials; But in Education, Ministers freely make policies without consulting evidence / experts • In good educ systems, and ideally, policy will be not made on supposition, ideology or political expediency. Govt seeksevidence

20. UK policy makers’ use of evidence fromimpact evaluations • Early intervention - quality & effectiveness of pre-school experience in securing better long term outcomes was used to justify more investment (Sammons et al, 2006) • Class size - evidence of no stat difference was used to justify not extending the policy into later years (Blatchford et al, 2002) • Formative assessment - effect sizes in attainment (alongside pupil & teacher perspectives) led to inclusion in national policy (Wiliam et al, 2004) • Education Maintenance Allowance - evidence from pilot study of post 16 retention led to national roll out, though long-term sustainability was worse in pilot areas (Middleton et al, 2005)

21. Policies could be better, if evidence based • The narrative in the 6th Pay Comm for across-the-board doubling teacher salaries (without increased accountability) was : it will motivate teachers. Was this effective? Did it raise teacher effort? No one checked. [next slide] • A state govt recently announced it would regularise 176,000 para teachers, in the name of quality of education; it did not look at the relative effectiveness of regular and para teachers (3 papers) • Under RTE it has been made mandatory for teachers to have B.Ed. Certification, reduced PTR to 30, and many inputs mandatory; where is the evidence for this? no pilot testing • If there were garnering of evidence on the impact of policies, then ineffective policies could be weeded out

22. SchoolTELLS survey (2008) Higher resources, lower effort Structure of accountability matters more than resources

23. What kind of evidence? • What kind of evidence is useful / acceptable? • Distinguishing correlation and causation • The importance of methodology • Using the force of the federal chequebook to nudge researchers to use robust methods capable of yielding causal inferences.

24. Interest in evidence of impact • Field of medicine long interested in evidence of impact • Recent upsurge of interest in impact evaluation in many fields • nutrition, labour, governance, rural development, education, poverty • ‘International Initiative for Impact Evaluation’ (3ie) established 2008 • Network of Networks on Impact Evaluation (NONIE), comprised of: • the OECD/DAC Evaluation Network (DACEN), • the UN Evaluation Group (UNEG), • the Evaluation Cooperation Group (ECG), and • the International Organization for Cooperation in Evaluation (IOCE) • International Development Evaluations Association (IDEAS) • Development IMpact Evaluation (DIME) • Spanish Impact Evaluation Fund (SIEF)

25. What is impact evaluation? • Measuring outcomes (eg learning) is difficult • But measuring outcomes is NOT impact evaluation • measurement of net effects of a program on the outcomes of interest (e.g. on learning) • In IE, it is important to appreciate the difference between correlation and causation

26. Correlation vs Causation • As ice cream sales increase, number of drowning deaths increases sharply • Therefore, ice cream causes drowning • Sleeping with one's shoes on is strongly correlated with waking up with a headache • Therefore, sleeping with one's shoes on causes headache. • Taller people have higher earnings • Therefore, higher height causes higher earnings

27. Salary and Height

28. Salary and Height?

29. Spurious Relationships • This relationship is said to be “spurious” • When we did the bivariate relationship, we said, • In reality, things look like this: Think of Sarva Shiksha Abhiyan and its impact Height Salary Height Salary Gender

30. Correlation vs Causation • In achievement equation with class-size, the coeff on class-size var is -0.46, i.e. an increase in class-size by 10 is associated with a 4.6 point lower ach mark • Can we say from this that: • class-size is negatively related to achievement? • reducing class-size will lead student achievement to rise? • what is the diff between these? • what else determines ach? • where is that included in the way the reg. equation is written

31. Quantitative impact evaluation methods • OLS production function (not causal relation) • Instrumental Variables • Panel data • Randomised experiment • Quasi-experimental • Propensity score matching method • Each method has strengths & drawbacks • RCT, PSM, Panel, IV

32. Evidence on impact of policy interventions • Performance related pay • Duflo and Hanna, RCT • Muralidharan/Sundararaman, RCT • Contract teachers • Atherton & Kingdon, panel data • Muralidharan et al, RCT • Goyal & Pandey, OLS, school FE • DPEP • Jalan and Glinskaya, PSM • Schmid, IV • Private schools • Desai et al, panel • French/Kingdon, panel • Mid Day Meal • Afridi, 2010, panel • Union membership • Kingdon Teal, panel

33. Greater emphasis on evidence • NCERT evaluated impact of programs in 4 states, under TCF • Data becoming available – EI, ASER, DISE, SEMIS, NAS • What’s imp is the quality of the data / studies, i.e. robustness of methods/ designs; degree to which they tease out causal effect • Long term investment needed in capacity dev • MHRD / JRM approval for estb of National Assessment & Evaluation Centre

34. Some important considerations • Even evidence showing reliably what works is insufficient • Policy makers need to: • seek, read such evidence • discern good/bad evidence • act on evidence, i.e. make evidence-based policy • Freedom from political interference • Independence from funding body, hence PPP better

35. Good evidence does not always permit good decisions • Even when we have good evidence a policy has capacity to substantially improve outcomes, there can be powerful political economy barriers to the implementation of policies. • E.g. Duflo and Hanna (Rajasthan) say : “Although this study suggests that a system of automatic monitoring with enforcement by physically remote agents who are prepared to enforce the rules is technically feasible and indeed provides better incentives for teachers, a later effort to introduce this system with higher-skilled, higher-status, and more politically powerful health-care workers ran into strong political obstacles (Banerjee et al. 2007b).” Another e.g. para T in UP • We need to understand the political economy constraints, and how can they be eased

36. Thanks

37. VOCABULARY TASKS : DO TEACHERS KNOW WORD MEANINGS ? Four difficult words are given. Please write their meaning using simple words

38. SUMMARIZING TEXTS : EXAMPLES FROM TEACHERS

39. SUMMARIZING TEXTS : CAN TEACHERS SUMMARIZE ?

40. SUMMARIZING TEXTS : CAN TEACHER SUMMARIZE USING SIMPLE LANGUAGE Of those writing meaningful summary

41. TASK 4 : PERCENTAGE PROBLEM : Findings

42. TASK 5 : ARITHMETIC : AREA PROBLEM

43. 80% primary school teachers have difficulties in teaching maths % teachers who agree with the statement “Sometime I have difficulties in addressing mathematical queries and problems of my students” Only about 20% of govt. school teachers believe they don’t face problems. About 80% admit to have difficulties sometimes. This suggests possible interest in in-service training to upgrade maths skills