Impact Evaluations and Social Innovation in Europe

1. Impact Evaluations and Social Innovation in Europe Sofia, 24 January, 2012 Joost de Laat (PhD) Human Development Economics Europe and Central Asia The World Bank Comments: jdelaat@worldbank.org

2. 15 December 2011 Deadline PROGRESS

3. Inputs Activities Outputs Results Framework for an Early Childhood Education Program Impacts on Outcomes • Finance • State budget • European Social Fund • Human resources • Min. of education – social inclusion unit • Slovak Education NGO • Office of the Plenipotentiary • Municipal Authorities • Preschool staff Project preparation activities (4 months) Identify 20 communities Hire 20 mediators and provide early childhood education training Provide monitoring training to mediators Design monitoring database Poject implementation activities (1 year). Mediators: Identify the vulnerable families Provide information to these families on early childhood education parenting techniques Assist parents enroll children in nearest preschool Provide material needs to poorest families Organize weekly reading clubs for Roma mothers Record activities and outputs in database Project preparation outputs 20 communities selected 20 Roma mediators trained on early childhood education and monitoring Monitoring database in place Project implementation Est. 600 vulnerable families identified Est. 600 vulnerable families received information on ECD Est. 400 children assisted with enrolment into preschool Est. 200 parents received material needs for their young children Est. 300 mothers participate in reading clubs Database with information on 600 families Improved knowledge on parenting skills Increased preschool enrolment of students from vulnerable families Improved socio-emotional skills of young vulnerable children Improved cognitive skills of young vulnerable children Improved health outcomes of young vulnerable children Lower enrolment into special primary schools among vulnerable children Improved primary and secondary school performance Greater long run employment outcomes and reduced poverty

4. Outline What? Impact Evaluations ? Who? How? Ethics? Why?

5. Isolates causal impact on beneficiary outcomes • Globally hundreds of randomized impact evaluations • Canadian self-sufficiency welfare program • Danish employment programs • Turkish employment program • India remedial education program • Kenya school deworming program • Mexican conditional cash transfer program (PROGRESA) • United States youth development programs • Different from: • e.g. evaluation measuring whether social assistance is reaching the poorest households What?

6. Often coalitions of: • Governments • International organizations • Academics • NGOs • Private sector companies • Examples: • Poverty Action Lab (academic) • Mathematica Policy Research (private) • Development Innovations Ventures (USAID) • International Initiative for Impact Eval. (3ie) • WB Development IMpact Evaluation (DIME) Who?

7. Ex 1: Development Innovation Venture (USAID)

8. Ex 2: International Initiative for Impact Evaluation (3ie)

9. Ex 3: MIT’s Poverty Action Lab (JPAL)

10. Ex 4: WB Development IMpact Evaluation (DIME)

11. Make publicly available training materials in partnership with other WB groups (e.g. Spanish Impact Evaluation Fund)

12. Organize trainings on impact evaluations in partnership with others (e.g. Spanish Impact Evaluation Fund)

13. Impact Evaluation Clusters • Conditional Cash Transfers • Early Childhood Development • Education Service Delivery • HIV/AIDS Treatment and Prevention • Local Development • Malaria Control • Pay-for-Performance in Health • Rural Roads • Rural Electrification • Urban Upgrading • ALMP and Youth Employment Help coordinate impact evaluations portfolio

14. How to carry one out? • Basic Elements • Comparison group that is identical at start of program • Prospective: evaluation needs to be built into design from start • Randomized evaluations generally most rigorous • Example: randomize phase-in (who goes first?) • Qualitative information – helps program design and understanding of the 'why'

15. Implementation considerations • Most programs cannot reach all: randomization provides each potential beneficiary fair chance of receiving program (early) • Review by ethical review boards • Broader considerations • Important welfare implications of not spending resources effectively • Is the program very beneficial? If we know the answer, there is no need for an IE Ethics?

16. EU2020 Targets (selected) • 75% of the 20-64 year-olds to be employed • Reducing school drop-out rates below 10% • At least 20 million fewer people in or at risk of poverty and social exclusion • Policy Options Are Many • Different ALMPs, trainings, pension rules, incentives for men taking on more home care etc. etc. • For each policy options, also different intensities, ways of delivery… Why?

17. Selective Use of Impact evaluations • Help provide answers to program effectiveness and design in EU2020 areas facing some of the greatest and most difficult social challenges • But impact evaluations can also • Build public support for proven programs • Encourage program designers (govts, ngos, etc.) to focus more on program results • Provide incentive to academia to focus energies on most pressing social issues like Roma inclusion! Why?

18. Help Encourage Social Innovation Why? +

19. Thank you for your attention!