330 likes | 422 Views
Evaluating and (Re)-Designing Conditional Cash Transfers . Orazio Attanasio (Centre for the Evaluation of Development Policies – EDePo at IFS & UCL) Impact Evaluation Network – Bogotá October 3 rd 2007. Introduction. CCT have become extremely important and visible
E N D
Evaluating and (Re)-Designing Conditional Cash Transfers Orazio Attanasio (Centre for the Evaluation of Development Policies – EDePo at IFS & UCL) Impact Evaluation Network – Bogotá October 3rd 2007
Introduction • CCT have become extremely important and visible • Since PROGRESA/Oportunidades perceived success CCT have become a model intervention • PROGRESA/ Oportunidades was also a model for the evaluation work that generated. • Oportunidades and some of the other programs have now run for a number of years and are now on the verge of a new phase • There are new challenges ahead: • Design and changes to the program • Evaluation
Introduction • I will use this presentation to talk about some of the results obtained and present some thoughts on the future of these programs • I will use mainly my experience in the evaluation of two programs: Oportunidades in Mexico and Familias en Acción in Colombia
Outline • Results: • Rural areas • Education • Health & Nutrition • Urban Areas • Take up • Other results • Challenges: • Limitations of CCT and changes to CCT programs • Supply interventions? • CCT and social policy in Latin America • Evaluation Challenges Beyond impact evaluation • The long run: • Results? • GE consequences
Results • The evaluation has been crucial to the political success of the program • Randomization is difficult to manage • Design and new modules evaluation work should be informed by the results of the evaluations obtained so far
Results: rural areas • Both in Colombia and in Mexico the results have been quite positive. • Secondary school enrollment: large increases. • Primary school enrollment: small or zero increases. • Health and nutrition: positive effects • Inputs (diet, growth and development check ups, • Height • Occurrence of illnesses • Some limitations: anemia
Results: rural areas • Consumption: • Increases, especially in food • Within food, protein rich commodities • No effects on adult goods • Some saving (investment –see Gertler et al.) • Other effects • Transfers • Social capital? • Migration • Knowledge?
Results urban areas • Mexico: • Low take up • Positive but not as positive as in rural • More ambiguous due to the lack of randomization • Colombia: • Positive in education but not as large • Zero in nutrition and health • Positive in consumption (and some saving)
New challenges: CCT expansions • Different context from rural PROGRESA/ Oportunidades or rural Familias en Acción. • Very poor regions/areas/countries • Africa, or even very marginal regions in LAC • Not so poor regions/areas countries • Urban areas. • Are CCT the right answer in these situations? • And if they are, do they need adjustment?
CCT limitations and new challenges. • Nutrition and early years: • Are we satisfied with the effects? • Anemia • Should we use the CCT to deliver Early Years stimulation? • Should the intervention be adjusted from area to area? • Malnutrition and obesity • Evaluation and design are key! • Oportunidades experiments with Nutrisano, sprinklers. • Delivering Early Years stimulation in marginalized communities.
CCT limitations and new challenges. • Education • The effect on primary school enrolment is often negligible. • (and for a reason) • Enrolment is up: what about learning and academic achievement? • Tertiary education is the big challenge, especially in urban areas? • What role do credit constraint play? • What about the quality of education? • Design and evaluation are key! • Bogotá experiment
Supply interventions • Effects have been shown to be very heterogeneous. • Are there heterogeneous effects driven by the availability of health and education services supply? • What is the quality of these services? • What are the bottlenecks?
CCT politics and social policies in Latin America • CCT and populism. • Brazil • Colombia • Mexico • CCT should not be burdened with many other aims and objectives • Operation difficulties • Transparency and accountability • The political economy of evaluation
Evaluation Challenges: beyond impact effects • Rigorous evaluations have been instrumental to the political success of the programs and their continuation • It should be instrumental to the design of changes to the programs • Experimental and quasi-experimental methods have played an important role • But it is necessary to understand the mechanisms behind the impacts observed in the evaluation
An example • Changes to the structure to the grant • Increase in the secondary school grant and reduction or elimination of the primary school grant • (but what is the effect of the primary school grant on nutritional outcomes at early ages?) • We need structural models: • Todd and Wolpin (2002, forthcoming AER) • Attanasio, Meghir and Santiago (2002)
Other changes to the program • New modules: • Nutrition • Cognitive development • Education
Results: the long run • Next to nothing is known • The analysis is difficult • … and yet these results are crucial for any cost benefit analysis. • GE effects.
Conclusions • CCT have been the most exciting piece of news in Social Policy • The risk is that they are seen as a ‘silver bullet’ that can work in all sort of situations. • … and that one neglects the new challenges ahead: • New modules (nutrition, early year stimulation). • Supply interventions • Integration with other policies (Chile Solidario, Red Juntos in Colombia). • Evaluation has a key role to play in helping to shape the new generation of CCTs and maintaining them independent from the executive.
Impact on school enrolment: 8-13, 14-17, 2nd follow-up Notes: Based on “all” (original plus switchers) treated areas Parametric specification using 4 periods of data. * Nivel de significancia del 5 % o menos ** Nivel de significancia del 1 % o menos
Es el consumo afectado por el Programa? DD. Tratamiento vs Control p<0.1, ** p<0.05, *** p<0.01 Nota: En SS se incluye a los municipios que pasaron de control a tratamiento entre línea de base y segundo seguimiento
Efectos sobre diferentes alimentos Tratamiento vs control, (D-D) ($ pesos) p<0.1, ** p<0.05, *** p<0.01 Nota: se incluye a los municipios que pasaron de control a tratamiento entre linea de base y segundo seguimiento
Efecto en la probabilidad de poseer bienes durables Variable dependiente: posesión de bien durable Dummy de tratamiento en el lado derecho de la ecuación p<0.1, ** p<0.05, *** p<0.01 Nota: se incluye a los municipios que pasaron de control a tratamiento entre línea de base y segundo seguimiento
Lactancia materna Evaluación de impacto Nivel de significancia del 10 % o menos Nivel de significancia del 5 % o menos Nivel de significancia del 1 % o menos * * * * * * Impacto de la variable lactancia exclusiva en menores de 6 meses: Para total urbano rural 0.134 *** (0.050)
Consumo de Alimentos Impacto en el Promedio de días en la semana que se consumieron los alimentos ricos en hierro * p<0.1, ** p<0.05, *** p<0.01 Nivel de significancia del 10 % o menos Nivel de significancia del 5 % o menos Nivel de significancia del 1 % o menos * * * * * *
Impacto en la probabilidad de cumplir con: Nivel de significancia del 10 % o menos Nivel de significancia del 5 % o menos Nivel de significancia del 1 % o menos * * * * * * La vacunación se realiza a los niños pequeños, pero estos no son elegibles por lo que no se aprecia impacto. Sin embargo, los controles de crecimiento y desarrollo se hacen en todas las edades
Impactos por grupos de edad(probabilidad de cumplir) El programa no tiene impacto en niños de 0 a 48 meses pues la inmensa mayoría no está inscrito en el programa
Crecimiento en la Talla Promedio. (Desviación estándar y puntos porcentuales) Impactos positivos en la zona rural, pero nulos en la urbana Nivel de significancia del 10 % o menos Nivel de significancia del 5 % o menos Nivel de significancia del 1 % o menos * * * * * *
Impacto en la Talla Promedio (cont). (Desviación estándar y puntos porcentuales)
Anemia (descriptivo) ENSIN 2005 Anemia por grupos de edad en todos los niveles del sisben: 12 a 23 meses 53.2% 24 a 35 meses 33% 36 a 47 meses 25.4% 48 a 59 meses 22.1% ENSIN 2005 Anemia en niños menores de cinco años: Nivel 1 del sisben 41.9% Nivel 2 del sisben 29.9% Niños de cinco a doce años: Nivel 1 del sisben 47.7% Nivel 2 del sisben 32.9%
Anemia (impacto) No se aprecia impacto en anemia
Impacto sobre la probabilidad de padecer diarrea El programa mejora la prevalencia de diarrea (autopercibida) en la zona rural (niños pequeños)