Background , achievements and challenges in the external evaluation process of Oportunidades

1. Background, achievements and challenges in the external evaluation process of Oportunidades November 19th, 2013

2. Background • The external evaluation process of Oportunidades (formerly PROGRESA) started at the same time the programme was implemented in rural and, eventually, urban areas. • In both areas: • The main objective of the evaluation process is to assess the fulfillment of its objectives and goals, as well as to determine its efficiency, effectiveness, operation and impact. • The external evaluation used panel data samples of households. • The external evaluation considered the full package of interventions among different age groups within the households, and did not take them separately (components). • Quantitative as well as qualitative approaches were considered in the evaluation process.

3. Background: Rural evaluation 2002 1998 1999 2000 2001 1997: Original evaluation sample (randomized) Treatment (320 localities / 16,651 households) Control – Late treatment (186 localities / 10,834 households) 152 localities/ 6,728 selected households by matching New control group

4. Background: Rural evaluation

5. Background: Urban evaluation • Quasi-experimental methods: Propensity Score Matching (blocks). • 17 states, 179 localities y 746 blocks. • States: Campeche, Chiapas, Colima, Guerrero, Guanajuato, Hidalgo, México, Michoacán, Morelos, Querétaro, Puebla, San Luis Potosí, Veracruz, Sinaloa, Sonora, Tabasco, Tamaulipas and Tlaxcala. • Methodology challenge : • The inclusion of households into the programme was made by self-selection. • Block boundaries are less clear than localities in rural areas.

6. Recent External Evaluation Framework The Performance Evaluation System (or SED in Spanish), is leaded by the Ministry of Finance, CONEVAL and the Ministry of Public Function. SED’s main objective is that budgetary decisions are taken considering sound and reliable information about the design, results and performance of social programmes, which stems from (i) external evaluation process and (ii) monitoring & supervision activities.. Performance Evaluation System Components • Indicators • Logical Framework Approach • New management of the budget process • Resources assigned by results Results Based Budget (PbR) Performance Evaluation System (SED) Programmes and Public Policy Evaluation Public expenditure quality management Annual Evaluation Program (PAE) Source: SHCP. SED .

7. Monitoring Today: Indicators Matrix Goal Contribute to brake the intergenerational transmission of poverty, by favoring the development of capacities associated to feeding, health and education of the beneficiary families of the Programme Purpose The beneficiary families of Oportunidades increase their capacities of feeding, heath and education Components HEALTH Beneficiary families that complied with their health responsibilities, received the Basic Health Kit and the in-kind grants NUTRITION Beneficiary families that complied with their responsibilities, received the cash transfer EDUCATION Beneficiary families with children and adolescents that complied with their responsibilities in primary, secondary and high school education received the school grants Activities School attendance certification of the scholarship holders in primary and secondary education School continuity certification of scholarship holders in high school education Promotion of the incorporation of schools with Oportunidades scholarship holders into the Quality Schools Programme Provision of health services Compliance certification in health responsibilities of the elderly beneficiaries Compliance certification in health responsibilities of beneficiary families Transversal Activities 1101. Beneficiary families coverage 1102. Good assistance to the beneficiary female head of family in the Assistance Centers and Oportunidades Services 1103. Resources transferred

8. Achievements and challenges in the external evaluation process • Documentation of Oportunidades’ impacts and results led to: • It’s continuity through three different presidential administrations. • It´s expansion to urban areas & the increase in its coverage • More transparency and accountability in public expenditure • Support several improvements in design and operative issues • Better identification of the external elements that hinder the impact of the Programme • Also, with the information provided by external evaluation, supervision & monitoring it is possible to identify: • Methodology challenges: (i) attrition in panel data samples (rural and urban areas), (ii) loss of control groups, (iii) combination of quantitative as well as qualitative methodologies in the evaluation process, (iv) external validity in the results • Need to monitor the program’s results using data from national surveys • Improve the coordination of the increasing actors involved in the evaluation process • Increase the opportuneness of the results in order to be taken into account into the decision making process

9. External evaluation 2014 - 2018 • Advice from government experts, academics, researchers in technical and methodology topics • Coordination among the actors involved in the evaluation process • → Improve planning activities • Search for new approaches and methodologies • Increase the participation of institutions and researchers in the evaluation activities: (i) design and interventions analysis, (ii) surveys, (iii) look over the studies. • Continuity of the panel data samples and implementation of specific surveys to evaluate new interventions. • Samples in national surveys: ENSANUT, ENIGH, ENOE.

10. Thankyou