DIRECCIÓN DE PLANEACIÓN Y NORMATIVIDAD DE LA POLÍTICA DE EVALUACIÓN

1. DIRECCIÓN DE PLANEACIÓN Y NORMATIVIDAD DE LA POLÍTICA DE EVALUACIÓN The Importance of the Institutionalization ofM&E: The Mexican Case Gonzalo Hernández Licona 2013

2. 1997Progresa Evaluation 2000 Congress’ Decree: annual external evaluations to all federal programs 2001 Evaluation Units within ministries 2001 National Audit Office. Congress 2002 First official poverty estimates. Ministry of Social Development 2004 Law of Transparency and Public Access to Information 2005-6 Social Development Law CONEVAL. National Council for the Evaluation of Social Policy 2006 Budget Law: Performance Evaluation System 2007 Evaluation Guidelines CONEVAL, Ministry of Finance, National Audit Office 2009 CONEVAL. Multidimensional Poverty Methodology 2009 Government Accountability Law InstitutionalmilestonesMexico 1997-2009

3. USING THE RESULTS fromm&e in social policy

4. Changes in social policy due to the M&E System • The government launched in January 2013 an aggressive strategy to reduce hunger and extreme (multidimensional) poverty. • In September 2013 the President sent a bill to Congress to reshape the way social security has been working for almost 50 years. • The fiscal reform of 2013 changed the way resources will be allocated between States and Municipalities. • Since 2009, the Ministry of Finance uses information from the evaluation process to elaborate the annual budget. • There is a constant public and in-house debate and analysis (Presidency, ministries, programs, Congress, media) about the public evaluations of social programs and policies. • CONEVAL published that poverty and food deprivation increased between 2008 and 2012, and the Government acknowledged it. • Multidimensional poverty estimations showed that the larger deprivation is the access to social security (61%) • The reform has the goal to reduce extreme poverty according to the new multidimensional poverty methodology. • There are now annual information for almost 270 social programs. For some of them there are impact, process, design evaluations. • All evaluations are discussed with policy makers and all of them are public.

5. Changes in social policy due to the M&E System • The impact evaluation demonstrated reductions of gastrointestinal diseases for children do to the program • The design evaluation of the program showed various design and operational flaws at a considerable cost. • Theimpactevaluatonshowedtheironhadnotbeen absorbed bychildren. • The distributional analysis of various programs illustrated that the program was very regressive. • In July 2013 CONEVAL launched and evaluation to the CNCH, which was discussed with the Ministry of Social Development and also was public. The goal: On going improvement of the strategy. • The budget for the Cement Floor program was increased nearly 400% between 2007 and 2012. • The program Employment for the Youth was cancelled in 2009. • The food supplement of the Progresa-Oportunidadesprogramwas modified in order to have a better iron formulae in 2003. • The rural program Procampo is now less regressive, due to a design change in 2009. program were changes due to the evaluation. • Due to the design evaluation of the National Crusade Against Hunger, the strategy improved definitions, diagnosis and coordination tools

6. Varioussocial programs and theirbudgethavechangedduetothe“Evaluation-ImprovementMechanism”: /1 InformationfromtheDiagnostics of theIndicatorsMatrixesforResults 2008 and 2010. /2InformationfromtheFollow-up of Aspects Susceptibles of ImprovementSystem. Theestimationconsidersthe aspects susceptibles of improvementattendenby a 100% tomarch 2012. /3 Theestimationsincludestwo programas thatfinishedtheirobligationsbut are not in operationsince 2011.

7. Changingthe rules of thegame

8. Challenges for constructing an Evaluation system: • Institutional: It’s almost impossible to have a public Evaluation system without a proper institutional arrangement: evaluation mandate, evaluation unit, feedback procedures for policy improvements, norms about transparency. • Technical:Suitablemethodologies,trained evaluators, administrative records, information. Constructing an M&E system is a political and institutional task with technical elements….not the other way round….

9. Them&esystem

10. M&E: Budget decisions based on Results. MeasuringPoverty (variousdimensions) Evaluation of Programs and Policies • Identify the main social problems to prioritize strategies and resources. • Public Policy Actions • Which programs can be related to the social priorities? • Improvement of Programs and Policies

11. Evaluation guidelines for all institutions, together with the Ministry of Finance: The Demand for evidence was clearer now Planning Evaluation Annual Evaluation Plan National Development Plan Results Consistency & Results Evaluations Ministries’ Strategic objectives Impact Evaluations Process Evaluations Logical Framework: All Programs Policy Evaluations Recommendations’ follow-up Annual Performance Report

12. Products and facts • There are poverty figures at a national, state and municipality level • Almost 600 programs have Log Frameworks. All social programs plus others… • We hired ECLAC to help us with the capacity building for the Log models. • We have offered an evaluation course every year for policy makers and every 2 for researchers. • 25% of all indicators are oriented to measure Results • 150 programs are evaluated every year (now every two). There are around 179 social programs. This is 90% of the total budget. • 270 one-page summaries every year. • Between 3-4 impact evaluations are done every year. • CONEVAL budget is around $23 millions on even years (household survey); $16 millions on odd years. But there is more budget for evaluation within ministries. • We can find on the internet: • Poverty estimates • All the evaluations • The program’s point of view about its evaluation • Each program’s Work Plan

13. annex

14. Since 1997 there is a balance of power between Congress and the President. Congress demanded the creation of an independent institution, CONEVAL, for the measurement of poverty and for evaluation. Information on poverty and evaluation is public and transparent. The Finance Ministry has been an important ally in the process. CONEVAL and the Ministry of Finance produced Guidelines for the Evaluation of Programs. The center of the Guidelines: Results The Guidelines also created an “Evaluation-Improvement Mechanism” Most ministries have cooperated in the design of the M&E system. Fine balance between: Accountability and Policy Improvement. The (autonomous) Statistical Office has invested a lot on data collection. Good academic skills and suitable evaluation methodologies have been increasingly disposable for the past 15 years. Why all these changes in a country where citizens usually mistrust government information and where political parties fight for resources?

15. 1997 ProgresaEvaluation 1997 Balance of powerbetweenCongress and theExecutive 2000 Congress’ Decree: annualexternalevaluationstoall federal programs 2001 EvaluationUnitswithinministries 2001 NationalAudit Office 2002 Firstofficialpovertyestimates2004 Law of Transparency and Public Access to Information 2004-5 Social Development Law CONEVAL. National Council for the Evaluation of Social Policy 2006 Budget Law: Performance Evaluation System 2007 Evaluation Guidelines CONEVAL, Ministry of Finance, National Audit Office InstitutionalmilestonesMexico 1997-2007

16. CONEVAL • Mandate: • Measurement of Poverty at the National, State and Municipality level • Evaluation of social programs and policies • Governance • CONEVAL is part of the Executive, but • The Board has 8 seats. The majority of Board members (6) are academic researchers elected by all the States, representatives from Municipalities, Congress and the Executive (44 votes) • Technical and managerial independence

17. Evidence must be part of a planning system • Why do we need evidence? • Improvesocial policy • Make better decisions (management, design, budget…) • Accountability • Identify and measure social challenges • Analysis • What works? • Program design • Budget • Implementation • Monitoring & Evaluation

18. Measuring poverty by Law • Current income per capita • Educational gap • Access to health services • Access to social security • Quality of living spaces • Housing access to basic services • Access to food • Degree of social cohesion Social Development Law Dimensions for poverty measurement National States Municipalities

19. Main Products Social Programs Evaluation Poverty Measurement and Analysis • Multidimensional Poverty Measurement (National and State Estimations) • Economic wellbeing • Education lag • Access to health services • Access to social security • Quality and spaces of the dwelling • Access to basic services in the dwelling • Access to food • Social cohesion • Social Gap Index • Poverty Labor Trend Index (ITLP) • Poverty Maps • Income Poverty Measurement • Annual Evaluation Plan • Log Framework: Programs • Consistency & Results Evaluations • Impact Evaluations • Complementary Evaluations • Performance Evaluations • Programs’ Performance Summary • Social Development Programs’ Inventory • Thematic Evaluations • Policy Evaluations • Recommendations’ Follow-up • Document for Budgetary Considerations

20. Information: examples

21. Evolution of multidimesional poverty, Mexico 2008-2010 Millions of persons Social Deprivations 2008 10.6 % 11.7 millions Population whose income is below the wellbeing treshold 6 4.8 Access to food 4.1 2010 10.4% 11.7 millions 3.5 3.2 4 2 Extreme Poverty 0.0 0 Population whose income is below the minimum wellbeing treshold Poverty -0.8 -2 Educational gap -2.3 -2.5 2008 44.5 % 48.8 millions -2.9 2010 46.2 % 52.0 millions Quality and spaces of the dwelling -4 Access to basic services in the dwelling -6 Access to social security -8 -9.0 -10 Access to health services Sorce: estimations of CONEVAL based on information of MCS-ENIGH 2008 y 2010

22. Percentage of population living in povertybymunicipality. Mexico, 2010 Sorce: estimations of CONEVAL based on information of MCS-ENIGH 2010 and the Census 2010

23. Evaluation: • Programs’ Performance Summary

24. challenges

25. ChallengesforMexico • Bettercoordinationbetweenevaluationinstitutions and budget allocation decisions based on results:Planning vs Budget; CONEVAL, Ministry of Finance and Ministry of Audit • More capacitiesonevaluation and measurementtecniques • Improveindicators of diverseprograms • Indicators of quality of educational and healthservices • List of beneficiaries • Evaluation of legislation and norms’ changes • Evaluation of Sectors/Ministries. Strategicobjectives. • Rigorousevaluations and transparencyin states and municipalities. • Take more intoaccounttheinformationfromevaluations in budgetary, operative and strategicdecisions.

26. Somelessons

27. Lessons Countriesshouldfindtheirowninstitutionalarrangement: Chile, Colombia, South Africa, USA, Canada, Mexico, China, etc. Thebalance of powerbetweenCongress and theExecutiveisimportant Publicpressurehelps Credibilityis at the center of theinstitutionalarrangement Ifthe country engaged in specificevaluationspreviously, use theexperienceto set up a system in thefuture Champions are always a key factor Internacional helpisimportant…but try tobuildyourownpath Impactevaluations are notalwaysthefirststep, more basicevidenceissometimes more important South-Southexchanges of knowledge and innovation Keepthe fine balance betweenTransparency and PolicyImprovement. On of themainusers: Ministry of Finance