Monitoring and Evaluation of Social Programs

1. Monitoring and Evaluation of Social Programs Kathy Lindert, World Bank Qualidade do Gasto Publico no Brasil June 26-27, 2003

2. Outline • Objectives of M&E • Conceptual Framework, example • Monitoring (role, steps, etc.) • Evaluation (role, steps, etc.) • Dissemination and Uses • Institutional Issues and Sustainability

3. Objectives of M&E • Monitoring and evaluation can help policy makers in many ways: • As planning and budgeting tool: to ensure effective allocations of government resources – planning, decision-making and prioritization • As a management tool by revealing the performance of ongoing activities at the sector, program or project level, to inform decisions about expansion, modification, or elimination of programs depending on their effectiveness • As an accountability mechanism, so that managers can be held accountable for the performance of their projects/programs and so that government can be held accountable for its performance • To obtain and maintain political support, institutional resilience (even across administrations), and financing for successful activities

4. Metas(Impactos) • Longo prazo, ampla melhoria na sociedade • Efeitos intermediários dos produtos sobre os clientes Resultados • Produtos e serviços produzidos Produtos • Tarefas empreendidos para transformar insumos em produtos Atividades Insumos • Financeiros, humanos, e recursos materiais Marco Conceitual Objetivos Implementação

5. Higher income levels; increase access to higher skill jobs Goal(Impacts) • Increased skills; more employment opportunities Outcomes • Number of youths completing secondary school Outputs Activities • Schooling Inputs • $, facilities, teachers, materials Example: Secondary Education

6. Monitoring

7. Monitoring of Implementation and Results • Monitoring is a continuous process of collecting and analyzing information to compare how well a program is being implemented against expected results • Monitoring tracks both implementationandresults • Enfoque tradicional: Monitoring of activities • Definir atividades e responsabilidades • Monitoring activities is important • No entanto: • É possível executar as atividades, cumprir as responsibilildades, mas não fazer progresso até as metas Question: how will you know when you have been successful? “Se não se reconhecer o sucesso, vai se valorizar o fracasso”

8. Impact Results Outcome Output Activity Implementation Input Main Types of Monitoring Results Monitoring Implementation Monitoring (enfoque tradicional)

9. Key Steps for Monitoring • Acordo sobre os resultados a monitorar • Seleção de indicadores de estos resultados • Medição de “baseline” • Eleição de metas quantitativas • Gerenciamento com relação às metas

10. 1. Acordo sobre os resultados a monitorar • Outcomes make explicit the intended objectives of governmental action – the vision • Consensus about these outcomes, and this vision, is important • Participation in agreeing on outcomes is key • Key stakeholders should be involved (e.g., government, civil society, donors, etc.) • Desenhar para sucesso • Evitar resultados impossíveis • Motivar pessoal com uma visão inspiradora • Criar linha de vista (“line of sight”) para todos que trabalham no sistema entre as suas ações e o resultado desejado “Se você não sabe para onde vai, todos os caminhos estão certos”

11. 2. Seleção de indicadores • Outcome indicators are not the same as outcomes • Indicadores devem medir resultados mas tambem processos e productos • Um resultado corresponderá típicamente a mais de um indicador • Respondem à pergunta: “Cómo conheceremos o sucesso, se o vermos?”

12. 2. Seleção de indicadores, cont.

13. 3. Establishing Baseline Data on Indicators A performance baseline is… • Information (quantitative or qualitative) that provides data at the beginning of, or just prior to, the monitoring period. The baseline is used to: • Learn about recent levels and patterns of performance on the indicator; and to • Gauge subsequent policy, program, or project performance

14. Data Sources May Be Primary or Secondary • PRIMARY data are collected directly by the organization, for example, through surveys, direct observation, and interviews. • SECONDARY data have been collected by someone else, initially for a purpose other than that of the program. Examples include survey data collected by another agency, a Demographic Health Survey, or data from a financial market. • Secondary data often can save money in acquiring data, but be careful!

15. Dados de “Baseline” Pesquisas de painel Entrevistas cominterlocutores chaves Conversascompessoasafetadas “Grupos de foco” Pesquisas únicas Análisesdos registrosofficiais Observação de participantes Observação direta Census Entrevistasdas comunidades Experiências controladas Visitas de campo Questionários Métodos informais/menos estruturados Métodos formais/mais estruturados

16. Practicality • Are the data associated with the indicator practical? • Ask whether… • Quality data are currently available • The data can be procured on a regular and timely basis • Primary data collection, when necessary, is feasible and cost-effective

17. 4. Metas • Targets are the quantifiable levels of the indicators that a country wants to achieve at a given point in time • Precondições • Clara compreensão do passado recente • Conhecimento dos recursos disponíveis • Considerações políticas • Talvez o mais importante sendo o custo de não cumprir a meta… • Gerenciamento para sucesso • Seja realista! • Faixas? • Nota: gestão por resultados não depende críticamente das metas

18. 5. Monitoria e gestão por Metas • “Abertura” faz parte do enfoque • Desenvolver estratégias de divulgação para o público, internet, etc. • Monitoramento deve ser contínuo • Informações devem correr “verticalmente” e “horizontalmente” • A nenhum ponto do sistema devem ser coletados dados sem ser analisados • Cria incentivos em favor da qualidade dos dados

19. 5. Monitoria Metodologia estávele consistente Fidelidade Validade Pontualidade Medida clara e diretado desempenho relevante Dados atualizados esuficientemente frenqüentepara influenciar decisões gerenciais

20. Accesso Accesso Tempo Melhorar acesso aos mercados Tempo Melhorar acesso aos mercados 5. Monitoria: Freqüência • Quanto mais observações tem, tanto mais os diagnósticos serão • Seguros • Sofisticados ?

21. Evaluation

22. O papel das avaliações • Results-Based Evaluation • An assessment of a planned, ongoing, or completed intervention to determine its relevance, efficiency, effectiveness, impact and sustainability. The intent is to incorporate lessons learned into the decision-making process. • Evaluation Addresses • “Why” Questions: What caused the changes we are monitoring • “How” Questions: What was the sequence or processes that led to successful (or not) outcomes • “Compliance/ Accountability Questions”: Did the promised activities actually take place and as they were planned?

23. Uses of Evaluation • To make resource decisions • To re-think about the causes of a problem • To identify issues around an emerging problem, i.e. children dropping out of school • To identify alternatives • To build support of public sector reform / innovation • To help build consensus among stakeholders on how to respond to a problem

24. Evaluation Means Information on:

25. General Types of Evaluations & PERs • Evaluations can occur at many levels: • Evaluations of individual staff performance (including project managers) • Evaluations of organizational units • Evaluations of programs (focus of this presentation) • Evaluations of sectors/policy areas (e.g., cross-sectoral issues, fiscal policy, overall education policy) • Public expenditure reviews generally evaluate the broadest level (evaluations of sectors/policy areas), but they also draw on evaluations of programs

26. Types of Program Evaluations • Process Implementation Evaluations: • Provides detailed information on whether program is operating as intended (are we doing things right?) • Provides continuous feedback to assist managers • Provides detailed information to those interested in scaling up or replicating • Identifies bottlenecks • Cost-Benefit Evaluations: • Calculates the cost of the intervention relative to results • Are we doing things efficiently?

27. Types of Program Evaluations, cont. • Impact Evaluations: • Measure net effect of program to determine whether or not objectives are being achieved • Tries to establish causality between activities and impacts • Impacts include both those that were intended and those that were not • Impact evaluations require a clearly established counter-factual: • What would have happened without the project? • Generally need to compare those “with” the project (beneficiaries, “treatment group”) and those “without” the project (“control group”) both before and after project

28. Stages of Evaluation • Clarify objectives of evaluation • Explore available information • Design evaluation • Form evaluation team (capacity?) • Collect information (sample, questionnaire, pilot, data collection, entry, quality control, etc.) • Analysis • Reporting, disseminating, discussing results • Using results: feedback to policies, budgets, planning

29. Dissemination and Uses of Information

30. Divulgação • Informar sobre o estado dos projetos • Prover índicessobre problemas • Explicar problemas • Criar oportunidades para considerar melhoras o alternativas • Prover informaçãosobre tendências

31. Usando as Conclusões • As a planning and budgeting tool: • Formular/justificar orçamentos • Ajudar alocação operacional de recursos • As a management tool: • Melhorar os serviços • As a performance/accountability tool: • Responder às demandas para “accountability” • Motivar pessoal • Monitorar fornecedores • Provocar debate sobre problemas • Reforçar confiança pública

32. Institutional Issues and Sustainability • Demand for M&E is crucial • Structural requirements (ex. national law that mandates M&E of all federal programs) • Budgetary and planning links • Papéis claros • Papel formal para organismos de planejamento e finanças • Credibilidade • Sistema deve entregar boas e más notícias! • Proteção dos gerentes contra conseqüências gravas • Responsabilidade (“Accountability”) • Papéis da sociedade civil, mídia… • Capacidade técnica