Monitoring & Evaluation

Monitoring & Evaluation David Wilkinson Presentation to AAA-Cambodia 22 November 2012

Overview • What is monitoring and evaluation (M&E)? • How does M&E contribute to the effectiveness of development programs in Cambodia and in the region? • What are the skills required to be an M&E specialist?

What is monitoring and evaluation? • Wrong question! • What ARE monitoring and evaluation? • Why are they linked?:- monitoring & evaluation

Other linked pairs: • F&A • F&B • D&D • M&CH • B&B

Other linked pairs: • Finance & administration (F&A) • Food & beverage (F&B) • Deconcentration & decentralization (D&D) • Maternal & child health (MCH) • Bed & breakfast (B&B) • Fish & chips • Any others?

Monitoring • The ongoing process by which stakeholders obtain regular feedback on the progress being made towards achieving their results. • Monitoring continuously tracks performance against what was planned, by collecting and analyzing data on performance indicators.

Why monitor? • Monitoring provides continuous information on whether progress is being made toward achieving results (outputs, outcomes, goals) through record keeping and regular reporting systems.

How monitoring is done • Systematic data collection • Field visits • Review meetings • Self-assessment questionnaires

Evaluation • A rigorous assessment of a completed and/or ongoing intervention to determine the extent to which it is achieving stated objectives. • Evaluation determines the relevance, efficiency, effectiveness, impact, and sustainability of the intervention.

Why evaluate? The major purpose of evaluation is to facilitate decision-making on policy, programmes and planning.

Complementary roles of monitoring and evaluation

How do monitoring and evaluation contribute to the effectiveness of development programs in Cambodia and in the region?

Planning Monitoring Evaluation Reporting

Planning: The process of establishing results frameworks, proposing strategies, outlining the implementation arrangements, identifying partners and allocating resources to achieve those goals .

Planning: The process of establishing results frameworks, proposing strategies, outlining the implementation arrangements, identifying partners and allocating resources to achieve those goals Monitoring: The ongoing process by which stakeholders obtain regular feedback on the progress being made towards achieving their results.

Planning: The process of establishing results frameworks, proposing strategies, outlining the implementation arrangements, identifying partners and allocating resources to achieve those goals Monitoring: The ongoing process by which stakeholders obtain regular feedback on the progress being made towards achieving their results. Evaluation: A rigorous assessment of completed or ongoing programme to determine the extent to which it is achieving stated objectives and contributing to decision-making.

Planning: The process of establishing results frameworks, proposing strategies, outlining the implementation arrangements, identifying partners and allocating resources to achieve those goals Monitoring: The ongoing process by which stakeholders obtain regular feedback on the progress being made towards achieving their results. Evaluation: A rigorous assessment of completed or ongoing programme to determine the extent to which it is achieving stated objectives and contributing to decision-making. Reporting: An essential element of an accountability process, where people accountable for performance report on what has been achieved compared to expectations.

Results-based management (RBM) • A result is a measurable or describable change arising from a cause and effect relationship • A result is NOT the completion of activities!! • Reporting must be done about a programme’sresults, NOT its activities.

Example 1 OLD WAY - Reporting on Activities: “The programme supported the creation and the coordination of clubs in 34 secondary schools. Social mobilization campaigns were organized, with prizes for high-performing girls.”  But what was the result? NEW WAY - Reporting on Results: The creation of clubs in 34 secondary schools (33% of all secondary schools) led to a 66% increase of SRH knowledge for girls aged 10-14 over the baseline measure.

Example 2 OLD WAY - Reporting on Activities: “Support was provided to the MoH to enhance condom promotion among at-risk youth. Activities included training workshops on condom use and sexual health.”  But what was the result? NEW WAY - Reporting on Results: UNFPA-supported training workshops in condom use and sexual health for at-risk youth led to a 21% increase in the rate of regular condom use for workshop participants, as compared to national data on condom use.

Some examples of how evaluations contribute to decision-making and increased effectiveness of development programmes

Evaluation of the MoH/NGO Home Care Programme for People with HIV/AIDS in Cambodia (June 2000) Evaluation found that home-based care (HBC): • is reducing the suffering of people living with HIV/AIDS and improving the quality of their lives and the lives of their families and caregivers; • is increasing understanding of HIV/AIDS by helping to forge links between care and prevention and reducing discrimination against PLHA in the community; • by providing social and economic support, HBC is helping to empower some of the poorest and most disadvantaged individuals and families in the community EVALUATION CONTRIBUTED TO: • IMPROVED MANAGEMENT OF HOME-BASED CARE PROGRAMMES • INCREASED COLLABORATION BETWEEN MOH AND NGOS • EXPANSION OF HOME-BASED CARE ACROSS CAMBODIA

Evaluation findings: • Paying for health care, particularly secondary or tertiary care, is still a major cause of destitution among the poorest sections of the community • major failure of exemption schemes in hospitals to protect the poor • Mechanism for financing exemptions must be separated from mechanism for financing salary supplements and operating costs EVALUATION CONTRIBUTED TO: • EXPANSION OF HEALTH EQUITY FUNDS ACROSS CAMBODIA • IMPROVED TARGETING OF EXEMPTIONS FOR THE POOR • INCREASED SUPPORT FOR VILLAGE HEALTH SUPPORT GROUPS The Impact of User Fees on Equity, Access and Health Provider Practices in Cambodia- report prepared for Health Economics Task Force, Ministry of Health / WHO, May 2001, Wilkinson D, J Holloway & Pierre Fallavier

Assessment of the Proposal Development and Review Process of the Global Fund (Feb 2006) Evaluation findings: • weak GF communications related to proposal development and review processes; • weak CCM governance and functioning inhibiting the success of CCM proposal preparation and submission ; • GF system of “rounds” undermining coordinated approaches such as SWAps, and is a major source of disharmony for national planning, implementation, monitoring and reporting systems; • quality of technical support to countries is uneven. NGOs and non-health ministries are severely disadvantaged in accessing TS EVALUATION CONTRIBUTED TO: • Improved communications and clarification of Global Fund principles, policies and procedures; • Improved CCM governance and functioning; • Introduction of rolling continuation channel and grant consolidation (and recent discontinuation of ‘rounds’’); • Improved country ownership, donor harmonization and Global Fund alignment with national systems; • Increased use of technical assistance and partnerships to improve the country proposal development process.

Evaluation of the GF Local Fund Agent System (April 2007) Evaluation findings: • LFAs had insufficient capacity in health programme skills; • Lack of a quality assurance system; • Limited use of in-country partnerships; • Lack of a comprehensive performance evaluation system for LFAs; • No operational manual to govern LFA management process EVALUATION CONTRIBUTED TO: • ENGAGEMENT BY GF OF MORE LFAs WITH health PROGRAMME skillS • INCREASED ENGAGEMENT OF LFAs WITH COUNTRY STAKEHOLDERS • IMPROVED LFA PERFORMANCE AND ACCOUNTABILITY

2012 Evaluation of UNFPA’s Asia-Pacific Regional Programme • Planning • Inception report • Data collection • Analysis • Shared ownership of findings, conclusions and recommendations with key stakeholders • Report writing & revision

Methodology • Document review of more than 1000 key documents • Stakeholder mapping • In-depth interviews/group discussions with over 100 key informants from a range of stakeholder groups • Five-day country visits to Indonesia, Sri Lanka, Cambodia • Electronic survey sent to 52 respondents from 23 counties • 11 case studies of regional initiatives • Benchmarking against other UN agencies • Debrief presentation with key stakeholders

What are the skills required to be an evaluation specialist?

What are the skills required to be an evaluation specialist? • Planning skills – conceptualize the evaluation and plan methodology, resources and timeframe • Teambuilding skills – train evaluation team, build team spirit, and ensure common understanding of how to use each team member’s comparative advantage • Programme skills & experience – understand how it all fits together • Curiosity – willingness and ability to follow-up and probe and to ask difficult questions in a non-threatening way • Skills in building and fostering stakeholder trust • Negotiation skills – getting access to hard-to-reach stakeholders • Management skills – of evaluation team, client and stakeholders • Communication skills – ensure shared understanding of evaluation objectives, process, outcomes and recommendations • Analytical skills – synthesize and triangulate a wide range of data from multiple data sources to provide a coherent picture • Writing skills – produce a clear, succinct report that tells a coherent story

So, would you like to be an M&E specialist? Online training: • https://training.measureevaluation.org/ • www.theglobalfund.org/en/me/learning/ • www.slideshare.net • www.jica.go.jp

THANK YOU!

Monitoring & Evaluation