T raining I n C linical R esearch Introduction to Global M&E Nathan Smith, MPH October 25, 2012

1. Training In Clinical ResearchIntroduction to Global M&ENathan Smith, MPHOctober 25, 2012

2. M&E Overview • Key Concepts in M&E • Main components of an effective M&E system • Examples of M&E in practice • Data Triangulation • ART Outcomes • Data Quality Assessments • Cohort

4. Key Concepts in M&E

5. Difference between ‘M’ and ‘E’ • Monitoring: • The routine tracking of the key elements of programme or project performance, usually inputs and outputs (may include also tracking of short term programme outcomes and long-term impact) • Evaluation: • The episodic assessment of the change in targeted results that can be attributed to the programme or project intervention. • Evaluation attempts to link (“causally”) a particular output or outcome directly to an intervention after a period of time has passed.

6. Use of monitoring and evaluation To manage, improve, and be accountable for: Resources—staff, money, equipment, and other things that you put into your project or programme Activities—things you do in a project or programme Results—things that happen as a result of what you put in and what you do in your project or programme

7. Why are monitoring and evaluation important? • Provides data on programme progressand effectiveness • Improves programmemanagement (managing by results) • Informspolicy-making, advocacy and decision-making • Ensures most effective and efficientuse of resources • Allows accountability to stakeholders, including donors

8. Routine collection of programmatic data Special studies Reporting Management Programme activities with M&E M&E

9. Results chain Impact: long-term effect on disease mortality and morbidity (e.g. prevalence, incidence, mortality) Outcome: intermediate effects on behaviour change (e.g. condom use, ITN use) Output: immediate effects of the intervention (e.g. people reached, services delivered) Process: capacity building (people trained, service points supported, drugs procured) Input: resources made available to support the interventions (e.g. funds, human resources)

10. Different sources and measurement frequency* Input  Process  Output  Outcome  Impact Disease surveillance Vital registration Household or facility surveys Programme monitoring Monthly/quarterly Every 2-5 years Annual * adapted from MEASURE Evaluation

11. Results chain Effectiveness Programme / Ministry level STRATEGY Efficiency Project / department level WORK PLAN 6months – 1 year After implementation immediate immediate 2 to 5 years 5 years or more Activities Outputs Outcomes Impact Inputs NationalLevel Indicators Project / Department Level Indicators Source: World Bank GAMET

12. Strategic planning for M&E: setting realistic expectations Monitoring and evaluation pipeline ALL Most Some #of projects Few Outcome monitoring/ evaluation Impact monitoring/ evaluation Input/ output monitoring Process Evaluation Levels of monitoring & evaluation effort Adapted from Rehle & Rugg, M&E Pipeline Model, FHI2001

13. Program Program Program Program Program Program Program Program Multi-agency Evaluation Logic Model Outputs Short-Term Outcomes Intermediate Outcomes NAP Global Fund inputs World Bank inputs Long-Term Impacts USG inputs Other inputs

14. CDC Global AIDS Programme evaluation framework and illustrative data types Assessment & planning Output Outcomes(Intermediate effects) Input Impact (Long-termeffects) Activities Situation analysis Response analysis Stakeholder needs Resource analysis Collaboration plans Provider behaviour Risk behaviour Service use Clinical outcomes Quality of life HIV Incid/Prev Social norms STI incid/prev AIDS morb/mort Economic impact Staff Funds Materials Facilities Supplies Trainings Services Education Treatments Interventions # Staff trained # Condoms distributed # Test Kits distributed # Clients served # Tests conducted Programme development data Population-based biological, behavioural & social data Programme-based data

15. A public health approach to unifying AIDS M&E Are collective efforts being implemented on a large enough scale to impact the epidemic (coverage; impact)? Surveys & surveillance Determining collective effectiveness OUTCOMES & IMPACTS MONITORING Are interventions working/making a difference? Outcome evaluation studies Monitoring & evaluating national programmes OUTCOMES Are we implementing the program as planned? Outputs monitoring OUTPUTS What are we doing? Are we doing it right? Process monitoring & evaluation, quality assessments ACTIVITIES What interventions and resources are needed? Needs, resource, response analysis & input monitoring INPUTS Understanding potential responses What interventions can work (efficacy & effectiveness)? Are we doing the right things? Special studies, operations research, formative research & research synthesis What are the contributing factors? Determinants research Problem Identification What is the problem? Situation analysis and surveillance

16. Moving Beyond Routine Program Monitoring: Program Evaluation

17. To answer key public health questions about populations and programmes to ensure: Appropriate tailoring of prevention programmes Correct, intentional implementation Effectiveness of programmes in terms of reach, coverage, intensity, quality and behavior change Purpose & Role of Evaluation 17

18. The “Big” Questions • Coverage:Are we reaching a critical mass of affected populations? • Reach: Are we reaching the right populations? • Quality: Are we doing the right thing well? Are interventions showing intermediate to long term effects? • Intensity: Are we doing enough of the right thing to make a real difference? 18

19. Formative: Provides information on populations (geographic, behavioral) that can be used to inform program planning & implementation Process: Provides information on program implementation Output monitoring: Provides information on immediate effects of program (for ex., number of people receiving services) Outcome Monitoring & Evaluation: Provides information on the intermediate effects of an intervention (measured as change) Types of Evaluation 19

20. Identify key questions evaluation needs to answer Prioritize questions and programmes for evaluation Engage stakeholders in process (program, SI & policy) Secure necessary resources Ensure evaluability of program SMART objectives Logic model Match evaluation approach to key questions Be practical! Identify data sources (existing, new) that can be used to answer key questions Develop & implement evaluation plan Plan for and follow-up with data use and feedback tailored for various purposes and audiences Process for Selecting Evaluation 20

21. Tools for Evaluating Programmes FHI – Monitoring & Evaluation Handbook(http://www.fhi.org/en/HIVAIDS/pub/Archive/evalchap/index.htm) Global Fund M&E Toolkit (http://www.theglobalfund.org/documents/me/M_E_Toolkit.pdf) Measure Evaluation Guide to M&E(http://www.cpc.unc.edu/measure/tools/hiv-aids/me-of-national-aids-programmes) Project SEARCH Global M&E Website: UNAIDS & USG 21

22. Standardized Framework • Context for each population • Drivers • Risk behaviors • Corresponding program goals • Program information • Level (individual, small group, community, structural) • Type of activities • Delivery mode • Program-level output indicators • Outcome indicators

23. Sample Framework for Output Monitoring

24. Assess Context

25. Surveillance and Risk Behaviors

30. Data Sources • Input, output data source: program monitoring records: monthly/quarterly/annually • Outcome/Impact data source: population based surveys, behavioral surveys with special populations and/or special studies – typically with bio markers to document prevalence: every 2-4 years; look for trends

31. But M&E exists to serve program 31

32. So – Return to Beginning: Overall Priorities for M&E M&E intended to provide support to program areas to enhance: • Sustainability • Quality • Alignment • Capacity Building • Scale • Utility

33. Practical Example: M&E for TCE • Logic model developed/adapted for actual program implementation & data collection tools • Review & alignment of program indicators with NSF, UNAIDS/UNGASS and PEPFAR output monitoring and outcome indicators • Meeting with Field Officers on use of the data collection tool • Revision of the tool to better measure prevention strategies and to match actual use of the tool by field officers

34. Main components of an effective M&E system

35. Background Over the last several years, there has been an evolution in the field of M&E involving a movement away from traditional implementation-based approaches toward new results-based approaches

36. A functioning M&E system provides a continuous flow of information that is usefulinternally and externally. • The internal use of information on progress, problems, andperformanceis as a crucialmanagement toolthat helps managers ensure that specific targets are met • The information from an M&E system is important to those outside the organizationwho are expecting results, wanting to see demonstrable impacts

37. How can results-based M&E systems support better governance • Transparency and accountability • External and internal stakeholders will have a clear sense of the status of projects, programmes, and policies • Changes in the political dynamics – the political side of M&E • Information can empower policymakers to make better-informed decisions

38. 12 components of the organizing framework

39. 12 Components Source: World Bank/GAMET

40. Why relevant and important for you? New ‘gold standard’ for M&E systems Create a common language Endorsed by a global reference group for M&E Agreed to by all development partners Results-based approach to M&E: an objective for each component Why components? Systems approach System is a group of interconnected and interrelated components to form a whole (Senge, 1990) Developed from operational experience Initially for HIV sector, but expanded to other sectors 12 components of the Functional M&E system Source: World Bank GAMET

41. People, partnerships and planning 1. Organisational structures with HIV M&E functions 2. Human capacity for HIV M&E 3. Partnerships to plan, coordinate, and manage the HIV M&E system 4. HIV M&E plan 5. Annual costed HIV M&E work plan 6. Advocacy, communications, and culture for HIV M&E

42. Collecting, verifying, and analyzing data 7. Routine HIV programme monitoring 8. Surveys and surveillance 9. National, sub-national, and organisational HIV databases 10. Supportive supervision and data auditing 11. HIV evaluation and research

43. Using data for decision-making 12. Data dissemination and use

44. Activity rings of the 12 components model • The 12 components are rooted in a results-based approach to M&E system building: • For every component, long, short and medium term results have been identified.

45. 1. Organisational structures with HIV M&E functions The National system should be coordinated by one organisation (NAP) which has an M&E Unit; there should also be structures within organisations that support M&E functions M&E staff are required at all levels, including HIV M&E focal points at the MOH and other line ministries, at sub-national governmental levels and in organisations or facilities providing HIV services. Efforts should be made to mainstream HIV into already existing structures for planning and coordination of health programmes at the sub-national and organisational level.

46. 2. Human capacity for HIV M&E • M&E human capacity building requires a wide range of activities: • formal training, in-service training, mentorship, coaching and internships • it should focus on technical skills, but also address skills in leadership, financial management, facilitation, supervision, advocacy and communication

47. 3. Partnerships to plan, coordinate, and manage the HIV M&E system National M&E partnerships should be managed through a national M&E Technical Working Group that meets regularly and includes all relevant stakeholders, (all organisations responsible for M&E activities listed in the costed national HIV M&E work plan) At the organisational level, M&E partnerships are important both internally and externally; M&E, program and management must be closely aligned and share common goals

48. 4. HIV M&E plan Describes the implementation of all relevant components within an organisation of its M&E system. Describes a 3-5 year implementation strategy, indicate resource requirement estimates and outline a strategy for resource mobilization. Should be reviewed and updated regularly to make adjustments in data collection associated with revisions of the National Strategic Plan Objectives of the M&E plan should be explicitly linked to the HIV National Strategic Plan to ensure that relevant data are collected to measure the progress in the HIV response

49. 5. Annual costed HIV M&E work plan The M&E work plan contains activities, responsible implementers, timeframe, activity costs and identified funding. It is a joint work plan that integrates M&E across various programmes and stakeholders The costed M&E work plan should reflect agreement on who will implement and finance each activity

50. 6. Advocacy, communications, and culture for HIV M&E Obtaining political support for transparency and accountability related to the HIV response is an important component of the communication and advocacy strategy One way to gain political support is to identify an ‘M&E champion’ within an organisation who can promote M&E among his/her peers, to help foster an understanding about the importance of investing in quality data for policy formulation