Planning for Measurement, Monitoring and Evaluation the Reform

1. Planning forMeasurement, Monitoring and Evaluation the Reform Assoc. Prof. Dr. WongsaLaohasiriwong Faculty of Public Health, KhonKaen University

2. What is it?

3. Monitoring and Evaluation (M&E) Plan • is a guide as to what you should evaluate, what information you need, and who you are evaluating for. Source: Developing a Monitoring & Evaluation Plan. From evaluationtoolbox.net.au/index.php? option=com_content&view=article&id=20&Itemid=159

4. Characteristic of M&E Plan • Utility - serve practical information needs of intended users. • Feasibility - be realistic, prudent, diplomatic and frugal. • Propriety - conducted legally, ethically, and with regard to those involved in and affected by the evaluation . • Accuracy - reveal and convey technically accurate information. Source: Developing a Monitoring & Evaluation Plan. From evaluationtoolbox.net.au/index.php? option=com_content&view=article&id=20&Itemid=159

5. Developing & Implementing an M&E Plan: Logic Model Outcomes • Inputs • Human resources • Understanding of the program • Authority and mandate • Stakeholders • Processes • Advocate • Assess strategic information needs • Assess information systems capabilities • Achieve consensus and commitment • Develop mechanism for M&E plan review • Prepare document for final approval • Output • M&E Plan Document • Short-term • M&E System for obtaining Strategic Information decision making • Long-term • Evidence-based decisions for improving programs Impacts Improved health status

6. Performance Measurement • is the ongoing monitoring and reporting of program accomplishments, particularly progress towards pre-established goals. Source: Stephanie Shipman and Joseph Wholey. 1998. Performance Measurement and Evaluation: Definitions and Relationships

7. How to Develop a Monitoring and Evaluation Plan? Step 1. - Identify your evaluation audience. Step 2. - Define the evaluation questions. Step 3. - Identify the monitoring questions. Step 4. - Identify the indicators and data sources. Step 5. - Identify who is responsiblefor data collection and timelines. Step 6. - Identify who will evaluate the data, howit will be reported, and when. Step 7. - Reviewthe M&E plan. Source: Developing a Monitoring & Evaluation Plan. From evaluationtoolbox.net.au/index.php? option=com_content&view=article&id=20&Itemid=159

8. Characteristics of Good Monitoring Systems • Follows the reform process. • Cost-effective for the operating unit. • Describes the stages or events used for rating progress . • Describes the methodology or process used for rating the current stage of the reform. Source: Patricia Vondal. USAID. 2000. Monitoring the policy reform process

9. Characteristics of Good Monitoring Systems (cont.) • Defines key terms e.g. “operational,” “fully functioning,” or “fully implemented”. • Specifies which activities are most critical to the strategic objective. • Provides a rationale for how future performance targets are set. • Includes reform stakeholders as partners in reviewing the process of adoption and implementation and in setting future performance targets. Source: Patricia Vondal. USAID. 2000. Monitoring the policy reform process

10. Approaches to Monitoring Source: Patricia Vondal. USAID. 2000. Monitoring the policy reform process

11. 1) Simple Quantitative Approaches (SQA) • Using a purely quantitative approach is a common way of showing the progress. • Typically the agency promotes a list of reforms during a three-year period, and each year it records the number of reformsthat have been successfully completed. Source: Patricia Vondal. USAID. 2000. Monitoring the policy reform process

12. 2) Descriptive Approaches (DA) • The progress in adopting and implementing reforms is frequently not readily quantifiable. • It needs clearly define in advancethe key events in the reform process and the planned results and then monitor their achievement. • This permits a relatively objective comparison of planned with actual results throughout the performance period. Source: Patricia Vondal. USAID. 2000. Monitoring the policy reform process

13. 3) Composite Approaches (CA) (Qualitative and Quantitative Methods) • Based on constructing more complex indices. • These indices are frequently (but not necessarily) based on phases or stages identified as milestone eventsand are expressed numerically. • Each phase or stage in the reform process is assigned a number or value. • Missions should provide a detailed explanation to define each stagein a given index and to describe the related scoring scale. If not thoughtfully considered and well documented, can be imprecise and subjective. Source: Patricia Vondal. USAID. 2000. Monitoring the policy reform process

14. Monitoring • is collecting, recording, and reportinginformation concerning performance that tracking of services and programmes performances. • Routine Working Source: Chapter 10 Project Monitoring and Control. http://www.scribd.com/doc/37667340/Ch10-Project-Monitoring-Control

15. Why do we monitor? • Simply because we know that things don’t always go according to plan (no matter how much we prepare). • To better assess how well a policy or programme is achievingits intended target. • To detect and reactappropriately to deviations and changes to plans Source: Chapter 10 Project Monitoring and Control. http://www.scribd.com/doc/37667340/Ch10-Project-Monitoring-Control

16. Men (human resources) Machines Materials Money Technology Space Time Tasks Quality/Technical Performance What do we monitor? What we looking for? Progress, Schedule, Costs, Performances: Work begin, the completion Source: Chapter 10 Project Monitoring and Control. http://www.scribd.com/doc/37667340/Ch10-Project-Monitoring-Control

17. When do we monitor? • End of the Plan • Continuously • Regularly • Logically • While there is still time to react • As soon as possible • At task completion • At pre-planned decision points (milestones) Source: Chapter 10 Project Monitoring and Control. http://www.scribd.com/doc/37667340/Ch10-Project-Monitoring-Control

18. How do we monitor? • Through information collection. • Through meetings with stakeholders. • For schedule – Update Gantt Charts • Using Earned Value Analysis • Calculate Critical Ratios • Milestones • Reports • Tests, Inspections, Auditing Source: Chapter 10 Project Monitoring and Control. http://www.scribd.com/doc/37667340/Ch10-Project-Monitoring-Control

19. Evaluation • is the systematic process of the merit, value and worth of someone or something.

20. Evaluation Terms • Efficacy: Can it work? (normally for medicine) • Effectiveness : Does it work in reality? • Efficiency: Is it worth doing it, compared to other things we could do with the same money?(Cost-effectiveness) Evaluation

21. Effectiveness • Effectiveness refers to the connection between inputs and outputs, the achievement of objectives, means how well the job gets done i.e. the quality of the output.

22. Efficiency • Efficiency refers essentially to the extent to which outputs are attained while minimizing production costs, achievement of the ends with the least amount of resources. There are 2 types of Efficiency • Technical efficiency • Allocative efficiency

23. Impacts Outcomes Outputs Activities Implementation Inputs Types of Evaluation Impact Evaluation Outcome Evaluation Results Process Evaluation Cost-Benefit and Cost-Effectiveness Analyses

24. Four Stages of Evaluation (1) Define the study question and perspective Describe alternative health programs to be considered determine study perspective (decision making context of the study) (2) Identify, measure, and value costs and benefits Identify relevant costs and benefits for each program measure costs and benefits for each program in appropriate physical units value costs and benefits for each program, using market valuation of factor inputs, stated preference valuations of health gains, or other valuation methods (3) Analyse costs and benefits Adjust costs and benefits for differential timing (discounting) where appropriate perform incremental analysis of costs and benefits of alternatives perform sensitivity analysis to allow for uncertainty in estimates of costs and consequences (4) Derive decision rules Compare cost–benefit ratios , identify other decision criteriaaddress implementation issues

25. Key Performance Indicators (KPI) • Once an organization has analyzed its mission, identified all its stakeholders, and defined its goals, it needs a way to measure progress toward those goals. • Key Performance Indicators are those measurements. Source: F. John Reh, Key Performance Indicators (KPI) How an organization defines and measures progress toward its goals, management.about.com/cs/generalmanagement/a/keyperfindic.htm

26. Key Performance Indicators Reflect The Organizational Goals • Key Performance Indicators, also known as KPIor Key Success Indicators (KSI), help an organization define and measure progress toward organizational goals. Source: F. John Reh, Key Performance Indicators (KPI) How an organization defines and measures progress toward its goals, management.about.com/cs/generalmanagement/a/keyperfindic.htm

27. What is KPI? Key Performance Indicators (KPI) • “Measurable indicators that will be used to report progress that is chosen to reflect the critical success strategic” Source: John McGerty, Key Performance Indicator

28. KPI Should Be: SMART • Specific • Measurable • Acceptable • Realistic • Time frame Source: Clara Freeman. http://www.aboutfreelancewriting.com/2012/03/what-working-smarter-means-for-the-freelance-entrepreneur/

29. SMART is an abbreviation for the five conditions of good KPI’s: Specific • There has to be one widely accepted definition of the KPI to make sure the different users interpret it the same way and, as a result, come to the same and right conclusions which they can act on. Source: John McGerty, Key Performance Indicator

30. SMART is an abbreviation for the five conditions of good KPI’s: Measurable The KPI has to be measurable to define a standard, budget or norm, to make it possible to measure the actual valueand to make the actual value comparable to the budgeted value. Source: John McGerty, Key Performance Indicator

31. SMART is an abbreviation for the five conditions of good KPI’s: Achievable It is really important for the acceptance of KPI’s and Performance Management in general within the organization that it possible to achieved.Nothing is more discouraging than striving for a goal that you will never obtain. Source: John McGerty, Key Performance Indicator

32. SMART is an abbreviation for the five conditions of good KPI’s: Realistic The KPI must give more insight in the performance of the organization in obtaining its goal and strategy. Therefore the action need to happen. Source: John McGerty, Key Performance Indicator

33. SMART is an abbreviation for the five conditions of good KPI’s: Time Frame It is important to express the value of the KPI in time. The realization and standardization of the KPI therefore has to be time frame. Source: John McGerty, Key Performance Indicator

34. Organization’s KPI • A business – the percentage of income that comes from return customers. • A school – the graduation rates of students. • A Customer Service Department – the number of monthly customer complaint. • A Health Promotion Hospital – The percentage of coverage Polio vaccination in children age under 5 years old.

35. Benefits of Evaluation Finding • Important input for government decision making and prioritization, particularly in the budget process. • Assists managers by revealing the performance of ongoing activities at the project, program or sector levels. • Used to assess the performance of organizations and institutional reform processes. • Contribute to accountability mechanisms, whereby managers and governments can be held accountable for the performance of their activities. Source: Keith Mackay. 1998. Public Sector Performance-The Critical Role of Evaluation. The World Bank Washington, D.C.

36. References: • John McGerty, Key Performance Indicator • F. John Reh, Key Performance Indicators (KPI) How an organization defines and measures progress toward its goals, management.about.com/cs/generalmanagement/a/keyperfindic.htm • Clara Freeman. http://www.aboutfreelancewriting.com/2012/03/what-working-smarter-means-for-the-freelance-entrepreneur • Patricia Vondal. USAID. 2000. Monitoring the policy reform process. • Developing a Monitoring & Evaluation Plan. From evaluationtoolbox.net.au/index.php? option=com_content&view=article&id=20&Itemid=159. • Stephanie Shipman and Joseph Wholey. 1998. Performance Measurement and Evaluation: Definitions and Relationships. • Keith Mackay. 1998. Public Sector Performance-The Critical Role of Evaluation. The World Bank Washington, D.C. • Chapter 10 Project Monitoring and Control. http://www.scribd.com/doc/37667340/Ch10-Project-Monitoring-Control