Monitoring and Evaluation of Implementing National ICT Policies for the ICB4PAC Project

1. Monitoring and Evaluation of Implementing National ICT Policies for the ICB4PAC Project Honolulu, Hawaii 18-20 July, 2011 Day 1 Session 2-2: Output/Outcome/Impact & Key Performance Indicators Norman H. Okamura, Ph.D., Faculty Specialist Christina Higa, Associate Director Telecommunications and Information Policy Group Social Science Research Institute University of Hawaii at Manoa

2. Understanding Key Components of Policy, Program, and Project Monitoring and Evaluation STEPSOUTPUT, OUTCOME, IMPACT AND KEY PERFORMANCE INDICATORS

3. Overview of Policy, Plan and Process

4. Theoria and Praxis World Bank – Building a Results-Based Monitoring and Evaluation System

5. Program Logic Model and Implementation Implementation with Lots of Resources and Integrity of Planned Activities M&E enables corrective actions Summative Post-Hoc Implementation Could Lead to a Not Desired State Implementation Ex-Ante Future State Actual State Desired State Future State Monitoring Formative Evaluations Today Future

6. Program Logic Model and Implementation World Bank – Building a Results-Based Monitoring and Evaluation System

7. Monitoring and Evaluation Basic Steps

8. 10 Monitoring and Evaluation Steps 3 Select Key Indicators to Monitor Outcomes 5 Planning For Improvement Selecting Results Target 9 Using Findings 7 Role of Evaluations 1 Readiness Assessment 6 Monitoring for Results 4 Baseline Data on Indicators – Where are we today? 2 Agreeing on Outcomes to Monitor and Evaluate 10 Sustaining the M&E Framework 6 7 5 4 8 2 1 3 9 10 8 Reporting Findings

9. Basic Steps – Simplified Version • Formulate outcomes and goals (Completed in Plan) • Select outcome indicators to monitor (Not Completed …) • Gather baseline information on the current condition • Set specific targets to reach and dates for reaching them • Regularly collect data to assess whether targets are being met • Analyze and report the results. World Bank

10. Monitoring and Evaluation Role of Indicators (Crude and Finely Meshed)

11. How do we Measure? • Measurement is done in M&E through collecting and analyzing data on indicators! • Indicator Indicator - a number or ratio (a value on a scale of measurement) derived from a series of observed facts; can reveal relative changes as a function of time or A measurable characteristic or variable, which represents a policy/program/project progress. or An objective, which is time-limited and can be measured. Targets are set for a baseline in successive measurement over a period of time.

12. Types of Indicators in M&E Framework

13. Process, Output, Outcome, Impacts

14. To Do Monitoring …. • Description of the problem or situation before the intervention • Benchmarks for activities and immediate outputs • Data collection on inputs, activities, and immediate outputs • Systematic reporting on provision of inputs • Systematic reporting on production of outputs • Directly linked to discrete intervention (or series of interventions) • Designed to provide information on administrative, implementation, and management issues as opposed to broader development effectiveness issues

15. Elements of Results Monitoring • Baseline data to describe the problem or situation before the intervention • Indicators for outcomes • Data collection on outputs and how they contribute toward achievement of outcomes • More focus on perceptions of change among stakeholders • Systematic reporting with more qualitative and quantitative information on the progress toward outcomes • Done in conjunction with strategic partners • Captures information on success or failure of partnership strategy in achieving desired outcomes Fukuda-Parr, Lopes, and Malik 2002, p. 11.

16. 10 Steps to Results-Based M&E System • Conducting a Readiness Assessment • Agreeing on Outcomes to Monitor and Evaluate • Selecting Key Performance Indicators (KPIs) to Monitor Outcomes • Baseline Data on Indicators – Where Are We Today? • Planning for Improvement-Selecting Results Targets • Monitoring for Results • The Role of Evaluations • Reporting Findings • Using Findings • Sustaining the M&E System within the Organization World Bank – Building a Results-Based Monitoring and Evaluation System

17. Monitoring and Evaluation Indicators and Measurement (Crude and Finely Meshed)

18. How do we Measure? • Measurement is done in M&E through collecting and analyzing data on indicators! • Indicator Indicator - a number or ratio (a value on a scale of measurement) derived from a series of observed facts; can reveal relative changes as a function of time or A measurable characteristic or variable, which represents a policy/program/project progress. or An objective, which is time-limited and can be measured. Targets are set for a baseline in successive measurement over a period of time.

19. Defining ICT Indicators/Measurements Plane of Theory Goal or Objective Affordable Phone Key Performance Indicator Plane of Concept Pop w Phones Price Plane of Measurement # of Phones Popu- lation $

20. Representation of World Space Cartesian Representation of X and Y Variables (World Space) The “world” is classified and defined. Y s y _ y _ t X x x Crudely and Finely Meshed

21. Program Logic Model and Implementation The Plan in the Plane of Theory Computers in Libraries will improve student achievement Resources Budgets Personnel Work Effort Computer in Schools No Change in Student Performance Conceptual Indicators $50,000 .5 FTE Procurement Installation 50 Computers in Library Level Examinations Plane of Measurement FTE = Full Time Equivalent

22. SMART Indicators Specific Measurable Achievable Relevant Time-bound

23. CREAM Indicators Clear Precise and unambiguous Relevant Appropriate to the subject at hand Economic Available at a reasonable cost Adequate Provide a sufficient basis to assess performance Monitorable Amenable to independent validation

24. CREAM and SMART Indicators Clear Precise and unambiguous Relevant Appropriate to the subject at hand Economic Available at a reasonable cost Adequate Provide a sufficient basis to assess performance Monitorable Amenable to independent validation Specific Measurable Achievable Relevant Time-bound

25. Indicators • The choice of core indicators in ICT4D is the key to determining the outcomes and impact of planned national policies, programs, projects on development. • In order to understand the outputs of any program, inputs must also be measured, such as ICT resources. • Outputs should be measured against these same variables as well as costs. • Data should be collected throughout the program’s implementation, and in sufficient breadth and depth such that conclusions have credibility with the consumer of the study.

26. Indicators • Indicators need to be context-specific; • Ideally indicators assess the direct issue. Most of the time, however, an indirect indicator is more feasible and therefore more reliably monitored; • Indicators are expressions of the values of those who chose the indicator. Both the selection and acceptance of an indicator depends on values; • Indicators often work best and sometimes only in combination - a single indicator does not necessarily tell you enough, and; • Indicators are transitory or sometimes seasonal – they need to be reviewed and adjusted;

27. Indicators • Over the course of the program, conditions change, objectives are altered or better indicators [may be] discovered. In these instances, it may be appropriate to change the indicators you are monitoring. Adapted from Wieman3; Sander4.

28. Key Indicators in National ICT? • What are some Key Indicators in National ICT4D? UNDP, UNESCO

29. Indicators in National ICT and Telecommunications Plans • Presence of a national ICT policy • Presence of a master plan with a timeline • National expenditure on ICT • Ratio of students to computers • Integration and availability of computer networks in schools, libraries, health care organizations • Telecommunications used more by people • Telecommunications availability to people • Percentage of households with: electricity, radio, television, computer, Internet access • Percentage of population that: is covered by mobile access, uses a computer, has Internet access UNDP, UNESCO

30. Monitoring and Evaluation Wow! That’s a lot of things we can look at!

31. Monitoring and Evaluation Data and Data Collection

32. Data • What are the sources of data? • What are the data collection methods? • Who will collect the data? • How often will the data be collected? • What is the cost and difficulty to collect the data? • Who will analyze the data? • Who will report the data? • Who will use the data? World Bank – Building a Results-Based Monitoring and Evaluation System

33. National ICT Data • What mechanism and authority exists within a country to routinely collect data regarding the progress of National ICT planning? • Do the regulatory agencies within the country have the authority to collect data from telecommunications carriers? • Could the National ICT planners requests that the regulatory agency collect information on core indicators? • Does the Pacific island country have the resources to conduct annual meetings to collect data and other information regarding the implementation of the National ICT Plan?

34. National ICT Data • Formulate outcomes and goals • Select outcome indicators to monitor • Gather baseline information on the current condition • Set specific targets to reach and dates for reaching them • Regularly collect data to assess whether the targets are being met • Analyze and report the results.

35. Monitoring and Evaluation Data Issues

36. Monitoring and Evaluation Application to National ICT Plans

37. Transforming Goals/Objectives into Outcomes • What does a strong outcome statement look like? • Expresses what change will look like after a period of time • Describes the end-state not a process or ver • Is time-bound • Contains one idea and not many • Is not expressed as a indicator or include a “target” • Should allow for measurement • Should not overlap with other outcomes World Bank

38. Outcome: weak * strong examples Weak: • To enable Internet access for the schools • To obtain computers for students • To …. (Process) Strong: • Each Village school has 5 Mbps of Internet Access with an QoS of 95% throughput to Australia • Each student shall have a laptop or netbook in the school • Result World Bank

39. For Each Goal or Objective • Develop Outcomes • Performance SMART and CREAM Indicators for each • Establish the Targets • Develop Evaluation Questions • Assess the Assumptions and Risks World Bank

40. Outcomes – Contentious Point • There are other outcomes that are also important not just intended outcome • e.g. Internet access provides access to pornography • We don’t always know what the outcome, change and indicators will look like at the start • e.g. Telecommunications in the States – Called great experiments for U.S. • Development is dynamic and interventions must be flexible • What is important can not always be measured World Bank

41. Example of a Strong Outcome Statement for National ICT Plans • Prepare a draft terms of reference – purpose, objectives/questions, and methods • Put together the evaluation team • Brief and support the team before and during fieldwork • Critically assess the draft report World Bank

42. Key Steps in Evaluation • Prepare a draft terms of reference – purpose, objectives/questions, and methods • Put together the evaluation team • Brief and support the team before and during fieldwork • Critically assess the draft report World Bank

43. Key Steps in Evaluation • Prepare a draft terms of reference – purpose, objectives/questions, and methods • Put together the evaluation team • Brief and support the team before and during fieldwork • Critically assess the draft report World Bank

44. Key Steps in Evaluation • Prepare a draft terms of reference – purpose, objectives/questions, and methods • Put together the evaluation team • Brief and support the team before and during fieldwork • Critically assess the draft report World Bank

45. Build an Example of a Strong Outcome Statement National ICT Plan

46. Measuring Outcomes:Monitoring or Evaluation? • If outcomes are important and worthwhile, the preference should be for monitoring over evaluation • Only evaluate when the cost and resource required for systematic monitoring is too high • What is evaluated should from the intended outcomes and the monitoring and evaluation f/work • A strong Framework will also identify the key research questions for evaluation World Bank

47. Be Mindful • Organizational (management and governance) commitment is essential • M&E is a tool, not the answer • M&E works best when it is used to support conversations and not as hard and fast ‘rules’ • Information collection (resources) should be scaled to importance • Focus on measuring the achievement and trust has been demonstrated • Consulting and involving stakeholders will improve the “reality” of the final product, and help buy-in World Bank

48. Large Scale Evaluations! • Prepare a draft terms of reference – purpose, objectives/questions, and methods • Put together the evaluation team • Brief and support the team before and during fieldwork • Critically assess the draft report World Bank

49. Key Messages During the Planning Phase • Allow sufficient time to include stakeholders • Allow sufficient time for appropriate methods in the field, including participatory approaches • If possible agree the key questions for the evaluation when designing the M&E F/work • Use monitoring information and past evaluations to help answer the evaluation questions • Consider and agree involvement (who) and ownership together – sharing power World Bank

50. Defining the Purpose • This process takes time….. • May require several rounds of consultation involving stakeholders • Formulate the purpose in a way that indicates how information provided will be used • If difficult, may signal evaluation not sufficiently useful to justify investment; consider postponing or whether needed at all • Timing is important – if evaluation can not be completed before is to be used to inform decisions; redesign or reconsider World Bank