The Challenges of Country Monitoring & Evaluation Governing For Results in China

1. The Challenges of Country Monitoring & Evaluation Governing For Results in China October 25, 2006 Harry P. Hatry The Urban Institute Washington, DC, USA

2. The Range of Evaluation Approaches 1. Continuous Performance Monitoring 2. Quick Response Assessments 3. In-depth Implementation Evaluations With Early-Outcomes 4. Statistically-Identified Comparison Groups 5. Randomly Assigned “Experimental” and “Control” Groups

3. In-Depth Evaluations Performance Monitoring Cost High Considerably Lower Coverage Very limited-few programs Wide-most programs Timeliness Poor for management Good, depending on frequency of reporting Validity of Information Much better for identifying causes and problems Very limited for identifying causes Pros and Cons of In-depth Evaluations versus Regular Performance Monitoring

4. Overall Relation of Monitoring and Evaluation (M&E) to Planning, Budgeting, and Operations • M&E provide information about the past. • Planning, budgeting, and operating management are primarily about the future! • The M&E process provides past data for planning, budgeting, and operating management – to help estimate future results likely from budgeted resources. • When in-depth evaluation information is available, it usually provides better information than data from the agency’s performance measurement process.

5. Relation of Monitoring and Evaluation (M&E) to Planning • The five-year planning process should include, and focus on Key Performance Indicators –relating to the Development Goals. • Plans should include targets, preferably annual targets for each indicator. • The M&E process should provide annual data on these indicators. • Investment strategies should be modified based on annual findings from monitoring performance . • Targets should be based, in part, on past time trends identified by the M&E process. • Progress should be reviewed, annually using data from the M&E process.

6. Relation of Monitoring and Evaluation (M&E) to Budgeting • Agencies should include Key Performance Indicators in capital and operating budget requests. • Budgets should include data on one or more past years’ performance and targets for each performance indicator. • The targets should be based on recent time trends identified from the M&E process. • Analysts should examine the targets against the resources requested in the budget request for reasonableness. • Capital and Operating budget strategies should be modified based on the findings.

7. Relation of Monitoring and Evaluation (M&E) to Operating Agencies • Each agency needs to establish its own M&E process and provide data on needed performance indicators -- including Key Performance Indicators in the five-year plan and those wanted by the MOF Budget Office. • The M&E process should provide timely data for agency managers. • Agency managers should revise service delivery based on M&E findings. • Agency managers should use M&E data to help prepare their operating and capital budget requests. • Agency managers should include outcome targets in budget requests -- based on M&E data.

8. Basic Performance Data Needs for M&E Inputs:Funds and Staffing used Outputs: Work completed, such as people served; inspections made Outcomes: Effects on citizens or businesses, such as unemployment or pollution • Intermediate Outcomes, such as unemployed persons satisfactorily completing an employment-training program • End Outcomes, such as number of unemployed persons completing an employment- training program and who became employed Efficiency: Cost per unit of product • Cost Per Unit of Output, such as cost per kilometer of road rehabilitated • Cost Per Unit of Outcome, such as cost per kilometer of road in satisfactory condition

9. IntermediateOutcomes End Outcomes Timing Occur early and before end outcomes Occur after intermediate outcomes Extent of Agency Influence Usually considerable Many outside factors can affect them Ease of Estimating Values (past or future) Relatively easy Some end outcomes require new data collection procedures Importance to Citizens Usually considerably less Very important to them Include and Track Both “Intermediate” Outcomes and “End” Outcomes Comparison table:

10. Use “Results Chains” (“Outcome Sequence Charts” or “Logic Models.”) To Help Identify Intermediate and End Outcomes and To Help Communicate What Programs and Services are Intended to Do

11. Output - Outcome Sequence Chart Venezuela Project for Water and Sanitation in Urban and Rural Zones Intermediate End Outputs Outcomes Outcomes Comp. 1. Institutional More pe ople have Water supply systems Strengthening rehabilitated / access to piped Urban Comp. 2. water supply expanded Water Supply / services Sanitation Increased Improved Infrastructure Access to Citizen Comp. 3. Rural Potable Health Water Supply / Water Sanitation Infrastructure with Com munity Participation Comp. 4. Operational & Fewer and Improved Commercial shorter capability of utility Efficiency interruptions in operating Comp. 5. supply personnel Project Management

13. Small and Medium Business SupportService Outcome Sequence Chart They Request and Receive Substantial Assistance They Take Action Based on Assistance More Earnings Existing Business Seek Help Training and Technical Assistance are Provided Persons Seek Help to Start a Business They Request and Receive Substantial Assistance They Take Action to Start a Business The Business is Reasonably Successful The Business is Started Jobs Created, Especially for Low Income Households Improved Quality of Life Reduced Poverty Jobs Created, Especially for Low Income Households

14. Coordinate Cross-Cutting Programs • Prepare table showing which agencies, which organizations contribute to each outcome indicator. For example: Outcome Indicator #59. Reduction of main pollutants Contributing Organizations EPA, MWR, MOH, MWR, MOA • Arrange for communication, coordination, and cooperative efforts. • Establish “Performance Partnerships.” Examples: U.S. EPA partnerships with state governments. U.S. Office of National Drug Control Policy

15. Key Performance Partnerships Elements • Include all relevant organizations (including other ministries and local government agencies). • Identify outcomes sought and outcome indicators to track progress. • Establish out-year targets. • Decide who needs to do what, by when, and with what funding to accomplish those targets. • Decide who needs to do what, and how frequently to track and report progress on those outcome indicators. • Track progress. • Make adjustments.

16. Analyzing Outcome Information -- I • Make basic comparisons. Compare latest outcomes: •  To past performance. • To targets • Across geographical areas (such as different regions, counties, or cities). • Among customer groups with other demographic characteristics (such as age, gender, and income groups) • Among organization units providing similar services (such as different facilities, offices, parks, hospitals, schools). • To different service approaches. • To those of other countries.

17. Analyzing Outcome Information -- II • For performance indicators where agencies have not met their targets, require them to provide meaningful explanations. • Example: Some state governments in United States require agencies to provide explanations if the latest indicator values are not within 5% of the target.

18. Major Recommendations-I • Use evaluation (outcome) information both in developing capital investments and operating programs. • Include in multi-year strategic plans key outcome indicators and time-phased targets for each. • Have a good cost accounting system so you can link expenditures for each significant program to the outcomes. • Ask each Ministry to establish its own M&E process. • Ask Ministries to group their performance indicators into: outputs; intermediate outcomes; and end outcomes.

19. Major Recommendations-II • Ask each Ministry and agency to provide a “Results Chain” for each program. • Prepare annual evaluation agendas. Plan evaluations so the findings become available when decisions are expected to be needed. • Provide technical assistance and training in using M&E for managers and professional staffs in planning, budgeting, and operating agencies. • Assist local governments in performance measurement. • Establish performance partnerships across agencies and governments.