1 / 22

What can we learn from successful country systems? Philipp Krause PRMPR

What can we learn from successful country systems? Philipp Krause PRMPR. Overview. Why bother? A selection of country cases How do differences matter? What do “successful” systems have in common?. Why bother?. Two schools of thought The “best practice” school

ekram
Download Presentation

What can we learn from successful country systems? Philipp Krause PRMPR

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. What can we learn from successful country systems?Philipp KrausePRMPR

  2. Overview • Why bother? • A selection of country cases • How do differences matter? • What do “successful” systems have in common?

  3. Why bother? • Two schools of thought • The “best practice” school • The “It’s all about the context” school • The truth lies somewhere in between – there are important characteristics shared by successful cases, but differences between countries matter tremendously • Most important reason: Countries do M&E for different reasons and different users – what you need depends on what you want to do with it • Our definition of success: • High degree of utilization • M&E information meets quality standards and is reliable • System is sustainable over time

  4. Overview • Why bother? • A selection of country cases • How do differences matter? • What do “successful” systems have in common?

  5. Australia: Main Features • Mandate to evaluate each program every 3-5 years • Portfolio Evaluation Plans to be prepared annually for the following three years • Department of Finance: Steering & quality control • Sector Departments: planning, implementation • Evaluation results primarily used for budgetary decisions: allocations of funds for new policies and reallocation of savings (i.e. the discretionary part of annual budget process) • System lasted from 1987-1996 – sustainability?

  6. Australia: How does it work? Inputs to Portfolio Evaluation Plans (PEPs) Department of Finance Citizens Publication of department’s evaluation reports Formal notification of PEPs Sector Departments (and outrider agencies) Involvement in evaluations Department of Prime Minister and Cabinet Inputs to PEPs Parliament Reporting of key Evaluation findings in department’s budget paper (prospective) and in their annual reports (retrospective) Inputs to PEPs Treasury Source: Mackay 2011

  7. United Kingdom: Main Features • Comprehensive system of performance monitoring and targets: Spending Reviews (multiannual budgets) and Public Service Agreements (“resources for delivery”) in return for (relative) managerial flexibility • Oversight and leadership from central executive (Prime Minister, PM’s Delivery Unit, Treasury) • No systematic evaluations in the executive government • Problem: “Gaming in Targetworld” • Value-for-Money Audits: 60 per year from National Audit Office

  8. UK: How does it work? HM Treasury Prime Minister’s Office is accountable to Delivery Unit* Parliament Negotiations over PSAs reporting/monitoring monitors priority areas Ministers reports to Spending Unit National Audit Office is accountable to audits * Moved to HM Treasury in 2007, abolished 2010

  9. Mexico: Main Features • M&E of social policies delegated to a specialized technical agency – CONEVAL • CONEVAL in charge of evaluation portfolio, development of methods, dissemination • Key oversight decisions by inter-ministerial committees – system involves many stakeholders (Congress, Finance, Presidency, Public Admin Ministry, Sector Ministries, CONEVAL) • Implementation of evaluation and day-to-day operation of monitoring done by ministries

  10. Mexico: How does it work? • Define the evaluation plan • Decide what and how to evaluate 1. Inter-ministerial Committee • Coordinate and steer evaluation system • Maintain quality control • Define monitoring parameters 2. CONEVAL • Select and hire evaluators • Supervise evaluation implementation • Monitor indicators and targets 3. Sector Ministry Sector Ministry Sector Ministry Programs are evaluated by ministries • Collection of evaluation results • Dissemination of evaluation results 4. CONEVAL Congress, Presidency, Sector Ministries, Finance • Utilization of evaluation results 5.

  11. Chile: Main Features • System is highly centralized – closely linked to and based on a highly centralized, top-down budget process • Budget office manages most details, and is main user of information – utilization fostered by close link to budget process • Management control by hierarchical oversight • Little buy-in from other (potential) stakeholders • Some impact on allocations and program management Percentage of all evaluated programs. Source: Guzman 2007

  12. Chile: How does it work? 12 • Define the evaluation plan • Decide what and how to evaluate 1. Budget Office • Coordinate and steer evaluation system • Maintain quality control • Define monitoring parameters 2. Budget Office Budget Office • Select and hire evaluators • Supervise evaluation implementation • Monitor indicators and targets Programs 3. Programs are evaluated directly by Budget Office • Collection of evaluation results • Dissemination of evaluation results Budget Office 4. Budget Office • Utilization of evaluation results 5. Congress

  13. Centralization vs. Delegation Mexico Chile • Define the evaluation plan • Decide what and how to evaluate 1. Budget Office Inter-ministerial Committee • Coordinate and steer evaluation system • Maintain quality control • Define monitoring parameters 2. Budget Office CONEVAL • Select and hire evaluators • Supervise evaluation implementation • Monitor indicators and targets Budget Office Programs 3. Sector Ministry Sector Ministry Sector Ministry Programs are evaluated directly by Budget Office Programs are evaluated by ministries • Collection of evaluation results • Dissemination of evaluation results Budget Office 4. CONEVAL Budget Office Congress, Presidency, Sector Ministries, Finance • Utilization of evaluation results 5. Congress

  14. Overview • Why bother? • A selection of country cases • How do differences matter? • What do “successful” systems have in common?

  15. Institutional Differences Matter: Index of Legislative Budgetary Powers Chile Mexico Source: Wehner 2007

  16. Difference in Purpose • Budgetary: To inform budgetary decisions, best allocation of resources between sectors and programs, also to enforce operational savings in annual budget • Accountability: External accountability towards legislature, stakeholders and public. To make systematic information on performance available and strengthen the public evidence base of policy decisions • Control: To develop better central government information on implementation and service delivery of public programs as tools to hold managers to account

  17. Matrix of Country Comparison * Supreme Auditor

  18. Features that Matter • Centralization requires the right institutional structure – a centralized M&E system design in a fragmented public sector will fail • In a system with multiple stakeholders delegation to an impartial agency might be a viable option – but beware of over-engineering and objective overload • Who is to gain and who has to worry about buying into a M&E system: The senior civil service, the legislature, ministries, service delivery units, the finance ministry, the head of government (PM or President)? • Staying in control of overall steering and quality of outputs does not equal having to internalize all aspects of M&E implementation – strategic delegation might be smart for buy-in and workload • Is it possible to imagine a long-term sustainable, well utilized M&E system that does not have a stable link to budgetary decisions?

  19. Overview • Why bother? • A selection of country cases • How do differences matter? • What do “successful” systems have in common?

  20. Lessons Start M&E Systems Successfully* (1) • Somewhere in government is substantive demand for M&E information. This is necessary to start and sustain an M&E system • Actors need to have incentives to engage with M&E. They are key for M&E to be conducted and for the information produced to be utilized • Simple is better – successful M&E systems tend to deliver just what users want, not more. They also serve only those objectives that result in utilization • Success is more likely with a powerful champion(s) to lead the push for institutionalization of M&E – rather than a legislative or technical exercise *very liberally adapted from Mackay 2010

  21. Lessons Start M&E Systems Successfully (2) • It is important to have the stewardship of a central, capable ministry that can design, develop, and manage the system • Some reforms may start with a bang, but it requires patience, determination, and a long-term effort to build an effective M&E system • For donors: It helps the process to start with a diagnosis of what M&E functions already exist in the country (and why other M&E functions do not exist – they usually don’t for a reason)

  22. THANKS! For further information, and to access the sources cited here, please visit: http://go.worldbank.org/7MZRWD6K50

More Related