Realizing the Potential of Evaluation for PBB

1. Realizing the Potential of Evaluation for PBB IEG Evaluation Week, 2011 Marc Robinson Prepared by Marc Robinson

2. The Issues How to ensure that evaluation serves budgeting • Evaluation essential to PBB • And results-based management more generally • Evaluation won’t automatically achieve its potential • Traps which must be avoided • Trade-offs between different roles of evaluation • Questions/unresolved issues identified

3. Past Experience with Evaluation • Major element of US PPBS in late 1960s • Major emphasis on program evaluation • Evaluation wave in 1970s and 1980s • Sometimes seen as disappointing • Disillusion and cut-backs to the evaluation function • Contemporary resurgence of evaluation • Including linked to budget (spending review) • Today’s challenge • Rebuild evaluation function in government • Must be seen as cost-effective tool

4. Past problems with evaluation • Relevance • Not always decision-oriented – giving managers answers • More emphasis on being ‘scientific’ • Reluctant to reach strong conclusions • Because hard to conclusively prove things • Timeliness • Evaluations which took too long to produce • “Experience shows that evaluations have often been too costly and time-consumingcompared to their real use and effect. There is also a risk of evaluations being used to slow the process of decision-making and justify inaction.” • OECD (1998) Best Practice Guidelines for Evaluation.

5. EnsuringRelevance of Evaluations • Addressing User Requirements • Not sufficient to ask “is the program effective” Need also to ask, e.g.: • How could it be made more effective? • Policy redesign? Management change? Something else? • Can the program be made effective? • Not sufficient to ask “how efficient is the program”? Need also to ask, e.g.: • How can the program be made more efficient? • What savings can be made? • Focus depends on who evaluation is for • Different users, different questions asked

6. Questions relevant to budget • PBB: evaluation mainly serves central budgeters • Ministry of Finance & other “central agencies” • Government ministers, president/prime minister • Parliament (at least where parliament has real budget power) • What central budgeters want to know: • What budgetary savings can be made? • In existing (baseline) expenditure • By eliminating ineffective programs • By eliminating inefficiency (waste) • What timeframe for realization of savings

7. Focus on budget savings • No apologies for focus on cutting baseline expenditure • Crucial to creating fiscal space for important new spending • Keeping aggregate expenditure to appropriate levels • Ensuring fiscal sustainability • Debt and deficits not excessive • Different from focus of much other evaluation • How can policy be improved? • How can delivery be improved? • Although these questions are of some relevance to the budget

8. Savings & program effectiveness • Identify programs which should be eliminated • Doesn’t mean all programs which are ineffective • Some ineffective programs can be made effective • By policy or management changes • If this is the case, shouldn’t be cut • Task is to identify inherently ineffective programs • Ineffective and cannot practically be made effective • Evaluation must therefore address both issues • Is the program ineffective? • If so, could it practically be made effective? • (Assuming that its objectives are sufficiently important)

9. Efficiency savings • Not just identifying areas of inefficiency • Three other matters need to be addressed: 1. Estimated value of potential savings • From addressing areas of inefficiency 2. Timeframe for realization of efficiency savings • One year? Several years? 3. Any support needed by agency to realize savings • Including funds for cost-saving technology • Basis for agreement with agency concerned • About adjustment to its budget baseline

10. Choosing what to evaluate • Selective approach • Choosing programs/topics which appear more relevant • For budget: selection based on potential for savings • Program which suspect are inherently ineffective • Business processes which appear inefficient • Alternative: automatic approach…. • US PART • Canadian approach • Advantages – nothing escapes; unexpected savings • Possible combination of the two approaches

11. Timeliness of Evaluation for the Budget • Evaluations timing must fit in with budget preparation • Evaluations need to be quick • E.g. commission at or before start of budget process • Results before final budget decisions taken • Rapid evaluation relevance • Compressing timeframe for impact evaluations • How quickly can they be done • Facing trade-off between rigor & timeliness • Merits of “quick and dirty” approach?

12. Relation of Evaluation to Budget Process Performance Indicators Budget-Linked Evaluations Spending Review Budget Preparation Process Time

13. Appraisal of New Spending Proposals • What about ex ante evaluation? • Appraisal of new spending proposals • Need for careful scrutiny of new spending proposals • Government-wide perspective • Central agency role – especially MOF • Mandatory processes for submission, review and debate • Requirement that proposed during the budget cycle • Minimize “urgent” out-of-cycle proposals • Who carries out scrutiny? • An evaluation unit (à la Chile)? • Sectoral policy specialists

14. How does budget evaluation relateto evaluation more generally? • Budget-focused evaluation only one part of evaluation • Other evaluations carried out for other purposes • Evaluation for policy & management improvement • Relation of this to evaluation for PBB • Same evaluations for both purposes? • Or separate evaluations for specifically budgetary purpose? • Who manages govt-wide evaluation policy? • MOF? Other central agency? • Split responsibility – depending on purpose?

15. Internal vs. external evaluation • Purely internal evaluations not sufficient for budget • Danger that these will be self-serving rather than objective • Many are detailed process evaluations • External evaluations needed • Under MOF leadership • Although ministry concerned may share direction • MOF will usually not carry out the evaluations itself • Commission evaluations from experts • Retired senior civil servants can be good evaluation leaders • Can’t necessarily expect ministry to “own” results • But that’s not the primary objective of these evaluations

16. Role of professional evaluators • Central role in respect to outcome (impact) evaluations • Role for policy experts as well • Need for appropriate collaboration • Efficiency reviews • Business process experts have major role • Particularly in reviewing specific activities • E.g. procurements, IT, human resources • Respect roles of evaluators and business process experts • Need for number of types of expertise to be applied

17. Low Income Countries • How far does all this apply to LICs? • Very tight constraints on budgets • Great human resource limitations • Relevance of Spending Review? • When resources are so limited • Cutting ineffective & inefficient spending even more important • Fiscal space for important new priorities

18. Relevance of PBB in LICs? • Prerequisites before PBB considered • Need for budget basics to be in place first • Budgeting framework which controls aggregate spending • Public money generally spent on approved purposes If these don’t exist, focus on them prior to introducing performance budgeting!! • Plain vanilla PBB for LICs • Avoid sophisticated complications • Unit-costing, indirect cost attribution, purchaser provider etc • Program budgeting • Practical performance information

19. Evaluation in Low Income Countries • Resource limitations affect • How much evaluation • Type of evaluation • Rapid evaluations should play dominant role • Because low cost • Particularly selective re outcome evaluations? • Evaluation capacity to be gradually built up • Availability of good performance indicators important • Otherwise rapid evaluations have to rely entirely on evaluation of program logic

20. Conclusions • Evaluation a vital part of performance information base of performance budgeting • Indicators alone not sufficient • But indicators important for good evaluation • Evaluation tailored to budgeting purposes • Subset of broader evaluation, including internal evaluation • Focus on informing budget decision-makers on • What programs can be cut or scaled back • Which agencies use their budgets effectively • Which agencies do not, and should generally not be given additional resources