Evaluation capacity building: the international experience

1. Evaluation capacity building:the international experience Scott Bayley, AusAID

2. Overview • What is evaluation capacity building (ECB)? • Why do it? • Strategies for ECB • International examples • What works? – guiding principles

3. What is ECB? • The UNDP (2002) defines capacity as the ability of people, institutions and societies to perform functions, solve problems, and achieve their goals. • ECB refers to activities and initiatives taken to mainstream a regime that supports the ongoing production and use of evaluations.

4. Why do it? • Governments and organizations build evaluation systems because they believe such systems will help them to improve their means and methods of governance. • In particular …

5. Why do it? ... • … ECB can be used to support: • Planning & policy making • Program management • Resource allocations, budgeting • Government control, coordination • External reporting & accountability • Participation by civil society.

6. What is successful ECB? • The production of appropriate quality evaluations • A high level of utilization of evaluation findings • Country/agency ownership • Sustainability over time as governments & officials change.

7. Strategies for ECB 1. Whose capacity and for what purpose? • Legislatures • Federal/state governments • Sectors, eg. health, education, water, agriculture, justice • Agencies that commission and fund evaluations

8. strategies for ECB … • Individual evaluation practitioners and networks • Those who use the results of evaluations to help guide their policy making and management activities, govt agencies, donors, NGOs

9. strategies for ECB … • Groups affected by the programs being evaluated • General public, the academic community, civil society. Whose capacity and for what purpose?

10. strategies for ECB … 2. Diagnostics • current practices • politics of reform • desired functioning

11. strategies for ECB … Current practices -current policies and systems, existing M&E activities, institutional capacities, linkages to planning and budgeting

12. strategies for ECB … Politics of reform -drivers for change, supporters and opponents of reform, resources, incentives

13. strategies for ECB … Desired functioning -intended users and uses, supply and demand factors, supporting institutional infrastructure

14. 2. Diagnostics X

15. strategies for ECB … Supply side 3. Tactics • Overseas fact finding tours • Training workshops for practitioners • Production of policy and operational manuals, evaluation plans • Legislation/regulation

16. sequencing ECB … •  Demand for and ability to use evaluative feedback, then • Supporting institutional infrastructure & • Supply side considerations • - Guiding principles (ECDG group):ownership; relevance; integration; usefulness.

17. strategies for ECB … • Identify and work with powerful champions of reform • Link ECB to other significant reforms • Actively manage the change process- ECB develops over time and in stages

18. strategies for ECB … • Incentives: • the example set by leaders, rewards, education and information, practical support • Incentives to increase demand and use • Incentives to reduce perceived costs and increase benefits within agencies (eg. guarantee of no job losses or budget cuts)

19. strategies for ECB … • Training to raise awareness of those who might demand evaluations to enhance service quality i.e. internal senior managers; external bodies such as NGOs, consumer advocates, media • L/T partnerships and joint evaluations for skill transfer/learning

20. strategies for ECB … • Train the trainer (to provide on the job training to practitioners) • Hold managers accountable for learning/making improvements • Capitalize on existing demand and windows of opportunity • Link evaluation into systems for policy making, planning, budgeting, reporting, etc.

21. International examples First wave, 60s & 70s Second wave, 80s & 90s Third wave 90s+

22. What works? – guiding principles 1. ECB is a political activity with technical implications. • and not vice versa! • ECB is about organizational change, link it to other significant reforms • ECB creates winners and losers- which means supporters and opponents.

23. principles… 2. Start with good diagnostics. • local context and history matter! • aim to build upon local strengths and interests; target functional needs rather than ‘correcting’ deficiencies.

24. principles… 3. Building and capitalizing on demand is the key. • seek to match the supply of evaluations to current demand, build demand (and capacity to use eval info) over time • attempting to force the creation of an evaluation culture through legislation or a supply side push simply doesn’t work • a limited availability of internal evaluation skills is not a fundamental constraint, use contractors if need be.

25. principles… 4. Institutionalize evaluation in a way that aligns supply to demand. • the evaluation function needs to be located (or anchored) where the demand and users are • we have a matrix arrangement to support accountability in government. Evaluation would benefit from a similar approach.

26. principles… 5.Learn from others but avoid best practice models. • different countries/agencies have different starting points, aspirations, constraints & opportunities • search for adaptable ideas, not blueprints for reform • Cautious evolving experimentation is the way to go (along with M&E of ECB activities)-ECB is not a linear process.

27. principles… 6.ECB is a long term process • 5 to 10 years for agencies in more developed countries • 10 to 20 years in developing countries • political and organizational interest in evaluation is cyclical-ECB is ongoing, not a “one time” event.

28. Summary • What is evaluation capacity building (ECB)? • Why do it? • Strategies for ECB • International examples • What works? – guiding principles

29. Questions & Discussion

30. For further information: scottbayley56@yahoo.com.au * Abonyi, George. 2002, Toward a Political Economy Approach to Policy-based Lending. ADB (on the web). * Boyle, Richard, and Donald Lemaire, eds. 1999, Building Evaluation Capacity. New Jersey: Transaction Publishers. * ECDG, 2011, International workshop agreement on evaluation capacity development (on the web). * Lahey, Robert. A Framework for Developing an Effective Monitoring and Evaluation System in the Public Sector (on the web). * Mackay, Keith. 2007. How to Build M&E Systems to Support Better Government. World Bank (on the web).