Impact Evaluation of Decentralization in Colombia (1991-2001)

1. Decentralization Thematic Group/Public Sector Group Funding 2002-2003 “Assessment of the Impact of Decentralization: The Case of Colombia (1991-2001)” Presentation by: Jonas Frank (LCSPS) March 10, 2004

2. The Puzzle: What is the impact of decentralization and can this impact be evaluated? • Better Services? • Economic Efficiency? • Democratic Government?  Evaluation is about finding causality

3. The Colombia Case: • The Study: • Sample of 148 municipalities, out of a total of 1100 • 70 indicators • 1000 pages • It took ca. 2 years to complete the evaluation • Appropriate time-frame: process started in 1986, evaluation covers years following 1991 • Integral reform: political, fiscal, administrative • Typical sequence: political reform first, then fiscal and administrative decentralization; municipal level

4. Three important questions: • Why evaluate? • What should be evaluated? • How can one evaluate?

5. I. The “Why” of Decentralization Evaluation • Objectives of Evaluation differ by actors of decentralization: • Minister of Finance, Indigenous People, Legislators, Mayors, Regional Governors, Donors,... • There is no single objective, but evaluations can... • (i) improve decision-making • (ii) create accountability among actors involved

6. I. The “Why” of Decentralization Evaluation • Implications: • Evaluation is a continuous exercise • Transformation of data into information • A participatory exercise vs. purely technocratic

7. I. The “Why” of Decentralization Evaluation Objective of the Colombia Study: • Crisis: fiscal pressures, weak budget constraints, bailouts • Inefficient services • Slow democratization Lesson: • Evaluate as early as possible • Ownership • Too many objectives of evaluation

8. I. The “What” of Decentralization Evaluation Some guiding questions: • Where is decentralization supposed to have an impact? • Were the initial goals of decentralization met and to what extent? Dilemas: • Decentralization goals remain vague • They are not expressively formulated and agreed upon • Decentralization is a “moving target”

9. I. The “What” of Decentralization Evaluation • Decentralization Goals in Colombia: • In 1986: democratization • In 1991: efficiency in services, redistribution, services for the poor, improve popular participation • In 2001: fiscal discipline, efficiency

10. LAC: Decentralization Objectives (1970 – 2002) Venezuela Rio Grande ARG Brazil Ecuador Chile Colombia Peru Bolivia

11. II. The “What” of Decentralization Evaluation Colombia Study: “Comprehensive evaluation” (not only fiscal): 1. Outcomes within eight sectors • Education • Health • Water and basic sanitation • Fiscal performance • Political decentralization • Management capability • Economic development 2. Municipal Progress: (i) sectoral outcome (ii) aggregate outcome,

12. II. The “What” of Decentralization Evaluation 1. Sectoral Evaluation: Results in Education • “Decentralization has improved and contributed to: • Increase in coverage • Improved teacher/student ratio • Higher schooling levels • Reduction in illiteracy rate”

13. Results in Education

14. Sectoral Evaluation: Results in Education

15. II. The “What” of Decentralization Evaluation 1. Sectoral Evaluation: Results in Health • “Decentralization has improved and contributed to: • Increased coverage • Higher public spending • Greater equity • Lower infant mortality rates • Greater ratio of physicians per inhabitant”.

16. Sectoral Evaluation: Results in Health

17. II. The “What” of Decentralization Evaluation 2. Performance Evaluation of Municipalities • (i) Municipal Performance in Six Sectors (Health, Education,...) • (ii) Aggregate Performance: Most municipalities have achieved an average performance; only 4% achieved and acceptable standard • But: • What were the starting conditions? • Where there several observations in the past and when?

18. Results in Economic Development

19. II. The “What” of Decentralization Evaluation Lessons: • Some important areas are excluded: • Allocative efficiency • The regional level: departmental government • Conclusions about the causality of decentralization are primarily hypothetical • “Less is more”

20. III. The “How” of Decentralization Evaluation • 1. Base-line of evaluation • 2. Finding causality • 3. Indicators • 4. Selection of samples of local government • 5. Periodicity of evaluation

21. III. The “How” of Decentralization Evaluation 1. Base-line • Usually no such base-line has been established before decentralization was implemented •  The use of proxies •  Select indicators that are available and mirror the starting conditions

22. III. The “How” of Decentralization Evaluation 2. Finding Causality To date, impact studies of decentralization are either: • (i) Evidence-based • (ii) Subjective • (iii) Hypothetical All of these three types of studies provide a useful understanding and appraisal at various stages of the decentralization process.

23. III. The “How” of Decentralization Evaluation 2. Finding Causality • The Colombia study is primarily hypothetical • Statistical analysis is not sufficient to prove causality • Lessons: • Discerning between decisions that are in hands of local government and which are not • Clarifying the value added of local government in carrying out new responsibilities? • Continuous monitoring

24. III. The “How” of Decentralization Evaluation 2. Finding Causality: Example of the Education sector • Local government defines curriculum content: • ”Instruction Indicator” • Local government is responsible for construction and maintenance of school buildings •  “Infrastructure Indicator” • Local government develops education improvement plan •  “Planning Indicator” • Local government allocates budget •  “Budget Indicator” • Is this operationally possible?

25. III. The “How” of Decentralization Evaluation 3. Indicators • 70 indicators were used in the Colombia study • Lesson: • Selection of indicators depends on “what” will be used, but: • use only a minimum set of indicators, and... • indicators that can be monitored regularly and with low cost

26. III. The “How” of Decentralization Evaluation 4. Comparability of data and selection of local government • Colombia: “representative sample” = most diverse municipalities were selected • Lesson: • Use only most similar cases because diversity increases during decentralization process • Split them into groups: eliminate some structural factors that lie outside control of municipalities (population size)

27. III. The “How” of Decentralization Evaluation 5. Periodicity of evaluation • Colombia: evaluation after 10 years of time • Lesson: • Time period sufficiently large, but... • Regular monitoring is necessary • It is important to separate between short-term, medium-term, and long-term goals •  helps create accountability •  helps better decision-making

28. Conclusion: Risks and Opportunities • Evaluation as a purely bureaucratic exercise • (i) Evidence and (ii) subjective evaluation are still important: the only “early-warning system” that is operational • Ready to propose and implement corrective measures: “exit” of decentralization process? • Giving “erroneous” answers to the “wrong” questions: “Decentralization has not worked well enough because there was not enough of it”