The Value of Disciplinary Diversity Evaluating Community-Driven Development

1. The Value of Disciplinary DiversityEvaluating Community-Driven Development Vijayendra Rao, World Bank

2. OUTLINE • Some Limitations of impact evaluations. • Some Concerns about randomization. • Alternatives and Complements • Added Value of Qualitative Methods. • Evidence for Community Driven Development (CDD) • Open Questions About CDD • Methodology of Indonesia CDD Evaluation

3. General Concerns • Question Should Drive Method And Not Method The Question • Intervention versus Nothing. • Internal versus External Validity • Impact versus Process

4. Impact Versus Process • “Process”: What were the series of events instigated by the intervention that led to the outcome? (e.g. local politics, interactions between govt. official and school headmaster). • Cannot be easily anticipated enough to fit into a structured questionnaire. • What is the real “impact.” Positive and Negative Externalities. (e.g. project to improve condom use leads to empowerment of prostitutes)

5. Internal Versus External Validity • Contextual differences may be missed. (Role of culture, social structure, politics and geography in determining impact.) • Differences in Implementation (Learning by Doing) • Was there any spill-over, and to what extent? How did this happen? • Rubber Hits the Road (Scaling-Up)

6. Additional Concerns About Randomized Trials Solves the Problem of Selection Bias BUT General Concerns Can Be Accentuated: - Randomization Bias, Hawthorne Effects • Experimental versus real world • Political Constraints (Short term political gains override scientific concerns) • Ethical Concerns

7. Imperfect Solutions Substitutes and Complements

8. Non-Experimental Methods • Propensity Score Matching (Depends on exhaustive set of observed indicators) • Discontinuity (Super-local validity) • IV (difficult to find)

9. Monitoring As Evaluation • Complementary and Essential • MIS Data Bases • Pilots within an Intervention (E.g. Olken) • Facilitators as Key Informants • Helps Process of Learning by Doing

10. Qualitative Methods • Complementary but Adds to Cost • Participant Observation • Focus Group Discussions/PRA • In-Depth Interviews (Key Informants) • Textual Analysis (e.g. Local Newspapers)

11. Added Value of Qualitative • Think Quantitative act Qualitatively • Help Choose Identification Strategy (e.g. Sex Workers Study) • Help Measure Outcomes (e.g. CDD evaluation in Indonesia) • Help Track Processes (e.g. NREGA Study, CDD Evaluation Indonesia). • Triangulation

12. Community Driven Development (CDD) • Directly Give Community Access to Untied Funds • Community Based Targeting • Community Based Management • $ 7 Billion at the World Bank

13. Well Targeted Improves Supply and Quality of Public Services Improves Capacity for Collective Action – “Social Capital” More Sustainable Risk of Elite Capture is Low Can be Scaled Up Some Claims of CDD

14. Evidence to Date

15. Poverty Targeting • Easier to Target Poor Communities than Poor within Communities • Generally Speaking – CBT works better than external methods • Heterogeneity Matters • “Preference Targeting” Poor

16. Service Delivery • Community Involvement generally seems to produce effective projects . • No Evidence that it Causes Projects to Improve • No evidence of whether non-participatory alternatives work better.

17. Participation • Lots of evidence that high levels of participation are positively correlated with project effectiveness. • NO CONVINCING EVIDENCE OF CAUSAL LINK

18. Social Capital Creation • Lots of Evidence on Strong Correlation • But Very Little Convincing Evidence on Causal Links

19. Sustainability • AGAIN LIMITED EVIDENCE • Anthropological work points to crucial role of support from higher levels of government.

20. Elite Capture • Social Networks affect who benefits • Generally speaking – elites tend to dominate • Benevolent vs Malevolent Capture

21. Role of External Agents • Central to Local Level Project Effectiveness -but understudied • Good Facilitators are charismatic leaders, trainers, anthropologists, engineers, economists, and accountants

22. Scaling-Up Challenges • Low Experience, Poor Training of facilitators. • Poor Monitoring and Evaluation – “Praise Culture” • “Supply driven demand driven development” (Voices of the Bank)

23. UPP2 Project - Indonesia • $200 Million in 3 provinces • “Urban” • “Kelurahan” Based • $20,000 per community of 10,000 • Committee Elected to Manage (BKM) • Village Infrastructure Groups • Micro-Credit Groups

24. Quantitative Evaluation • Discontinuity Design • Sample: Approx. 200 treatment, 200 control • Baseline, Mid-term, Follow-Up

25. Questionnaires • 3 weeks fieldwork on UPP1 projects • “Social Capital/Gotong Royong” • Govt. and Community Projects • Credit Group Participation • Socio-economic • Questionnaires to Leaders and Activists • FGD – Oligarchy, Community Activities • Facilitators

26. Qualitative • 12 Control, 12 Treatment • Team of 4 spend 10 days for each round. • 6 FGDs • 20-30 In-depth Interviews • Mapping of Community

27. Qualitative continued.. • Baseline – tracks initial conditions • Mid-Term - Tracks formation of groups (BKM, Infrastructure and Credit) - Tracks Facilitation Process • Final

28. Monitoring Systems • MIS Data Base • UPP Website • Facilitator Feedback and Learning

29. Baseline Findings • Public Goods (40% Govt., 60% Community Contributions) • High-Level of Civic Participation (Gotong Royong) • Overlap between Religious and Govt. Institutions

30. References • E. Duflo, R. Glennerster, M. Kremer, “Using Randomization in Development Economics: A Toolkit” CEPR Discussion Paper No. 6059, January 2007 • G. Mansuri and V. Rao, “Community-Based and Driven Development: A Critical Review”, World Bank Research Observer, vol. 19, no. 1, pp. 1-39 2004 • V. Rao and M. Woolcock,” Integrating Qualitative and Quantitative Approaches to Program Evaluation,” Chapter 8 in Francois Bourguignon and Luiz A. Pereira da Silva (editors), The Impact of Economic Policies on Poverty and Income Distribution: Evaluation Techniques and Tools, World Bank and Oxford University Press, 2004. • M. Ravallion, “Evaluating Anti-Poverty Programs,” T.P. Schultz and J. Strauss (editors) Handbook of Development Economics (forthcoming)