The SPADA/MCLE impact evaluation: work in progress

1. The SPADA/MCLE impact evaluation:work in progress Markus Goldstein AFTPM/DECRG

2. The acronyms • This is an impact evaluation of the Mediation and Community Legal Empowerment of the Support for Poor and Disadvantaged Areas Project in Indonesia • It’s a big project, and the impact evaluation will only be for one component (other impact evaluations for other components)

3. The big idea • We have (almost) no rigorous evidence of the effects of governance reform • Why? • It’s hard to measure • It’s hard to set up • Setting up control groups • Getting sufficient sample size • It’s political science & IE has tended to come from economists • We know very little about poverty-governance-justice nexus

4. MCLE: project objectives • Strengthen the capacity of institutions and individuals to resolve disputes in an open, independent, and just manner • Assist and facilitate the resolution of disputes involving community wide interests • Increase public trust in the formal legal system by enhancing community legal awareness and access to the system

5. MCLE: project activities • Community legal education • Capacity building for informal justice actors • Legal aid and complaint handling network • Awareness raising of community legal needs and challenges  NGO implemented under government contract

6. MCLE activities - 2 • Support network for providing legal aid and empowerment activities • Focus at community level • Sub-district: establish posko (community legal aid posts), staffed by facilitator and providing legal information, mediation and complaints handling • Village level: paralegals (one man/one woman) – work with local leaders and community figures • District level: community lawyer – part time technical legal support

7. Why evaluate this project? • Global • What does access to law get you? • What works to get people access to justice? • Top down vs bottom up access to legal system & formal vs informal dispute resolution • Local • Gov’t is preparing first ever access to justice strategy for inclusion in the 2009-14 medium-term development plan • Inform nation-wide poverty and empowerment program • Scale up or not?

8. Evaluation questions: primary • Does MCLE increase beneficiaries ability to claim rights, enforce rights, and resolve disputes through non-violent means? • Does MCLE reduce violent conflict and increase safety and security? • Does MCLE improve HH welfare in the villages that receive paralegals?

9. Note on analyzing the causal chain… • Idea here is to look at different points of the causal chain: • Knowledge of rights (ask about legal K) • Enforcement of rights (e.g. get a land cert.) • Better dispute resolution (e.g. mode used, hypothetical dispute resolution) • Investment in property (e.g. in land improvements) • Improvements in household consumption

10. Evaluation questions: secondary • Does MCLE increase the reporting of corruption and bribery and decrease these events in the long term? • Does MCLE improve individuals’ capacity to engage with the state and the state capacity to respond, particularly for dispute resolution? • Does MCLE improve gender relations at the household and community level?

12. Evaluation design • Two provinces: selected for level of conflict (Aceh and Maluku) • Districts chosen by gov’t based on need and demand: issues of generalizability • Sub-districts: random selection agreed by gov’t • Villages: random selection • Randomization done in Washington with evaluation team

13. Evaluation design:quantitative core • Method is a randomized difference in difference • Baseline quant survey to run right as contracting for implementation goes out • Mid-point to run 2 years after implementation (early results for policy) • End-line survey to run 4 years after implementation

14. Evaluation design: survey structure • Innovative household survey (“best of” other governance/justice surveys) • Pretest and adjust • Links with SUSENAS • Rotating panel • community survey (capture major disputes, village level variables) • Key informant survey (capture village/district authority attributes, perceptions of safety, corruption, legal claims) • Program facilitator survey (get at heterogenity of treatment – ed levels, experience, social connections)

15. Evaluation design:other data sources • Newspaper monitoring: conflict levels, security and safety, reported corruption • Project MIS – implementation and also dispute tracking and other case handling information

16. Evaluation design:Qualitative core • Quant answers: does MCLE have an impact while Qual answers: How does it have an impact • Village mapping (understanding village dynamics) • Village profile (demographics, livelihoods, etc) • Dispute timeline (types, sequence, actors,etc) • Claims and collection timeline (indiv vs community) • Power relations and networks – map these

17. Evaluation design:Qualitative core • Comparative case studies • In-depth studies in treatment and comparison locations – similar to quant in structure • Community ethnographies • Two per province • How does program work, how does it interact w/local power structures, experiences of key individuals • Cooperation across qual-quant data collection, not just iterative data collection

18. Complication: CRT • Add a conflict resolution training component in a select number of villages • Creates sampling headache • Power issues: the effect of treatment A vs treatment B requires (in general) a larger sample than treatment A vs control • Solution: lay it on top of MCLE (still have avg effect, control for CRT), separate CRT only villages. CRT contributes to surveys • Danger here is that things get too complicated and we end up with nothing done well…

19. Evaluation team • East Asia SDV: project design team, evaluation/survey expertise • PRMPR/DECRG: evaluation design/analysis team (quant/qual) • Gov’t: project design/M&E staff • Legal: eval design, guidance, dissemination, funding • DECRG: survey design specialist • Consultants: sampling, local qualitative work • Survey design firm

20. Dissemination plan • Core team builds design • Disseminate evaluation design w/ wider gov’t audience, implementing NGOs • Baseline analysis – informs program design, gov’t agencies • IE results: • Workshop w/ gov’t early – compare treatment arms • Disseminate more widely – other relevant programs (national pov/empowerment program), NGOs, Bank, donors, academics, regional J4P program

21. Budget and funding • Survey costs: still unknown, problem of Maluku islands • Qual contract out • Initial seed funding from BNPP grant, Aceh TFs, project $. More grant applications on the way • Donated time helps a lot (save on BB) • Missions at end of FY…

22. A couple of points of reflection • The process was important, esp initial workshop with gov’t & project team • Project & IE team really, really have to work together (lots of early am VCs) • Be prepared for transitions in all teams • Doing something new? Early survey pretest helps for sample size calculations • Timing of “end-line” survey depends on policy cycle – but also time for effects to manifest • Keep it as simple as you can