Combining different methods in impact evaluation

1. Combining different methods in impact evaluation Presentation by Steen Folke, Senior researcher at the Danish Institute for International Studies International Workshop on Impact Evaluation Paris, November 15 2006

2. Important banalities • Impact evaluation is not an exact science – and it is dangerous to pretend that it is • The use of a combination of different – quantitative and qualitative – methods can highlight different dimensions of impact • Impact is the result of development processes that depend on the context as well as the intervention

3. Definition of impact • ”Impact assessment is the systematic analysis of the lasting or significant changes – positive or negative, intended or not – in people’s lives brought about by a given action or series of actions” (Chris Roche: Impact Assessment for Development Agencies, Oxford 1999)

4. Classical effect evaluations • Quasi-experimental survey design • Ceteris paribus assumptions • Before/after and/or with/without • Quantitative methods • Attempted objectivity • Dubious assumptions

5. Participatory impact assessments • Involvement of beneficiaries • External facilitators • Participatory techniques • Qualitative methods • Subjective • Problems of reliability

6. Wider impact studies • Heterogeneous category of in-depth studies • Contextualised, tailor-made approach • Unintended consequences as well as stated objectives • Extensive fieldwork • Combination of quantitative and qualitative methods • Development interventions and societal processes

7. Model of development intervention, context, process and impact

8. Noakhali Rural Development Project, Bangladesh • Danida funding 1978-92 390 m. DKK (> $50 m.) – flagship in Danish aid • 2 phases, 15 components: irrigation, cooperatives, rural poor prog., mass education etc. • >60 expatriate advisers, staff >1000 • Waning enthusiasm, many implementation problems (complexity) • 3rd phase planned, but aborted

9. Ex-post impact evaluation • Did the flagship float or sink? • No terminal evaluation • Ex-post impact study 9 years after • Contextualised, tailor-made approach • Combination of quantitative and qualitative methods • 8 researchers, 15 assistants, 4 months fieldwork

10. Methods I: Project focus • Documentary study (project documents) • Archival work in Danish embassy, Dhaka • Questionnaire survey with former advisers and Danida staff • Stakeholder interviews (Danida staff, former advisers, Bangladeshi staff) • Quantitative analysis of project monitoring data

11. Methods II: Study of context • The national context: books, articles, statistics (economy, policies) • The local context: census data, other sources (environment, population, socio-economic development) • Institutional mapping (esp. NGOs) • Extensive village studies (12 villages, 9 with and 3 without NRDP) • Intensive village studies (4 NRDP villages, restudy 20 years later)

12. Methods III: Quantitative • Surveys of 5 important project components: irrigation, infrastructure, fisheries, cooperatives for rural poor, mass education • Random sampling (beneficiaries) • Questionnaire-based interviews with beneficiaries • Some interviews with non-beneficiaries (’control group’)

13. Methods IV: Qualitative • Assessment of roads, buildings and irrigation canals (function, maintenance) • Key informant interviews • Focus group interviews • Observation • In-depth interviews (issue-based and life stories)

14. Example of findings: Irrigation IEconomic situation of household

15. Example of findings: Irrigation II Most important reason for improvement in economic situation compared to 15 years ago (total: 59 respondents): Increased production (irrigation): 20 resp. Remittances: 20 resp. Petty trade/business: 8 resp. Other reasons: 11 resp. Most important reasons for deterioration (17 resp): Illness, large family, loss in business

16. Conclusion: The project • The Danida flagship did not sink – the evaluation team found substantial impact in many areas 9 years after • The impact was primarily in the form of marginal improvements for the rural population, mainly the poor and women • The socio-economic inequalities and the local power structure were not challenged

17. Conclusion: The methods • The combination of a range of different methods contributed to unravel different dimensions of impact from a complex project • Study of the context was essential to understand impact of the project versus other factors • Quantitative and qualitative methods were complementary • Attempts to assess the counterfactual brought limited insight