1 / 17

Combining different methods in impact evaluation

In this presentation, Steen Folke highlights the importance of combining different quantitative and qualitative methods in impact evaluation, using the Noakhali Rural Development Project in Bangladesh as a case study. By examining the project's context, process, and impact, the evaluation team uncovers the lasting changes brought about by the intervention. The findings reveal both the successes and limitations of the project, emphasizing the need for a holistic approach to impact evaluation.

crowell
Download Presentation

Combining different methods in impact evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Combining different methods in impact evaluation Presentation by Steen Folke, Senior researcher at the Danish Institute for International Studies International Workshop on Impact Evaluation Paris, November 15 2006

  2. Important banalities • Impact evaluation is not an exact science – and it is dangerous to pretend that it is • The use of a combination of different – quantitative and qualitative – methods can highlight different dimensions of impact • Impact is the result of development processes that depend on the context as well as the intervention

  3. Definition of impact • ”Impact assessment is the systematic analysis of the lasting or significant changes – positive or negative, intended or not – in people’s lives brought about by a given action or series of actions” (Chris Roche: Impact Assessment for Development Agencies, Oxford 1999)

  4. Classical effect evaluations • Quasi-experimental survey design • Ceteris paribus assumptions • Before/after and/or with/without • Quantitative methods • Attempted objectivity • Dubious assumptions

  5. Participatory impact assessments • Involvement of beneficiaries • External facilitators • Participatory techniques • Qualitative methods • Subjective • Problems of reliability

  6. Wider impact studies • Heterogeneous category of in-depth studies • Contextualised, tailor-made approach • Unintended consequences as well as stated objectives • Extensive fieldwork • Combination of quantitative and qualitative methods • Development interventions and societal processes

  7. Model of development intervention, context, process and impact

  8. Noakhali Rural Development Project, Bangladesh • Danida funding 1978-92 390 m. DKK (> $50 m.) – flagship in Danish aid • 2 phases, 15 components: irrigation, cooperatives, rural poor prog., mass education etc. • >60 expatriate advisers, staff >1000 • Waning enthusiasm, many implementation problems (complexity) • 3rd phase planned, but aborted

  9. Ex-post impact evaluation • Did the flagship float or sink? • No terminal evaluation • Ex-post impact study 9 years after • Contextualised, tailor-made approach • Combination of quantitative and qualitative methods • 8 researchers, 15 assistants, 4 months fieldwork

  10. Methods I: Project focus • Documentary study (project documents) • Archival work in Danish embassy, Dhaka • Questionnaire survey with former advisers and Danida staff • Stakeholder interviews (Danida staff, former advisers, Bangladeshi staff) • Quantitative analysis of project monitoring data

  11. Methods II: Study of context • The national context: books, articles, statistics (economy, policies) • The local context: census data, other sources (environment, population, socio-economic development) • Institutional mapping (esp. NGOs) • Extensive village studies (12 villages, 9 with and 3 without NRDP) • Intensive village studies (4 NRDP villages, restudy 20 years later)

  12. Methods III: Quantitative • Surveys of 5 important project components: irrigation, infrastructure, fisheries, cooperatives for rural poor, mass education • Random sampling (beneficiaries) • Questionnaire-based interviews with beneficiaries • Some interviews with non-beneficiaries (’control group’)

  13. Methods IV: Qualitative • Assessment of roads, buildings and irrigation canals (function, maintenance) • Key informant interviews • Focus group interviews • Observation • In-depth interviews (issue-based and life stories)

  14. Example of findings: Irrigation IEconomic situation of household

  15. Example of findings: Irrigation II Most important reason for improvement in economic situation compared to 15 years ago (total: 59 respondents): Increased production (irrigation): 20 resp. Remittances: 20 resp. Petty trade/business: 8 resp. Other reasons: 11 resp. Most important reasons for deterioration (17 resp): Illness, large family, loss in business

  16. Conclusion: The project • The Danida flagship did not sink – the evaluation team found substantial impact in many areas 9 years after • The impact was primarily in the form of marginal improvements for the rural population, mainly the poor and women • The socio-economic inequalities and the local power structure were not challenged

  17. Conclusion: The methods • The combination of a range of different methods contributed to unravel different dimensions of impact from a complex project • Study of the context was essential to understand impact of the project versus other factors • Quantitative and qualitative methods were complementary • Attempts to assess the counterfactual brought limited insight

More Related