1 / 33

A new way of impact e valua tion

A new way of impact e valua tion. Who values Whose values Value for what. Contents. Why do we do this? Shortcomings of current evaluation methods How do we do this? Some first findings. Shortcoming 1. Mostly not on impact Focus on ongoing projects

derora
Download Presentation

A new way of impact e valua tion

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A new way of impact evaluation Who values Whose values Value for what

  2. Contents • Why do we do this? • Shortcomings of current evaluation methods • How do we do this? • Some first findings

  3. Shortcoming 1 Mostly not on impact • Focus on ongoing projects • Focus on implementation (‘did we do well?’) • Focus on direct results: • Outputs: ‘What did the project produce’? • Outcomes: ‘How did the target group make use of the outputs of the project?’ • Impact: ‘What are the sustainable results in the communities?’

  4. Shortcoming 2 Lack of attention for context(even in impact assessments) • What did other projects do? • External factors?

  5. backdonor INGO NGO • Activities • Results • Outputs • Outcomes • Impact • M&E Project a In community x

  6. Other backdonors Other INGO’s NGO 2 NGO 3 • Government • Local • State • National Project c Project c Project b Project b Project a Project a Companies (e.g. telecom) Private initiatives backdonor INGO NGO 1 Project c Project b Project a Community x

  7. Natural Physical Economic Human Social Political Cultural Local influences National influences Global influences backdonor Other backdonors INGO Other INGO’s NGO 1 NGO 2 NGO 3 • Government • Local • State • National Project c Project c Project c Project b Project b Project b Project a Project a Project a Companies (e.g. telecom) Community x Very poor – poor – average – rich – very rich Private initiatives

  8. backdonor Other backdonors INGO Other INGO’s NGO 1 NGO 2 NGO 3 Project c Project c Project c Project b Project b Project b Project a Project a Project a Natural Physical Economic Human Social Political Cultural • Government • Local • State • National Companies (e.g. telecom) Community x Very poor – poor – average – rich – very rich Local influences Private initiatives National influences Global influences

  9. Natural Physical Economic Human Social Political Cultural Local influences National influences Global influences backdonor Other backdonors INGO Other INGO’s NGO 1 NGO 2 NGO 3 • Government • Local • State • National Project c Project c Project c Project b Project b Project b Project a Project a Project a Companies (e.g. telecom) Community x Very poor – poor – average – rich – very rich Private initiatives

  10. Shortcoming 3 • Often mainly based on expert opinion • What the expert defines as ‘impact’ • Often looking for predetermined results only • Little attention for unexpected effects

  11. Shortcoming 4 If evaluations look at target group perceptions, there is almost always bias “yes sir, we like the project and we want some more”

  12. Shortcoming 5 • Focus on ‘what’ and little on ‘how’, ‘with which values’ or ‘why’ projects were done

  13. How do we do this? Prisma with ICCO and Woord en Daadwith University of Amsterdamand University of Development Studiesand Expertise Développement Sahel In Burkina Faso and Ghana

  14. Not this perspective backdonor backdonor backdonor INGO INGO INGO NGO NGO Project NGO Project a In community x Project a In community x Project a In community x

  15. But this perspective Projects Actors Community x Very poor – poor – average – rich – very rich Changes in context

  16. 3 day workshops • 60 people from area of 20,000 • Subgroups: men, women, old, young(officials and project staff separate group) • Individual life history questionnaire • + data about parents • + data about brothers / sisters • + data about children (total about 600 persons per workshop) • 4 rounds of 3 workshops each

  17. Part 1: context • “What were major events?” • “What are changes over past 30 years? Are these positive or negative” (natural, physical, human, economic, social/political, cultural) • “Who are considered the • Very rich • Rich • Average • Poor • Very poor by local standards?”

  18. Part 2: interventions • “Which interventions happened?” (by government, private sector, secular ngo’s, christian ngo’s / churches, muslim ngo’s, individuals) • “How do you assess their impact?” (negative impact, no impact, impact in past, positive impact) + explanations !!!

  19. Part 3: Best and Worst • “What were best 5 and worst 5 projects?and why?” • “What did you tell your friends about these projects when they started?And what do you tell your friends about them now?” • “What is the effect of these projects on the 5 wealth classes?” • “What is the effect of these projects on the different domains?” • “What is the link with the changes in context?” (cause / mitigation)

  20. And then • 60 small reports per workshop • Analyse, compare and combine all subgroups

  21. First findings • Incredibly detailed historical profile • Big picture: • Natural domain: negative • Human domain (education / health): positive • Socio/political: mixed • Cultural: mixed • Economic: ‘we have more but we feel poorer’ • Wealth categories: • rich anthropological descriptions (meals, behaviour, jobs, funeral, family life, properties, etc.) • Very different per region

  22. Numbers of Projects

  23. Best projects by actor

  24. Worst projects by actor (Ghana)But different in Burkina

  25. Effects best pr. on Wealth Classes(Sandema, Ghana)

  26. Effects best pr. on Wealth Classes(Tô, Burkina Faso)

  27. Reasons exclusion very poor Miss preconditions for benefiting • Knowledge about the existence of an intervention • Access to the place where selection takes place • Having time available for themselves (or their children in case of education) to take part in the activities of an intervention • Having the feeling that they are allowed to participate (e.g. feeling that their dresses are sufficiently appropriate to be part of a meeting) • Being involved in the activities at which the intervention is focused (e.g. veterinary services will not benefit the very poor in some societies, since they do not have livestock)

  28. Reasons for best/worst • ‘Hard results’ • Relevance • Manner in which projects are done • Respect • Mutual trust (‘we can trust them’ and ‘we feel that they trust us’) • Participation (decision making, monitoring) • Long term commitment (‘they have us at heart’) • Honesty (‘something went wrong and they did not explain to us’) • Local presence (‘they are one of us’) • Results / expectations ratio • Expectations about length of stay (‘they left and never came back’) • Expectations about results (‘poverty will be ended here’) • Negative side effects

  29. Some final conclusion • Projects have many side effects • Best projects have effects in more than 1 domain (often including economic) • There are no black and white conclusions • Always nuances and exceptions • Christian among best and worst • Differences between men / women, old / young • Some of the biggest negative changes are hardly addressed • Climate change • Loss of cultural identity • Negative sides of individualism • Men, Women and Project officers all have different opinions

  30. Questions ? • http://users.fmg.uva.nl/kgeest/pda/index.htm

More Related