Impact Assessment

1. Impact Assessment • What is an impact assessment? • An impact assessment looks at what has changed within a society and tries to unravel what factors have contributed to it. Participatory assessments involve groups of beneficiaries reflecting and analysing the project together; as a means to understand how things worked out and why. • When do you do it? Once a project is well-established, an impact assessment is a means to understand your contribution to a community by asking – what has changed and for whom? How significant is that for them? How did the project contribute, How sustainable is it? • Who is it for? It primarily provides learning to an internal audience but can be adapted to provide information to donors, as well as beneficiaries and your community linking peers. • Why bother? Participatory reflection can prompt action and impact, as well as help explain changes to ourselves, to support and enhance our impact by enhancing the multiplier effects and tackling the negatives. • Where do we begin? How you go about it is essential – an open sharing process can create a safe space to look at both success and failure and go beyond the gilding of the lily. Utilise your existing trips and listen to a variety of beneficiaries’ opinions. As well as getting a narrative of change through stories, it is important to gather numbers too – they can describe the scale and scope of change. Work with partners to see what figures they have which could support your assessment. examples • Focus on Impact: • Impact is beyond the remit of a project’s direct activity or influence and is about broader change within a community towards long term goals such as sustainable livelihoods and poverty reduction. • People often mistake outcomes for impact. If you train someone, that’s an outcome, if they go on to train someone else, that’s an impact. The differences are the attribution of influence and the time frame over which change occurs. • Impact assessments look at what meaning your project has in people’s lives, often delving into norms, power, knowledge, values and relationships, to understand what supports and what hinders change. • EXAMPLE ONE: Reveal meaning with photos • Goats for vulnerable families in Uganda: DolenFfermiosupports orphans by providing goats, fruit trees, and technical services to enhance the meat and the milk of goats and improve family life. With proper care, these gifts can provide fruit or milk to sell and support their schooling. • Photo diaries: DolenFfermiohas used photo diaries in schools as a means to deepen exchange and understanding between communities. This fun participatory tool can be used as a form of impact assessment by posing questions or working around a specific subject. Beneficiaries go and take pictures and then write stories explaining why the image is significant to them and their family. They’ll need initial practice at taking photos - training on composition, content and techniques such as choosing a subject, holding the camera, lighting, etc. Digital cameras are great for training and disposable cameras can be used for the project (i.e. the assessment). Once the pictures are developed, let each child discover what they have created. Encourage them to choose the pictures they think are most significant and why and then ask them to write it up. • The process is as important as the goal: This is a fun opportunity for children to learn, share and play through photography. Don’t get too focused on assessment and forget to enjoy the process!

2. EXAMPLE TWO: Focus group discussions reveal new insights • Motorbike ambulance service in Uganda: PONT wants to support safer childbirth by addressing the distances between villages health centers and hospitals by providing a motorbike ambulance service. Traditional Birth Attendants and Community Health Workers have been given mobile phones and training to call the ambulance and refer women into the clinic. Log books show that some women are going to health centres when in labour, but the project would like to know if there has been a wider change in attitudes towards home births. • Focus group discussions with potential service users: A focus group discussion with groups of women who have recently had a child (preferably some first time mums too) would form a basis to find out current and changing attitudes to child birth. Would women would prefer to give birth at home? Why? Are there hidden costs or barriers to assisted births? Such as needing food or money while in hospital? Would their husbands prefer them to stay in the village? Do they need to seek permission to leave the village for medical attention? Do their mother’s in law prefer them to stay locally? These are some example questions which provide insight into social norms and behaviours which might prevent safer childbirth. • Consider power and vested interests: To gather a group of appropriate women it will be necessary to work through the TBA. She may be concerned that the meeting could criticise and threaten her position. It is very important to work with her and to reassure her this process is to identify any additional support for the motorbike service. Yet, if possible, try to conduct the meeting without the TBA present as she may stifle voices. • EXAMPLE THREE: Participatory score cards highlight current attitudes and behaviours: • Rain water harvesting in Rwandan schools: This project works to improve child health, learning outcomes and sustainable practices by harvesting rainwater to provide a sustainable supply of fresh water in school. • A participatory scorecard: A participatory scorecard is a powerful way of getting groups to provide feedback around beliefs , attitudes and current practices through group discussions. These conversations shed light on change and whether new infrastructure has challenged traditional behaviours. Identify a series of statements which you can get groups to rate: agree/strongly agree or disagree/strongly disagree with a series of smiley or sad faces. These statements will then be presented to individuals within focus groups for them to score on their own, before discussing them as a group and agreeing an overall rating. The subsequent discussion about how it has been rated and why will enable you to see differences in attitudes, behaviours and experiences. The act of scoring forces people to justify their story and can lead to further questions and self-analysis among the group. This itself can prompt action. This exercise can be repeated over time to see if there are any changes. • Sample scorecard questions: I can get clean water at school even in the dry season; I drink clean water at school every day; I am happy to use the latrines every time I need the toilet; When I fetch water every morning it is for my home not my school. These questions will bring up a variety of other issues including what they consider clean water to be, whether drinking water is limited at the school or at certain times of year. However hand-washing behaviours, for example, may be more accurately assessed through observation, as people will often say what they think you want to hear. • Consider age gender and ethnicity: Always split groups into those with common experiences – for example, male and female, or teenagers and primary pupils and reflect that in your groupings. This comes back to the issue of power and the need to understand the poorest and most vulnerable. • Don’t forget: NUMBERS!>? • Use your body language, tone and manner to convey that you are listening when talking to beneficiaries. • Take notes – there’s no way you can possibly remember your impressions and what was said afterwards. • Utilise existing trips and visits to villages to uncover the meaning in your work. • Participatory techniques can be used for evaluation purposes too – why not use them as a means to deepen your understanding of the context? • Hidden agendas and vested interestscan affect what people are willing to say – are they gilding the lily? Think about how you can get different perspectives of similar issues to triangulate your information. • Stand bank and consider whether there are any changes in the community. Has your project contributed and how? What worked? What’s stopping further change? Consider how this implicates upon what you do – do you need to modify your position? Or change your strategy to maximise the good? Disclaimer: these assessments are fictional but based upon real projects. If you want to find out more about these participatory processes which can support your project – contact Ed and Craig at WACL