Capturing stories of change and The Most Significant Change technique

1. Capturing stories of change and The Most Significant Change technique Introductory workshop Prepared by: Zuleika Arashiro

2. I. Participation in Evaluation

3. What leads to a useful evaluation? Source: Ontario Nonprofit Network, https://theonn.ca/learned-talking-evaluation-funders/

4. Concepts of participatory evaluation “Participatory evaluation is an evaluation that involves all the stakeholders in a project - those directly affected by it or by carrying it out - in every phase of evaluating it, and in applying the results of that evaluation to the improvement of the work” (Community Tool Box) “Participatory evaluation is an approach that involves the stakeholders of a programme or policy in the evaluation process. This involvement can occur at any stage of the evaluation process, from the evaluation design to the data collection and analysis and the reporting of the study. A participatory approach can be taken with any impact evaluation design, and with quantitative and qualitative data. However, the type and level of stakeholder involvement will necessarily vary between different types…” (Better Evaluation)

5. Larger purpose: Origins in democracy and citizenship debates, linked to the higher steps of the ladder. • Program-specific purposes: Reflect on what the evaluation is about and the motivation to conduct it. Then consider: • Is this a suitable approach for this program? • Is it doable (resources, commitment) within the set timeframe? • Who will organise it? Is there any conflict of interest? Source: Clemens Ley and Maria Rato Barrios 2010, ‘Participatory evaluation and research in context of violence and conflict’ (https://www.slideshare.net/ISDPA/clemens-ley-participatory-evaluation-and-research)

6. Disadvantages of Participatory Evaluation • It takes more time than conventional process. • It takes the establishment of trust among all participants in the process. • You have to make sure that everyone's involved, not just "leaders" of various groups. • You have to train people to understand evaluation and how the participatory process works, as well as teaching them basic research skills. • You have to get buy-in and commitment from participants. • People's lives - illness, child care and relationship problems, getting the crops in, etc. - may cause delays or get in the way of the evaluation.                     • You may have to be creative about how you get, record, and report information. • Funders and policy makers may not understand or believein participatory evaluation. Advantages of Participatory Evaluation • It gives you a better perspective on both the initial needs of the project's beneficiaries, and on its ultimate effects. • It can get you information you wouldn't get otherwise. • It tells you what worked and what didn't from the perspective of those most directly involved - beneficiaries and staff. • It can tell you why something does or doesn't work. • It results in a more effective project. • It empowers stakeholders. • It can provide a voice for those who are often not heard. • It teaches skills that can be used in employment and other areas of life. • It bolsters self-confidence and self-esteem in those who may have little of either.  • It demonstrates to people ways in which they can take more control of their lives. • It encourages stakeholder ownership of the project. • It can spark creativity in everyone involved. • It encourages working collaboratively. • It fits into a larger participatory effort. Source: Community Tool Box, Participatory evaluation presentation (https://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/participatory-evaluation/powerpoint)

7. 2. The Most Significant Change technique

8. ‘MSC is a form of participatory monitoring and evaluation. It is participatory because many project stakeholders are involved both in deciding the sorts of changes to be recorded and in analysing the data collected. It is a form of monitoring because it occurs throughout the program cycle and provides information to help people manage the program. It contributes to evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program as a whole.’ Rick Davies, developer of the MSC http://mande.co.uk/special-issues/most-significant-change-msc/

9. Background • It was initially developed by Rick Davies in the context of rural programs in Bangladesh. In Australia it has been adapted and disseminated by Jess Dart, from Clear Horizon. • The MSC is a process (6 months/one year/2 years), not something that happens at the end of a program. • It involves telling stories in a systematic way, with purpose and method for collection and analyses of the most relevant stories of change. It therefore requires selecting stories, which can be a sensitive issue in participatory processes. • It focuses on learning (rather than accountability) and it should be combined with other evaluation techniques to make for a sound ‘evaluation.’

10. MSC – Not recommended • Organizations want to only capture the expected change • Organizations which only want the good news stories for public relations • Where the program evaluation is already completed • When the organization only wants to understand the average experience of participants • When an organization needs to produce a report for accountability purposes • When a more inexpensive evaluation is adequate for reporting purposes MSC – Most useful • In programs that are complex and produce diverse and emergent outcomes • In large organizations with numerous layers • Initiatives that are focused on social change • Initiatives and organizations that are participatory in nature • Initiatives which have repeated contact with stakeholders over time • Initiatives which are struggling with conventional monitoring systems • Initiatives which have highly customized interventions for a small number of beneficiaries What is your project’s situation? Source: Tamarack Institute, Tool: Most Significant Change, http://www.tamarackcommunity.ca/library/most-significant-change

11. Exercise1 : Share a significant story of change Share with your partner a story of change associated to your participation in a program/project. Think about: • What was the program/project aims? • What was the most impressive change for you? (positive or negative) • Why did you select that in particular? • How do you see the program/project contributing to that change?

12. Are there common areas of change?

13. Note: Step 3 was modified to highlight the importance of process design in a non-hierarchical style. Source: Davies, R & Dart, J 2005. The ‘Most Significant Change’ technique. A guide to its use’, https://www.clearhorizon.com.au/f.ashx/dd-2005-msc_user_guide.pdf

14. Things to keep in mind • How does each community like to communicate its stories? Oral? Visual? Written? • How would people like to share it (photos, songs, paintings, dance)? • How can we increase community inputs, specially if they are not comfortable with selecting ‘the most relevant’ stories at the exclusion of others in the group? • For this project, do they prefer individual or collective story telling? (Consider sensitivity of issue, privacy…)

15. Building bridgesHow can we collect and communicate stories of change across our communities in a way that is inclusive?communicates well?allows our communities to learn more about each other?

16. Key resources • Clear Horizon https://www.clearhorizon.com.au/resources.aspx • Monitoring and Evaluation NEWS (Ricky Davies’ blog) http://mande.co.uk/special-issues/most-significant-change-msc/ • Community Development Toolbox https://ctb.ku.edu/en • Better Evaluation https://www.betterevaluation.org/