The credit union difference: how to measure and report it – PPT materials for the Toolkit

1. The credit union difference: how to measure and report it – PPT materials for the Toolkit Funded by:

2. Social Impact Measurement for Credit Unions Workshop 1 on Design Unify 23 July 2018 Hoot 24 July 2018

3. Agenda • 11.00 - Introduction • 11.10 - Session A – Unpacking the ideas • 11.50 - Session B – Mapping stakeholders' issues • 12.30 - Session C – Formulating the theory • 1.10 - Lunch • 1.30 - Session D – Refining the theory • 2.10 - Session E – Reviewing the theory • 2.50 – Conclusion • 3.00 – Ends

4. 11.00 - Introduction • Background to the project • Agenda • Expectations

5. 11.10 - Session A - Unpacking the ideas in ‘SIM’ • Impact... • Measurement... • Reporting... • ToC thinking... • Management... • Learning Questions A

6. Impact... • Is a long term, positive, durable & sustainable feature of the financial circumstances of members & their communities to which the credit union contributes through working with them; • Flows from the members’ engagement with the credit union; • Varies with the financial circumstances of the members, which reflects differences between ‘affluent’, ‘coping’ and ‘vulnerable’ members;

7. Measurement... • Is the means through which we collect evidence to demonstrate impact; • Makes use of either or both qualitative & quantitative methods; • Should extract value from existing before collecting new data; • May rely on direct or indirect (proxy) sources; • Should be proportionate (costs to benefits); • Egs SROI / HACT’s Social Value; BSC's Outcomes Matrix;

8. Reporting... • Typically written reports, making extensive use of graphics to enhance visual appeal; • ‘Case studies, ‘stories’, ‘dashboards’, ‘scorecards’, ‘personas’, etc; • The issue is whether the reporting meets the legitimate interests of the stakeholder; • Does it convince them that the credit union is having the impact it claims?

9. ToC thinking… • 1) Analyse context about needs, barriers, enablers, etc, 2) examine assumptions about how we may intervene, 3) assess the evidence that what we do brings about the desired change; • Articulating / surfacing & linking / connecting the imp elements of contexts, assumptions, evidence; • Making explicit how what we deliver (memberships, loans, savings) has effects that lead to positive changes in financial circumstances;

10. Management... • Sustains relationships with stakeholders; • Evidences the case for resource support from public sector; • Enhances governance through facilitating accountability; • Creates basis for persuasive / marketing communications; • Contributes to more informed decision-making at operational and strategic levels;

11. Discussion Questions • What do you want social impact measurement to deliver for the credit union? • What challenges for the credit union arise from social impact measurement? • Who wants to know about the impact of the credit union - to whom should the credit union report?

12. 11.50 - Session B - Mapping stakeholders' issues • Why map? Clarifying (any) gaps in current reporting to these stakeholders? • Consider what is the impact that the interested and influential stakeholders want the credit union to deliver.

13. Why map? • Who is the audience for the reportage? • Map / matrix of stakeholders (s), what aspect of your impact (i) do you want to communicate to them?

14. The key questions are: • What is the issue that you and they want addressed? • What are the key factors that influence this issue? • Whose behaviour needs to change in ways that address the issue? • How ill the credit union enable these people to change their behaviour?

15. Map / matrix of stakeholders (s), showing what aspect of your impact (i) do you want to communicate to them?

16. Clarifying (any) gaps in current reporting to these stakeholders? • Discuss (any) gaps in the reporting through which the credit union communicates impact to these stakeholders (esp those in blue cells). • What evidence would convince the stakeholder(s) about this aspect of the impact? • is the credit union providing this evidence? • What needs to be done to fill these gaps? • Is it feasible that the credit union would do this? • Would this be proportionate to the benefits?

17. Consider what is the impact that the interested and influential stakeholders want the credit union to deliver. • Is it sensible to frame this issue as 'too few people borrow from (or save with) the credit union'? • EXERCISE 'But why do too few...'

18. 12.30 - Session C - Formulating the ToC • What is ToC? Why is ToC important for impact measurement? • Assumptions & Propositions • Pathways & Hypotheses • What should they look like? • Understanding the theory / logic underlying the credit union's impact

19. What is ToC? • an approach • a method • a diagram • a definition • claims for ToC

20. What is the ToC approach? • Is about understanding the dynamic in which the credit union intervenes (to secure the impact) • Financial wellbeing, and how it is fostered, is complex rather than linear; • We should have a set of ideas to understand the process, ie a theory; • With the theory we will be better able to learn about what does or not work and communicate this to interested parties;

21. ToC, a definition • A set of hypotheses and critical assumptions that make up a causal pathway of change which is the basis of the ... design [of the intervention]. • Hypotheses are ‘if-then’ statements between different levels of the change pathway (Drinkwater; 2009)

22. Claims for ToC • Integrate planning, implementation, and evaluation. • Prevent mismatches between activities and effects. • Enhance accountability by keeping stakeholders focused on outcomes. • Help set priorities for allocating resources. • Reveal data needs and provide a framework for interpreting results. • Enhance learning by integrating research findings and practice wisdom. • Define a shared language and shared vision for community change. (Ordonez; 2014)

23. Why is ToC important for impact measurement? • Facilitates analysis of assumptions about context and effect of the intervention • Helps show 'reality’ of the causal pathways • Explores propositions about 'if A, then B’ • About the logic / why of pathways • Does the intervention make sense?

24. Hypotheses • about change as it occurs in different domains • Domain A: changing people's behaviours? • COM-B - Capability, Opportunity, Motivation & Behaviour • Domain B: changing agency practices? • Advocacy in media about change as it occurs in different domains

25. What they look like? • 'left to right' v 'top to bottom’ (too static?) • 'box & arrow’ (too messy?) • mechanistic / linear v holistic / complex • A ‘boxy’ example… • EXERCISE: What additional elements would align this to ‘good practice’ (as above, what aspect of context is missing)?

27. Basics of how to ‘do’ ToC? • Forward mapping: inputs  activities outputs  outcomes  impact; • Backward mapping from desired impact to inputs is often better as it challenges us; • If we have the [specified] impact, then we need... • To have delivered these outcomes, we need... • To have produced these outputs, we need... • To have carried out activities, we need... • To have obtained these inputs for resources.

28. ToC EXERCISE • Using the suggested ‘backward mapping’ & starting from the impact the credit union intends to have, outline / sketch the elements and how they are linked through the causal process… • Impact, what is it, who benefits from it, at what level does it occur (community v individual) what is the ‘problem’ to which it is a ‘solution’?;

29. ToC Exercise Continued • What are the outcomes from the work that contribute to the impact – esp moving to ‘low value / high income’ loans? • How do the outputs, (savings, loans, others) feed into these outcomes? • In what ways do the credit union’s activities produce the outputs? • What inputs does the credit union require as resources fuel the process?

30. ToC QUESTIONS • Are the links from inputs to impacts clear? • Would the way you present the links help convince a sceptical person? • Does the ToC 'add up' as a way of addressing the issues you want the credit union to address? • Are there assumptions in the theory that may not be true? What are these? Should the ToC change so that it wont matter? • How would you summarise the basic hypothesis(es) about credit / wellbeing that underpins the ToC? (Try to formulate this as a testable idea).

31. 1.30 - Session D - Refining the ToC, Recap... • Inputs resource activities that produce outputs that generate results that deliver outcomes that create impact. • If you were sceptical what would allay any concerns you might have about the claims made about the process? • EXERCISE Taking each of the questions in turn, write brief notes on how you would convince a sceptical observer.

32. How would we answer the sceptic? • Is the process in the ToC consistent with different explanations about how and why the actions influence the effects? • Is the impact identified plausibly linked to other explanations (apart from that of the credit union's intervention)? • Are the claims made in the ToC open to communicate in plain English what is the theory about the links in the chain of causality, eg, If we do x action, then y change will follow because...? • Are there particular / key / critical points in time (between, eg, loan made' and 'enhanced financial resilience'), which we would like to know about in order to trace the change in resilience back to the borrowing profile?

33. 2.10 - Session E - Review of ToC • Does the theory make sense as a response to the financial circumstances of people's lives in the area (the context)? Would it make sense to a) a senior officer in local government and b) a member of the credit union?

34. Summary Review Questions • Is the theory plausible as a statement of how local people live / stakeholders operate? • Is if feasible for the credit union to implement it? • Is it sensible to think that we could test the theory?

35. Review Questions cont’d • Does the impact that the theory identifies offer a reasonable prospect of addressing the 'too few / too many' issue you identified in the stakeholder mapping exercise? • Does the diagram make clear the causal process? Are the assumptions made explicit? What is the quality of the evidence for the assumptions? • EXERCISE In the light of your answers to these questions, how would you wish to have the theory further refined?

36. Does the diagram make clear the causal process? • Is there enough detail that links the 'if... then' logic? Are there any missing links? • Is it clear how the different elements relate to each other (activities >>> results >>> outcomes)? • Does it communicate enough of the specific character of the credit union?

37. Are the assumptions made explicit... • about the causal process • about the actions of the credit union (to activate this process) • about the enablers and barriers that arise from the context?

38. What is the quality of the evidence? • Is there evidence noted for the major hypotheses (about behaviour change?) that underpin the theory? • Is there some assessment of the strength of this evidence? • Is the assessment plausible?

39. EXERCISE In the light of your answers to these questions, how would you wish to have the theory further refined?

40. 2.50 - Conclusion • Participants to say if, and extent to which, workshop fulfilled their expectations and comment on the nature of any gaps in these expectations left unfilled. • Expectations revisited – did it deliver? • Evaluation – what worked, what didn’t’?

41. Next Steps • Small Change to write-up report on workshop for the credit union; • Small Change to prepare for Workshop 2 on implementation in light of this; • Ends [3.00]

42. Social Impact Measurement for Credit Unions Workshop 2 on Data Unify 31st July 2018 Hoot 1st August 2018

43. Agenda • 11.00 - Session A - Overview of workshop 1 & progress • 11.30 - Session B - Practical measurement options • 12.00 - Session C – Quantification & wellbeing • 12.30 - Session D – Monetisation & partners’ savings • 1.00 - Lunch • 1.20 - Session E - Qualitative commentary on impact • 1.50 - Session F - Questions for discussion on 3 options • 2.20 - Session G - Data system given preferred option • 2.50 - Conclusion, evaluation & next steps

44. 11.00 - Session A - Overview of workshop 1 & progress to date • Unify • logic model • matrix form of ToC • Hoot • systems mapping • Levels / domains form of ToC

45. Questions for discussion on progress • If the CU reports on these impacts to local government / housing & local employers will its credibility rise? • What factors will enhance the credibility of the reportage? What factors could detract from its credibility? What else would need to be done? • For these stakeholders, what are the hallmarks of a quality report?

46. 11.30 - Session B - Outlining the practical measurement options that demonstrates impact • Option 1) in-house requiring staff time and quality assurance of independence / validity / credibility; • Option 2) out-source to external consultant or University through securing grant-aid; • Discuss advantages & disadvantages of the two options;

47. 12.00 - Session C - Quantitative measurement of contribution to members' wellbeing (HACT) What is HACT’s wellbeing valuation approach? The value of a change in satisfaction with life that correlates with (predicts) an outcome (eg ‘reduced stress over unmanageable debt,’ ‘feeling more in control of own finances’) expressed as the {average} amount of additional money a person would require for s/he to report an equivalent change in satisfaction;

48. How does HACT value wellbeing? • Use quality survey data eg Understanding Society • Identify outcome variables, eg health status, fin inclusion • Construct statistical model that predicts outcomes • ‘Controls’ for other factors eg working, education • Model amount of additional money that predicts this • Make allowance for ‘what would have happened anyway’ • Do surveys of CU’s members before loan & after loan • Use comparable questions, note diffs ‘before & after’ • Multiply number changed by HACT’s £ estimates

49. HACT’s Financial Inclusion Outcome Values

50. HACT’s Health Outcome Values