Measuring impact : Using quantitative research

1. Measuring impact: Using quantitative research Eibhlín Ní Ógáin London, November 2012 NPC - title of the document (can be changed under view/header and footer. Add date here if needed.

2. AGENDA NPC - title of the document (can be changed under view/header and footer. Add date here if needed.

3. Transforming the charity sector Charity We are a think tank and consultancy working at the nexus between charities and funders Increasing the impact of charities eg, outcomes frameworks Increasing the impact of funders eg, effective commissioning Strengthening the sector eg, collaboration towards shared goals Funder Sector

4. What is impact measurement?

5. Impact: The difference an organisation or activity makes to the people or problem it aims to help Impact measurement: Planning, managing, tracking and reviewing how much difference you are making

6. NPC’s approach to Measuring social impact

7. NPC’s approach to measuring social impact A well-developed Theory of Change Strategic vision / goals Existing evidence Appropriate measurement tools

8. Quantitative research

9. Why use quantitative research • Qualitative: understand something in detail. Rich information and insight. • Quantitative: how many people? How large an effect? How much impact and how robust? • Use quantitative research when dealing with large numbers of people and when you want to show a consistent effect. • E.g does new type of psychotherapy lead to improved mental well-being? Need consistent measure and need large sample. • Depends on type of ‘intervention’ and questions you are trying to answer. • Know a lot, measure a little, know a little, measure a lot.

10. Different methods in quantitative research • Ranges from easy to measure to hard to measure • Easier: Hard and objective outcomes: move into employment, gain a qualification, reduction in reoffending. • Harder: Softer, difficult to observe outcomes: changes in feelings, personal growth, well-being. • NPC’s preference is for previously tested scales and questionnaires. • If there is a gap, develop own questions.

11. Types of quantitative tools and their uses 11

12. When to measure • Pre-post design: how outcomes change over time by collecting data before (pre) and after (post) the programme. • When you think change happens influences your decision about: • When to collect the before measure • When to collect the after measure Before measure Programme / service After measure 12

13. Think about attribution • Is the programme the only factor influencing any changes? What would have happened anyway? • Control groups • Compare before and after data for participants and non-participants, ideally matching on their key characteristics • Holy grail: Randomised Control Trial, where potential participants randomly assigned to control group 13

14. Think aboutsampling • Representativeness • Does your sample represent your population? • Response rates • Is your response rate high enough to avoid bias? • Statistical validity • Are your sample sizes large enough for analysis? • Oversampling (segmentation) Biased/small sample => Avoid percentages 14

15. You don’t need to measure everything

16. What: before you think about collecting data ask yourself: Do you already collect this data? Use this data (if it is fit for purpose) Yes No Has anyone already proved the causal link you want to make? Is it really important you need data on this outcome? Yes Yes No Does anyone else collect this data? Can you access this data? Yes Yes No No No Use research to limit your data collection Don’t collect unless you decide it is really important Develop your own data source

17. Prioritise outcomes that: • you directly influence (rather than indirectly support) • are important / material to your mission • are not too costly to measure • will produce credible data?

18. How robust does evidence of your social impact need to be? • What do your target stakeholders (investors?) think? • What is possible, given your resources etc?

19. lEvels of evidence Credibility Anecdotes / quotes Case studies Self-reported change Before and after survey Control groups Randomised control trial Basic Advanced Nesta level 2 Nesta level 1 Nesta levels 3, 4, 5

20. 6 tips for measuring your impact 1. Decide what’s important. Define your research question and be clear what you want to achieve. 2. Keep it simple. Concentrate on what you really want to know and don’t try to be too ambitious in the scope of your evaluation. 3. Know your limits. You need to choose an approach that matches your resources. 4. Consider investing in training. If you are not confident, get training or seek external help. 5. Think about the future. Don’t build a system that will quickly go out of date and look at the year-on-year costs. 6. Use the information that you collect: If you don’t use it, don’t measure.

21. Resources • Review of evidence of ICT and learning: http://educationendowmentfoundation.org.uk/toolkit/ict • A good website to look for measures and indicators: http://wilderdom.com/tools/ToolsSummaries.html • How to communicate your results: Hedley, S et al (2010) Talking about results. New Philanthropy Capital. • Bradburn et al. (2004) Asking questions: the definitive guide to questionnaire design. Jossey-Bass: San-Francisco. • http://www.theoryofchange.org/ • The innovation network’s logic model workbook: http://www.innonet.org/client_docs/File/logic_model_workbook.pdf • The Centre for What works (http://www.whatworks.org/) has an outcome portal where you can browse different outcomes and their corresponding indicators. • TRASI (http://trasi.foundationcenter.org/) has a database where you can search different approaches to impact assessment. • NPC has a page on shared measurement where it outlines some indicators for different sectors. http://www.philanthropycapital.org/publications/improving_the_sector/measuring_together.aspx

22. Resources • Survey question bank: A website where you can research widely used surveys and single questions by theme or using key word searches. http://surveynet.ac.uk/sqb/ • CES’s guide to theory of change: http://www.ces-vol.org.uk/downloads/makingconnectionsusingatheoryofchangetodevelopplan-800-808.pdf • Forum for Youth Investment’s guide to measuring soft skills in youth programmes: http://www.forumfyi.org/files/Soft_Skills_Hard_Data.pdf • A list of the most common psychological scales and questionnaires: http://www.ull.ac.uk/subjects/psychology/psycscales.shtml • Harvard Family Research Project’s tools for out-of-school time programmes: http://www.hfrp.org/out-of-school-time/publications-resources/measurement-tools-for-evaluating-out-of-school-time-programs-an-evaluation-resource2 • Bradburn et al. (2004) Asking questions: the definitive guide to questionnaire design. Jossey-Bass: San-Francisco. • Ritchie et al. (2003) Qualitative Research Practice – A Guide for Social Science Students and Researchers. Sage Publications Ltd • Measurement and assessment tools for family and parenting interventions: http://www.actionforchildren.org.uk/media/3986553/outcome_tools_1.pdf

23. Eibhlín Ní Ógáin-eibhlin.niogain@thinkNPC.org NPC - title of the document (can be changed under view/header and footer. Add date here if needed.