Monitoring and evaluation of carers’ services and projects

1. Monitoring and evaluation of carers’ services and projects Dr Andrea Wigfield - Associate Professor of Social Policy Centre for International Research on Care, Labour and Equalities (CIRCLE), University of Leeds Michael Shann – Carer Ambassadors Programme Manager, Carers UK Supporting Carers in Primary Care National Learning Event, Manchester, 16 April 2013

2. Why evaluate? • For learning and development • For accountability • To demonstrate value for money to supporters • Motivation of staff and volunteers • To save time and money

3. Who is measuring impact? • 74% of charities say they are investing more in impact measurement than five years ago • Change in funder requirements and priorities of trustees/senior management are main reasons for increasing impact measurement • BUT, main benefits are in improving services and being able to better demonstrate results (Source: Making an Impact (Oct 2012), New Philanthropy Capital

4. What do we want to find out? • Are we achieving our planned outputs and outcomes? If not, why not? • What has worked well and not so well? • Are we reaching the carers and other stakeholders we want to? • What do carers think of our services? How can we improve them?

5. Monitoring & Evaluation Terms • Monitoring – routine collection of data • Evaluation – using data we collect to make judgements about our projects and services • Inputs – resources we put into projects & services • Outputs – products and services we actually deliver • Outcomes – changes and benefits we achieve • Impact – effect at a higher level, often the result of contributions from several organisations • Quantitative – numbers of things that happen • Qualitative – people’s views and perceptions • Indicators – measurements we can use to determine whether outputs and outcomes are being achieved

6. The Logic Model

7. The 5 Step Approach to Monitoring and Evaluation Source: First Steps in Monitoring & Evaluation (2002), Charities Evaluation Services

8. Identifying outputs & outcomes

9. Identifying indicators (which data to collect) • Keep it in proportion – to size of project / service • Keep it manageable - time needed to collect data • Quantitative and qualitative • Consider existing information gathering systems – do these need adapting • Where are the gaps? What new data needs to be collected?

10. Group 1:What do you currently measure and how do you collect the data? What outputs do you measure and how? What outcomes do you measure and how? What difficulties have you experienced in measuring outputs and outcomes?

11. Group 2:Evaluation Challenge A GP Liaison Worker has provided a carer awareness training session to a group of Practice Managers in their area. The main aim of this session was to inform the group of local carers’ services that are available. What output and outcome data would you gather to evaluate whether the session was successful?

12. Group 3:Evaluation Challenge A GP Champion (GP recruited by RCGP) and Carer Ambassador (volunteer recruited by Carers UK) have been advising a Clinical Commissioning Group on how to set up a Carers Partnership Board in their area. What output and outcome data would you gather to evaluate whether their contribution was successful?

13. Group 4:Evaluation Challenge Carers Trust have set up a new Facebook group for GP Liaison Workers to encourage the sharing of information, ideas and good practice across the country. How would you evaluate the benefits of having this Facebook group?

14. Group 5:Evaluation Challenge A GP Liaison Worker has established links with 12 GP practices in their area. Some of the GP practices have made changes to the ways they identify and support carers and refer them to local carers’ services. How would you measure the value of the GP Liaison Worker’s intervention?

15. Internal or External Evaluation? • Factors to consider: • How important is it to have an independent assessment of your project/service? • Has your funder requested an external evaluation? • What proportion of your project/service budget can you allocate to evaluation? • What is appropriate for the size/budget of your project/service? • How much staff time will be involved?

16. Reporting to Donors • When applying for funding, ask about reporting requirements • Include donor requirements in your evaluation plan • Donors will be looking for evidence, but you could demonstrate progress using qualitative data as well as numbers • An interim report could focus more on outputs with some indication of progress towards outcomes • Learning from the project may make you want to make changes to the original objectives – this may be fine as long as you deliver the intended outcomes

17. Evaluation Good Practice • Write an evaluation plan when you are planning your project or service • ‘Proportionate and appropriate’ – don’t measure everything • Build impact measurement into job descriptions and appraisals • Take action on your results • Develop a commitment to continuous improvement • Communicate your impact • Don’t be afraid to evaluate

18. Contact Andrea or Michael A.Wigfield@leeds.ac.uk michael.shann@carersuk.org