Care to Share?

1. Care to Share? Capturing learning from the Short Breaks Fund Graeme Reekie, Evaluation Support Scotland 0131 243 2770 graeme@evaluationsupportscotland.org.uk

2. Background • Learning: Carers and cared-for Funded Groups Shared Care Government “What works, for whom, in what circumstances?”

3. The Plan! • Quick reminder on outcomes • Measuring soft outcomes • (using indicators) • Evaluation tools, including participatory • Evidence • Reporting • Planning for action

4. Reminder Outcomes: • The change or difference • That ‘comes out’ of your activities Tips: • Use change words (e.g. increase, reduce) • Ask ‘So What?’

5. Indicators Want to: • Involve users? • Spread the burden? • Strengthen your evidence? • Make better use of evaluation tools? Here’s how…!

6. Soft outcomes Need to break down into proxy indicators  Take each outcome. What would it look like if we achieved it? Pick 2-3 most important, most likely, easiest to collect Draw out the methods

7. Indicators must be … • Specific • Simple • Recognisable to your service users • Able to be measured more than once and show change • Ideally you need a baseline in order to measure progress or change.

8. Writing from the perspective of the user If it helps, write indicators from the perspective of the user: I feel … I do… I can ….We can…They can

9. Outcome Outcome indicator A Young people feel more confident 1 I feel surer of myself and less nervous about doing stuff 2 I make suggestions 3 I can take part in activities. 4 I have made new friends. 5 If I have to I find it easier to stand up to people who are mean to me. B Young people are better able to manage their anger and frustration and anxiety 1 I am less scared about things than I used to be 2 I find it easier to deal with the tough stuff going round my head 3 I can keep my temper when I need to 4 I don’t feel sad as much as I used to 5

10. Common sources of information on indicators

11. They tell you Third party tells you Behaviour changes Records Hard evidence Assessment Record casual feedback, Questionnaires, Interviews, Focus groups, Video diaries, Exercises, mapping and visual progress tools Observation notes Attendance sheets, Record of activities, Use of services Tracking what happens, Comparing statistics (eg illness, employment, debt, truancy) Badges, checklists, standards Common methods

12. Traditional methods People tell you: • Questionnaire • Interview • Focus group Either stand alone OR Part of activities

13. Questionnaires • Your questions must relate to your outcomes and indicators. • Short and simple - one piece of paper. • No leading questions! (‘How good is the service?’). • Think about closed or open questions. • Think about scales: numbers, smiley & sad faces etc • Stamped addressed envelopes and give a deadline. Offer rewards for replying. • Do you need to get information from everyone or can you select a sample. • Try it out first to check it makes sense (piloting).

14. Interviews Tips for group interviews: • Your topic guide must relate to your outcomes and indicators. • If possible use a colleague to facilitate if you need very personal feedback. • Give info in advance about the times, venue, expenses. • Provide refreshments, introductions, ice breakers. • Keep the discussion moving on when issues have been addressed. • Do not let one person dominate the group, invite less vocal people to offer their views. • Include group interview as part of an existing group activity?

15. Other methods • Observation • Group discussion (set and measure indicators) • Diaries & video diaries • Third party feedback (referrers, family members etc) • Visual representations of progress – eg trees, wheels, dart boards, ladders • Body maps • Time lines More info on our website

16. Relationship map (social network) At the start of the project At the end of the project Tom Dad Liz Mary Carol Susan Callum X X Liz Tom Mum Di Carol Mum Andrew

17. Body maps: Example 1 Before participation

18. After participation

20. Balanced Wheels • Ask people to shade portions of a circle • Could represent the importance of different things to them • Or time spent

21. Exercise The Short Breaks Fund generates increased understanding of the role of short breaks and respite in supporting caring relationships. What would that look like?

22. How do you like your evidence? • Poached? • Scrambled? • Hard boiled? • Fried? • All in one basket?

23. Evidence comes from a variety of sources Your can build pictures using those different sources So that you can identify: what outcomes for whom in what circumstances and why Jigsaw or mosaic?

24. Where do you get evidence for your decisions? The Esoteric Journal of Perpetual Enquiry, vol 5349? The last person you spoke to?

25. Take your evidence and TREBL it! • Transparency: methods clear, limitations acknowledged • Relevant: up to date, appropriate • Enough: strength of evidence v proportionality • Believable: accurate, representative, reliable • Legitimate: coming from the right sources

26. What goes in a report? • Original aims • Planned outcomes and activities • Headline achievements • Main facts about activities • Main facts aboutoutcomes • What went wrong and why? • Case studies • Learning and changes for the future • Financial information • Appendices

27. Scotland Funders’ Forum The best reports come from organisations that can • evidence their work • tell their story • use reporting to reflect on what they have achieved and learned (Scotland Funders’ Forum: Harmonising Reporting Working Group 2010)

28. Summary • Outcomes: What ‘comes out’? So What? • Indicators: What would that look like? • Tools: Traditional, maps, wheels. • Evidence: TREBL • Reports: Telling the story • More help on our website: • www.evaluationsupportscotland.org.uk