H ighlight the advantages of being able to demonstrate impacts and outcomes

1. East Lothian Learning Partnership Conference Demonstrating Impacts and Outcomes – Is it really all that important? Ann Kivlin & Colin Forbes, CLDS5th December 2011

2. Session AimsHighlight some of the advantages of being able to demonstrate impacts and outcomesExplore: WHY we need to do it, WHEN we should do it, WHAT is required in order to do it effectively WHO has responsibility.Explore and discuss a range of tools and techniques

3. Highlight the advantages of being able to demonstrate impacts and outcomes How confident are you? Extremely confident More than confident Confident Less than confident Find it a challenge Not confident at all

4. Highlightingthe advantages of being able to demonstrate impacts and outcomes • Hands up how many of you feel confident enough to facilitate a session on this topic with your staff teams?

5. mins • Settle in and intro 5 • Why, When, What, Who 40 • Building your Capacity 10 • Tools and Techniques 10 • Questions 10

6. WHY we need to demonstrate impacts and outcomes?WHEN we should be demonstrating impacts and outcomes?WHAT is required in order to do it effectively and demonstrate impacts and outcomes?WHO has the responsibility for demonstrating impacts and outcomes?

7. Group Task:In 4 groups spend 5 minutes considering one of the questions and then feedback your groups 3 most important answers

8. WHY - A list we prepared earlier! Value for money Spending Public Pound Accountability Transparency Measure improvement Report progress to stakeholder Proof the work makes a difference

9. WHEN - A list we prepared earlier! When you feel like it! For every piece of work? Significant pieces of work Best bits! Worst bits? Regularly? When your line manager requests!

10. Ability to prove what you are doing is really making a difference Use a range of effective tools Ability to measure before and after and monitor performance Agree what is good evidence Provide the evidence Confidence WHAT - A list we prepared earlier!

11. WHO - A list we prepared earlier! It’s everybody’s responsibility! Lead person

12. Building your Capacity in demonstrating Impacts and OutcomesTo increase staff understanding and confidence in monitoring and evaluating activities through constructing clear outcomes, indicators and methods which can demonstrate impact of their engagement. 27 SLIDES!!

13. Your Thoughts? • What do you think when you hear the word ‘evaluation’?

14. What are outcomes and why are they important?Discussion

15. The definition! Outcomes are the changes or difference you want to make through your services or activities (outputs)

16. Why dooutcomes matter? Focus: You need to know what difference you want to make before you can evaluate to see if you have. Expectations: The people who use your services will know what to expect from you Motivation: Staff satisfaction is increased when they know they make a difference. Accountability: Many funders want to fund or purchase outcomes. They want to know what difference their money makes and how it helps target groups have a better life.

17. The science bit OutCOMEs are the changes and differences that come out of your activities. OutPUTs are the activities or services you put on for your users. Inputs are the resources you put in to do the work: money, people, time, premises, equipment. input output  outcome  impact

18. output outcome Outcomes are lazy Outcomes don’t happen by themselves They need activities, services, people The output makes the outcome happen

19. Relevance Outcomes must be relevant to the activity What you do (activity) must link to youroutcome in a logical way

20. DEFINITION Outcomes are the changes or difference you want to make through your services or activities (outputs) GOOD wordsLESS SPECIFIC words  e.g. improve e.g. support

21. Outcome words decrease improve maximiseminimise betterexpand reduce enhance more less produce increase strengthen sustain - not exhaustive list - about change or difference

22. These words don’t show change!! Support Assist Engage Encourage Enable Help Offering Seeing Watch out for verbs (doing words) – they are about activity not change BUT you could include these words as long as you also describe the change e.g. – ‘INCREASE levels of community participation by OFFERING crèche facilities to local families’.

23. Expressing the outcome is changing (e.g. the service user, beneficiary, organisation, community etc.) Who is changing (e.g. knowledge, skills, environment, feelings etc.) What it will change (e.g. increase, improve, reduce, etc.) How

24. Examples of outcomes WHO WHATHOW • Parents’confidence isincreased • Young people have reduced, stabilised or eliminated their drug use • Communitiesarebetter represented in decision making

25. Examples of outcomes WHO WHATHOW Older Asian women have better access to community health and social care services There is an increase in the understanding of the recycling procedure within the office staff. Young carers are able to make informed choices about the future

26. The ‘so what’ test • I deliver an Employability Course at Ross High School - SO WHAT? • I work with a group of 16 S4’s -SO WHAT? • The group has visited colleges and taken part in indoor climbing sessions -SO WHAT? • The group have learned what courses are available at local colleges and how to access these courses  OUTCOME • The group have gained confidence and improved communication & team working skills through indoor climbing sessions  OUTCOME All important … but not all outcomes

27. AIM Outcomes = change • Skills • Knowledge • Behaviour  Feelings … SO WHAT? Outputs Activities and Services the main things you do with your users / clients Weaver’s Triangle

28. Outcomes and impact Lochniven Young Carers Impact (aim) – ‘Our young people are successful learners, confident individuals, effective contributors and responsible citizens’ Outcomes need to be meaningful and realistic steps on the way.

29. Example Lochniven Young Carers Project • Short term: Young carers know where to go for support & are confident to do so • Medium term: Young carers are less isolated and better able to cope • Longer term: Young carers participation and attainment at school is improved Impact – Our young people are successful learners, confident individuals, effective contributors and responsible citizens

30. The outcome test Outcomes • Are about change or difference. • Don’t happen by themselves – they are not the activity – they are the result of the activity. • Must link logically to the activity. • Use words like improve, decrease. • Answer the ‘so what?’ question. • Should be realistic – achievable in a few years. • Should be (largely) in your power to deliver. • Should be simple – not too many (3-5)!

31. Collecting information • Monitoring what you do (e.g. numbers, range of activities) OUTPUTS AND • Evaluating whether your activities are delivering your OUTCOMES

32. Measuring outcomes Thinking about each outcome: What would it look like if we achieved it? • These are your outcome indicators • What are the 2-3 most important (and most likely) indicators for each outcome? • Involve users and stakeholders. • Your indicator list = your evaluation plan

33. Getting it right • Measure indicators more than once to show change. • What is your baseline (the starting point) for the service and for individuals (from referral info, assessment, evidence of need etc) • Don’t collect information on all outcomes all the time. • Do you need to ask everyone or can you sample? • Follow up some users if you can to find out what happened next.

34. Getting it right 6. Use one system – but a range of methods. 7.Build into planning – and if possible activities. 8. Test your methods. 9. Be honest and involve clients. 10.Record unexpected outcomes (e.g. 6 volunteers got a job)

35. Common sources of info on indicators

36. Methods for collecting info • Questionnaires • Interviews • Observation • Group structured feedback • Wall charts • Visual progress tools • Mapping • Third party feedback • (Video) diaries

37. Outcome Evaluation Plan

38. Getting Started - understanding what you want to achieve. Setting and agreeing your outcomes Collecting information - what are you going to measure and how? Getting systems in place Learning from your findings - What did you learn? How will you do things differently? Any unexpected results? Analysing and Reporting - making sense of what you’ve collected and reporting on progress and learning. The evaluation pathway Acknowledgement: Evaluation Support Scotland

39. Tools & Techniques • Wide range of evaluative tools • Electronic Project Files • VOICE • National Standards for Community Engagement

44. Presentation adapted by

45. What is VOiCE? VOiCE is online planning and recording tool designed to assist individuals and organisations to design and deliver effective community engagement.

46. Why use VOiCE? • It’s simple to use • Provides evidence on meeting the National Standards for Community Engagement • You don’t need previous knowledge of the National Standards or experience of community engagement to use VOiCE • Online link to the National Standards for Community Engagement: www.scdc.org.uk/what/national-standards/10-national-standards/

47. How can VOiCE help you? • Analyse the need for community engagement • Plan your engagement • Do it – implement your plan effectively • Review the outcomes VOiCE enables you to monitor and record the process

49. Features of VOiCE • Multi user environment • Secure password protected accounts • Provides reports • Strategic analysis

50. Want support to use VOiCE? VOiCE training is supported by East Lothian’s Community Engagement Strategy action plan which is available online at: www.eastlothiancommunityplanning.org.uk/engagement Key contact: Meriel Deans, Public & Community Involvement Co-ordinator This is a shared post between East Lothian Community Health Partnership and East Lothian Community Planning Partnership Email: meriel.deans@nhs.net Tel: 0131 536 8025 or 07738 075232