Measuring Advice Outcomes: Welsh Government's Approach

1. Advice Services Alliance, UCL 30th November 2018 The Welsh Government’s approach to measuring advice outcomes Richard Thurston, Deputy Chief Research Officer Paul Neave, Head of Advice Services Siân Williams, Senior Research Officer

2. The Well-being of Future Generations Act "The Wales future generations Act captures the spirit and essence of two decades of United Nations work in the area of sustainable development and serves as a model for other regions and countries. ‘One Wales, One Planet' captures it all. We hope that what Wales is doing today the world will do tomorrow. Action, more than words, is the hope for our current and future generations.” Nikhil Seth, United Nations (April, 2015) Gwasanaethau Gwybodaeth a Dadansoddi Knowledge and Analytical Services

3. Design Tips . Gwasanaethau Gwybodaeth a Dadansoddi Knowledge and Analytical Services

4. Design Tips What does this mean for how we do evaluation? We all need to show how the work we do contributes to the Well-being goals and think about how we can mainstream the act into our work programme… The Act sets out a “sustainable development principle” which is about how the public bodies listed in the act should go about meeting their well-being duty under the act. The principle is made up of five ways of working that public bodies are required to take into account when applying sustainable development: Gwasanaethau Gwybodaeth a Dadansoddi Knowledge and Analytical Services

5. Long Term - The importance of balancing short-term needs with the need to safeguard the ability to also meet long-term needs • Prevention - How acting to prevent problems occurring or getting worse may help public bodies meet their objectives. • Integration - Considering how the public body’s well-being objectives may impact upon each of the well-being goals, on their other objectives, or on the objectives of other public bodies. • Collaboration - Acting in collaboration with any other person (or different parts of the body itself) that could help the body to meet its well-being objectives. • Involvement - The importance of involving people with an interest in achieving the well-being goals, and ensuring that those people reflect the diversity of the area which the body serves Gwasanaethau Gwybodaeth a Dadansoddi Knowledge and Analytical Services

6. Advice Services - a little history Gwasanaethau Gwybodaeth a Dadansoddi Knowledge and Analytical Services • Welsh Government’s 2013 Advice Services Review made a number of recommendations, including: • Promoting Advice Networks - maximise resources that fund advice services & ensure the advice service planning is integrated with other services; • Quality Assured Services – promote greater consistency across the sector enabling more partnership working, based on trust amongst providers; • Understanding Demand - a detailed study of the demand for social welfare advice across Wales is required; • New Approaches to Funding Services – funding ought to be longer term and ‘outcome-focused’.

7. Advice Services – the present Gwasanaethau Gwybodaeth a Dadansoddi Knowledge and Analytical Services • Information and Advice Action Plan for Wales– five year plan 2016 - 2021. • No single organisation can successfully implement the IAAP – needs effective collaboration across a range of stakeholders. • Good progress being made to: • champion the preventative approach; • encourage collaboration amongst stakeholders around the commissioning & delivery of services; • explore the development of outcome measures linked to the Well-being of Future Generations Act 2015; • improve the quality of information & advice services by implementing the Information & Advice Quality Framework for Wales.

8. Advice Services – the present Gwasanaethau Gwybodaeth a Dadansoddi Knowledge and Analytical Services • Information & Advice Quality Frameworkfor Wales – aims to: • provide a consistent approach to the provision of quality information & advice accessed by people in Wales; • build on existing quality standards – not reinvented the wheel. • Quality Standard Owners can now have their scheme accredited to the IAQF; • IAQF assesses a Standard Owner’s scheme within seven quality areas: including: • Quality Area 7: Delivering Outcomes – comprises seven overarching outcomes based upon the aspirations contained within the Well-being of Future Generations Act Wales 2015.

9. Measuring Outcomes - Research • Opportunity to explore new ways of doing evaluation in a policy context • Evaluative based approach to: • Develop a theory of change • Capture meaningful indicators that show contribution to outcomes and impacts • Key aims of the research: • To develop a theory of change for advice services • Identity meaningful indicators • Learn about some of the challenges with gathering data • Explore stakeholder relationships Gwasanaethau Gwybodaeth a Dadansoddi Knowledge and Analytical Services

10. The Research • Collaborative process: • Conversation involving policy makers, funders and funded organisations • National Advice Network (NAN) identified as participant group • Work is ongoing • Research undertaken in October 2018 – a work in progress • Three stages: • Online questionnaire to NAN members • Facilitated workshop using MeetingSphere to capture detail • Online workshop for 10 days post-event to allow additional information to be added Gwasanaethau Gwybodaeth a Dadansoddi Knowledge and Analytical Services

11. Theory of Change External factors and dependencies Gwasanaethau Gwybodaeth a Dadansoddi Knowledge and Analytical Services

12. Measuring Outcomes – key messages • Consistency and clarity – what are we measuring and how? • Tracking of clients – lots of considerations • Monitoring should not be at expense of delivery • Not all outcomes can be measured - realistic • Attribution – can we attribute outcomes to activities? Not always a link with quality • Longitudinal data collection • Counterfactual challenges Gwasanaethau Gwybodaeth a Dadansoddi Knowledge and Analytical Services

13. What next for the research? • Research is the start of a process • Feed back to NAN and Advice Services Commissioning Group • Use of findings to: • Inform policy development going forward • Develop of an outcomes based monitoring and evaluation framework • Consider support for monitoring and evaluation for advice services • Within the principles of the Well-being of Future Generations Act for Wales Gwasanaethau Gwybodaeth a Dadansoddi Knowledge and Analytical Services