WWF Webinar 2 Generating useful evidence # WWFSharing

1. WWF Webinar 2 Generating useful evidence#WWFSharing

2. Welcome! • The webinar will start at 12.15 • Please use this time for a sound check • Raise your hand so that we can see you can hear us #WWFSharing

3. Welcome and introductions Professor Sharon Collard Evaluation & Learning Partner Peter Bailey Money Advice Service Louise Scott Evaluation & Learning Partner

4. Shaping future financial capability services • The WWF aims to strengthen the evidence base and help identify ways to demonstrably improve people’s financial capability • It will be used by the new SFGB and other funders to inform commissioning in the future • It is intended to inform the future delivery of fincap interventions however they are funded • It is intended to develop sector expertise and capacity in designing, delivering and evaluating services • Strengthen evidence base • Inform future commissioning • Inform future delivery • Grow sector capacity to design, deliver and evaluate services #WWFSharing

5. Helping you to generate useful evidence • MAS tools and resources e.g. evaluation toolkit, outcome frameworks, Impact Principles • One-to-one support via Relationship Managers • Structured feedback on Interim Evaluation Reports via Relationship Managers • Guidance and templates, including guidance on analytical approaches and final report template • Webinars • Events • MAS expects you to deliver: • Good-quality analysis and, where possible, evidence of impact • Final evaluation report of publishable standard • Evidence Hub summary of publishable standard #WWFSharing

6. At the forefront of building financial capability knowledge #WWFSharing www.fincap.org.uk/evidence_hub

7. Living up to the IMPACT Principles • IMPACT champion • Mind the gap • Properly resource it • Add to the wider evidence base • Consistency when measuring outcomes • Take time to understand the results and their implications • Consider evidence at every stage of design and delivery • Take into account gaps in provision and knowledge • Allocate sufficient resources to evaluation • Commit to sharing findings – on the Evidence Hub • Use the Financial Capability Outcomes Frameworks • Learn from/act on lessons from your evaluations #WWFSharing

8. Today’s webinar The webinar is made up of 3 parts: • Generating useful evidence • Analysing and reporting • Managing your evaluation Please send us your questions and comments throughout the session and we will respond to as many as we can. • Please type in your questions and comments #WWFSharing

9. Generating useful evidence Project presentations

10. Money Workshops in Schools Randomised Controlled Trial • Stephanie Hayter • The Money Charity • themoneycharity.org.uk • steph@themoneycharity.org.uk

11. The Money Charity is the UK’s only financial capability charity that works with everyone We empower adults and young people across the UK to build the skills, knowledge, attitudes and behaviours, to make the most of their money throughout their lives. Workshops in schools RCT

12. How do we do this? Policy Adults Industry Young people Workshops in schools RCT

13. Our intervention - Money Workshops in schools • FreeMoney Workshops • Key Stage 4 and Post 16 (14-19 year olds) • Reached over 160,000young people since 2010 • England, Wales and Northern Ireland • PHSE, Citizenship, Maths, or off timetable days • Classroom sized groups • Delivered by trained expertfreelance consultants • Modular approach - Flexibility Workshops in schools RCT

14. Our intervention - Money Workshops in schools Workshops in schools RCT

15. Our intervention - Money Workshops in schools • Our workshops encourage young people to talk about money, and not to be afraid of asking questions. • They are: • Interactive • Aspirational • Experiential • Fun • Positive • Engaging • Confidence building Workshops in schools RCT

16. Our WWF Project • What? • Randomised controlled trial • Interviews with teachers, consultants and students • Who? • Key Stage 4 and Post 16 students (14-19 year olds) • 59 schools/colleges involved (69 phases) • Where are we now? • Finished delivering workshops to intervention schools • Students currently completing their follow-up surveys Workshops in schools RCT

17. Our WWF Project Primary impact evaluation questions Q1: Do The Money Charity Key Stage 4 and Post 16 Money Workshops have an effect on students’ financial capability when compared with a business as usual control? Does this differ across ages? Q2: Are particular outcomes more likely to be sustained than others? (as measured at 3 month follow up stage) Process evaluation questions Q1: How consistent is the quality and content of the delivery between schools? Q2: What factors constrain or support a school’s support of financial capability workshops and in what ways? Q3: To what extent is the intervention model scalable (as measured by admin data and teacher qualitative information)? Q4: To what extent do students perceive the intervention resources and training to be effective? Workshops in schools RCT

18. Working together – Us, Our Evaluators & our ELP • Our Evaluators • National Foundation for Educational Research • How we work together • Weekly conference calls • Sign off process • Close dialogue • TMC manage all relationships; NFER manages schools’ data • Collaborative partnership (but independent analysis) Workshops in schools RCT

19. Client Name The Money Charity and NFER RCT, cont’d. WWF Webinar on generating robust evidence and working with an evaluation partnerJanuary 2018 • Pippa Lord, Senior Trials Manager • National Foundation for Educational Research • https://www.nfer.ac.uk/schools/taking-part-in-our-research/MCWS/ • p.lord@nfer.ac.uk

20. What is an RCT? Workshops in schools RCT

21. Research design - generating robustness • Key design features • RCT, with mix of quantitative and qualitative process evaluation alongside • Developing a robust outcome measure • Informed by MAS outcome framework, The Money Charity Theory of Change, MAS WWF reporting framework • Protocol for reference • Key approaches • Collaborative in approach …. independent in analysis Workshops in schools RCT

22. Expressions of interest from schools/colleges NFER recruit KS4 and post-16 institutions; schools/colleges sign up to trial Bookings process – identify year group(s), select workshop(s), suggest dates Schools confirm student Ns, students complete baseline survey RANDOMISATION Confirm workshop dates Confirm workshop dates Schools/students receive workshop(s) Observe workshops; Teacher survey Students complete outcomes survey Students complete outcomes survey Schools/students receive workshop(s) Case studies

23. Research realities – tackling some challenges • RCT challenges • Fitting trial and programme design together without compromising programme or trial ... too much • Lower school numbers than expected ... analysis of KS4 and Post-16 combined trial • Developing the outcome measure as part of the trial ... pilot cognitive interviews, specify using baseline data only • Follow-up … conduct intention-to-treat (treatment effect across all participants) and on-treatment analysis (takes account of who attended which workshop(s)), check for any systematic differences/imbalance in any drop-outs • Achievements and limitations • All MAS WWF analysis and reports to be seen in light of limitations of studies ... but also recognising the achievements made Presentation Title

24. Thank you! • Stephanie Hayter • The Money Charity • themoneycharity.org.uk • steph@themoneycharity.org.uk Pippa Lord NFER Nfer.ac.uk p.lord@nfer.ac.uk

25. Evaluation of Kick Start programme Steve Korris MyBnk

26. About KickStart Money Evaluation KickStart Money is a collaboration of 20 leading savings and investment firms aiming to take financial education to primary school children across the UK, catalysing a movement to build a national savings culture for the future. Intervention designed and delivered by MyBnk. Evaluation conducted by MyBnk and external evaluators Substance

27. Developing an evaluation for an emerging project Conclusions on...

28. Education inputs Intervention aims to influence executive functions

29. Evaluation methodology

30. SamplingMethodology Sampling by

31. Changes to original evaluation methodology Changes to the structure of control group methodology Revising and enhancing delayed gratification test Focus sampling efforts around case study schools Changes to practicalities of data collection

32. Reflections so far Lead in times around emerging projects and new evaluations You never know before you try! Interim results are promising. Challenge to come will be to structure and articulate findings from across multiple data sets

33. Get in touch www.kickstartmoney.co.uk steve@mybnk.org www.mybnk.org

34. Questions on generating useful evidence ** Please type in your questions for the project presenters and we will read out a selection of these **

35. Analysing and reporting WWF data:Things to think about Professor Sharon Collard Evaluation & Learning Partner

36. Quantitative data • Quantitative data, e.g. data from survey questionnaires or monitoring data, is analysed using Excel, SPSS etc. • Can you go beyond simple descriptive analysis to analyse changes and the extent to which any change can be attributed to your intervention (i.e. statistical analysis)? • Is the sample size large enough to conduct statistical analyses e.g. to draw inferences about the population as a whole? (min. 50 cases). What about the quality of your data e.g. missing cases? • In your statistical analyses, can you draw comparisons between: sub-groups of participants at one point in time; before and after scores for the same project participants; before and after scores for project participants and a comparison group? • How confident are you that any observed changes are (a) statistically significant (b) a result of the intervention and not some other factors? • Your purpose in conducting analysis is to answer the project’s research questions and test your theory of change. • To do this, you should thoroughly interrogate the data you have collected, using appropriate analytical approaches. • Be careful not to over-state or over-interpret outcomes and impacts. • Consider what factors help explain your project’s outcomes and impacts e.g. profile of users, implementation of the service. #WWFSharing

37. Qualitative data • Qualitative data analysis, e.g. data from interviews or focus groups, is analysed thematically and used to help understand or challenge quantitative data analysis. • Qualitative data analysis helps explain ‘how and why’ the evaluation outcomes and impacts have come about (or not). • Is the sample size large enough to understand similarities and account for differences? (min. 10 cases). • Is your qualitative data analysis systematic, structured, transparent? • Has the basic process of qualitative data analysis been followed? This involves: organising and preparing data; coding and describing data; identifying themes and concepts; interpreting and providing meaning that is grounded in the data. • Verbatim quotes and case studies are useful to support the narrative developed from the data analysis – but use sparingly to maximise their effect. • Your purpose in conducting analysis is to answer the project’s research questions and test your theory of change. • To do this, you should thoroughly interrogate the data you have collected, using appropriate analytical approaches. • Be careful not to over-state or over-interpret outcomes and impacts. • Consider what factors help explain your project’s outcomes and impacts e.g. profile of users, implementation of the service. #WWFSharing

38. Managing your evaluation Louise Scott Evaluation & Learning Partner

39. The reporting process • Moving into the analysis and reporting phase it is important to ensure that evaluators have a clear understanding of expectations. • Ensuring a quality product is a shared responsibility between the grantee, their evaluator and the ELP. • The final report must be of a publishable standard, but should also be of value to your organisation (and partners). • Before the report is produced, meet with your evaluator to discuss/reconfirm expectations. • Review the draft report and make a note of any questions or concerns to feed back to the author. • Ask yourself, ‘are you happy with the quality of the report?’ • How would you describe the relationship with your evaluator? • In hindsight, is there anything you would change about the approach you have taken? • What are your top tips on management and co-production of an evaluation? #WWFSharing

40. And finally…… Professor Sharon Collard & Louise Scott Evaluation & Learning Partner

41. Next steps • All participants will receive a feedback form. • Slides and a recording of the session will also be made available, as well as a blog post. • Updated reporting template and guidance on analytical approaches will be circulated in the coming days. • Webinar 3 will take place in March (save the date email to follow) – this will focus on sharing and disseminating your learning. • Blog: • http://www.fincap.org.uk/blog • Interested in submitting a blog post? • Contact: Ben.Walker@moneyadviceservice.org.uk #WWFSharing