Consistent performance measures for post-16 learning

1. Consistent performance measures for post-16 learning November 2018

2. The measures Agenda • Achievement • Introduction and background to the measures • Destinations -discussion groups and feedback • Value added • Post-16 value added • Destinations

3. The measures The aim • Achievement • To develop a new set of learner outcome measures that can be applied consistently across FE and sixth forms, to inform: • Performance monitoring and improvement • Estyn inspections • Learner and parental choice of institution and learning programme • Evaluation of Welsh Government policies and investments • Post-16 value added • Destinations

4. The measures • Achievement • Value added • Destinations

5. The measures The story so far • Achievement • Post-16 value added • Destinations

6. The measures What you have now • Achievement • Achievement reports for 2016/17 – via DEWi • Value added reports for 2017/18 – via Aspire Analytics • Destinations reports for 2015/16 – via DEWi • All-Wales statistics on the new measures for 2016/17 (published 20 Sept-18) • Post-16 value added • Destinations

8. Value added reports 2017/18

9. Destination reports 2015/16

11. General education programmes [adjusted figures]: • Around a quarter of learners who start AS levels do not progress to their second year of A level study. • Just over half of A level students achieve three A levels and equivalent qualifications at grades A*-E.

13. Vocational achievement: minor changes: • Age split • Inclusion of terminated activities • Welsh Baccalaureate: • Achievement measure still under review • Looking at inclusion of ‘% enrolled’ for context • Achievement data tables issued to schools and colleges based on 2017/18 • Future annual cycle: publication in February

14. Value added • 2017/18 provisional release issued 7 November 2018 • Improvements included: • Better guidance • Vocational &academic breakdown at headline level • Three year trends • Streamlined dashboard & sites • New filters • 2017/18 Final release: End of February 2019

15. Value added – next steps Scoping: • Rankings & gauges • Older learners • Correlation cut off

16. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 2014/15 learners 37% 21% 24% 82% 2015/16 learners 34% 22% 26% 81% Total sustained employment Total sustained learning Sustained employment only Sustained learning only Sustained employment and sustained learning Destinations of FE, sixth form and WBL learners: • In 2015/16 116,080 learners finished a learning programme (sixth forms, colleges and work-based learning) • Of those 81% had a sustained positive destination of further learning and/or employment.

17. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% All learners 34% 22% 26% 81% FE/school learners 25% 25% 32% 83% WBL learners 64% 9% 5% 78% Female learners 36% 24% 24% 84% Male learners 31% 19% 29% 79% 18 and under 19% 27% 37% 83% 19 to 24 52% 15% 13% 79% 25 to 49 58% 14% 8% 80% 76% 50 and above 60% 9% 7% Sustained employment only Sustained employment and sustained learning Sustained learning only 2015/16 breakdown:

18. Consultation on the methodology at regional workshops finalised methodology for 2019 publication • Ongoing work to identify themes for final analysis • Future annual cycle: publication in September for learning programmes that ended two years before

19. The measures What to look out for • Achievement • February: Achievementreports for 2017/18 – via DEWi • February: Final value added reports for 2017/18 – via Aspire Analytics • February: All-Wales statistics on achievement and value added measures for 2017/18 • Post-16 value added • Destinations

20. The measures Post 16 Choices • Achievement • Site similar to ’My Local School’ • Comparable performance information • Post-16 value added • Destinations

21. The measures Key concerns • Data quality/timeliness • Awareness raising • Unintended consequences • Limitations of the measures • Reporting and benchmarking • Achievement • Post-16 value added • Destinations

22. The measures Post-16 collection • Data quality issues: • Welsh Baccalaureate/Skills Challenge activity completion • AS level programme completion • DEWi validation report • Achievement • Post-16 value added • Destinations

23. The measures DEWi summary and validation report • Achievement • Post-16 value added • Destinations

24. The measures DEWi summary and validation report • Achievement • Post-16 value added • Destinations

25. The measures DEWi summary and validation report • Achievement School Local Authority Post 16 Collection Summary/ Validation Report LA/School no: xxx/xxxx • Post-16 value added • Destinations

26. The measures DEWi summary and validation report Post 16 Collection Summary/ Validation Report LA/School no: xxx/xxxx School Local Authority • Achievement • Post-16 value added • Destinations

27. Questions?post16quality@gov.wales

28. Consistent performance measures for post-16 learning: Destinations November 2018

29. Consistent performance measures: achievement, value added and destinations • Experimental statistics released in September • Based on the 2016/17 destinations of those finishing learning programme in 2015/16 • Feedback welcomed

30. Destinations can be broadly categorised into two types: • Education • Employment

31. Destinations can be broadly categorised into two types: • Education • Employment Matched education dataset

32. Destinations can be broadly categorised into two types: • Education • Employment PLASC Matched education dataset LLWR WED Post-16 Collection HESA

33. Destinations can be broadly categorised into two types: • Education • Employment Employment (HMRC) Longitudinal education outcomes (LEO) dataset LLWR Benefits (DWP) Post-16 Collection

34. Administrative data: no additional burden • 98 per cent of learner records matched to DWP/HMRC Limitations: • No information on hours worked or employment sector • Sustained employment measure includes part-time workers • Don’t know if employment destination is related to programme of study

35. December: Latest LLWR and Post-16 data available • February: Data sent to DWP for matching • June: Matched data returned from DWP • September: Destinations statistics published July 2016: The 2015/16 cohort finishes education Sept 2018: Destinations published for that cohort

36. LLWR (FE/WBL) Learners finishing learning programme 2015/16 Post-16 Collection (sixth forms) Do they appear in these datasets? Matched education dataset LEO dataset 2016/17 (e.g. continuing in FE, entering HE) (employment)

37. 2015/16 ‘leaver’ cohort: • Post-16 learners that terminated a learning programme • early drop outs and Adult Community Learning excluded • most recently started programme (where learner has multiple terminated programmes) England – two separate releases: • 16 to 18 year olds completing A levels or other level 3 qualifications • all age apprenticeships, all age traineeships, and adult (19+) FE and Skills learners that completed a funded learning aim

38. Consistent with the definition used in England: • A learner must be in paid PAYE employment for at least one day a month in five out of the six months between October 2015 and March 2016; or • A learner must have completed a self-assessed return for tax year 2015/16 stating that they have received income from self-employment

39. A learner must be in learning for at least one day in each of the six months between October 2015 and March 2016, at the same or higher level than their terminated programme England: ANY level of destination learning accepted

40. Sustained employment; or • Sustained learning; or • Combination of both

41. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 2014/15 learners 37% 21% 24% 82% 2015/16 learners 34% 22% 26% 81% Total sustained employment Total sustained learning Sustained employment only Sustained learning only Sustained employment and sustained learning • In 2015/16, 116,080 learners finished a learning programme (sixth forms, colleges and work-based learning) • 81% had a sustained positive destination of further learning and/or employment

42. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% All learners 34% 22% 26% 81% FE/school learners 25% 25% 32% 83% WBL learners 64% 9% 5% 78% Female learners 36% 24% 24% 84% Male learners 31% 19% 29% 79% 18 and under 19% 27% 37% 83% 19 to 24 52% 15% 13% 79% 25 to 49 58% 14% 8% 80% 76% 50 and above 60% 9% 7% Sustained employment only Sustained employment and sustained learning Sustained learning only 2015/16 breakdown:

43. Earnings • Benefits

44. Have you had the opportunity to look at the data tables for your school/college? Did you find them useful and why? What changes do you think would be helpful to make to the methodology or terminology? For example: • Should the cohort for the measure be terminated or completed learning programmes? • Should we use the terminology ‘positive destination’? If not why not and is there any other terminology you would use instead? • Should we introduce an ‘immediate destination’ measure alongside a ‘sustained’, for example learners in learning or employment (or both) for one day in the October? • Are there any other changes you would introduce?

45. Is there any additional breakdown of the data you would like included? Given the rich possibilities for analysis are there any topics you think would be helpful for Welsh Government to look at eg: • Welsh language, • More able and talented learners, • Deprivation

46. Post-16 Value Added Gareth Moore (Wales Programme Manager, FFT) Andrew Bibby, (Data Analyst, FFT)

47. Workshop agenda • Measuring learner progress (value-added) • Understanding the data being used • Overview of the reports and changes made for 2018 • Future developments • Feedback and questions

48. Measuring learning progress (value-added) Value-added indicators have three main purposes: • Assessment of relative learner progress: self evaluation by Providers • Measure of accountability for the effectiveness of education services • Basis for learner and provider target-setting and improvement planning

49. What value-added is ….. • An indicator of learner attainment relative to that of ‘similar’ learners • ‘Similar’ learners defined, as a minimum, by their KS4 prior attainment (KS4 PA) • The reference to ‘similar’ learners can be refined to reflect other learner characteristics and Provider background factors (to give Contextual VA – CVA - indicators) where these have a statistically significant association over and above KS4 PA

50. What value-added is not …. • A measure of the effectiveness of Providers * • Danger is that some might assume it is! * unless a substantial amount of additional data is collected and complex statistical models are used