Introduction to Value-Added Data

1. Introduction to Value-Added Data Dr Robert Clark

2. Theory and Stats bits…

3. . Measuring Value-Added – Terminology Exam grade -veVA +ve VA Residuals BASELINE SCORE VA Trend Line/Regression Line

4. A* B C Aldwulf Beowulf Subject A D Result Cuthbert E Subject B F G U Low Ability Average Ability High Ability Baseline Score Measuring Value-Added – An Example National Trend ‘Average’ Student -ve (- 2 grades) +ve (+ 2 grades) The position of the national trend line is of critical importance

5. A* A B C D E C B A A* Some Subjects are More Equal than Others…. A-Level >1 grade

6. Some Subjects are More Equal than Others…. BTec National Diploma DDD Grade DDM DMM MMM MMP MPP PPP D E C B Average GCSE Score

7. Biology Business and Management Chemistry Design Technology Economics English_A1 Film French_B Geography History Mathematics Music Philosophy Physics Psychology Spanish_B Theatre Arts Visual Arts Some Subjects are More Equal than Others…. International Baccalaureate 7 6 Grade 5 4 C B A A* Average (I)GCSE Score

8. A* A B Art & Design Biology GCSE Grades Chemistry C Economics English French Geography German D History Ict Mathematics Media Studies Music E Physical Education Physics Religious Studies Science (Double) Spanish F Test Score Some Subjects are More Equal than Others…. GCSE

9. Definitions: • Residual – difference between thepoints the student attains and points attained on average by students from the CEM cohort with a similar ability • Standardised Residual – the residual adjusted to remove differences between qualification points scales and for statistical purposes • Average Standardised Residual – this is the ‘Value Added Score’ for any group of results • Subject VA – averageof standardised residuals for all students’ results in the particular subject • School VA – average of standardised residuals for all students’ results in all subjects for a school / college • Confidence Limit – area of statistical uncertainity within which any variation from 0 is deemed ‘acceptable’ and outside of which could be deemed ‘important’

10. Measuring Ability Key Stage Data or Baseline Test ?

11. Key Stage Baselines – KS2; GCSE Independent Baseline Test – CEM Adaptive Test

12. Burning Question : What is my Value-Added Score ? Better Question : Is it Important ?

13. SPC Chart VA Score Performance above expectation Good Practice to Share ? Performance inline with expectation Performance below expectation Problem with Teaching & Learning ? 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Year

14. 4.0 3.0 2.0 2.0 1.0 0.4 0.1 0.1 0.1 Average Standardised Residual 0.0 -0.1 -0.2 -0.2 -0.2 -0.2 -0.4 -0.4 -0.5 -0.5 -0.6 -0.7 -1.0 -0.7 -0.7 -2.0 -2.2 -3.0 -4.0 Art Music Drama Maths French History Biology German Spanish Physics Chemistry Geography Media Studies English Literature Religious Studies English Language Home Economics Information Technology Design and Technology Subject Summary Standardised Residual Graph

15. Grade Points Equivalent Baseline Score The Scatter Plot Look for Patterns… General Underachievement / over achievement ? Do any groups of students stand out ? – high ability vs low ability ? – male vs female ?

16. Other things to look for… Why did these students do so badly ? Why did this student do so well ? How did they do in their other subjects ?

17. Worked Example – Pre 16

19. Danger of Relying on Raw Residuals Without Confidence Limits Which Subjects Cause Most Concern ?

20. Which subjects now cause most concern ?

21. Business Studies

23. Religious Studies

24. Worked Example – Post 16

25. SPC Chart 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Year

26. Subject Summary - Current Year Subject Summary - 3 Year Average

27. A2-English Literature Statistical Process Control (SPC) Chart 2008 2009 2010 Year

28. A2 – English Literature Student Level Residuals (SLR) Report Scatter Plot General Underachievement ?

29. A2 – English Literature Student Level Residuals (SLR) Report Scatter Plot Too many U’s ?

30. Summary of Process • Examine Subject Summary • Determine ‘interesting’ (i.e. statistically significant) subjects • Look at 3 year average as well as single year if available • Look at trends in ‘Interesting Subjects’ • Examine student data –Scatter graphs • Identify students over / under achieving (student list or Paris) • Any known issues ? • Don’t forget to look at over achieving subjects as well as under achieving • Phone / Email CEM when you need help understanding / interpreting the data / statistics !

31. Baseline Choice

32. GCSE or Baseline Test ? • Do students with the same GCSE score from feeder schools with differing value-added have the same ability ? • How can you tell if a student has underachieved at GCSE and thus can you maximise their potential ? • Has a student got very good GCSE scores through the school effort rather than their ability alone ? • Does school GCSE Value-Added limit the ability to add value at KS5 ? • Can you add value at every Key Stage ? • How can you check for this ?

33. ? ?

34. Average GCSE = 6 Average GCSE = 6 Average GCSE = 6 The Effect of Prior Value Added Beyond Expectation +ve Value-Added In line with Expectation 0 Value-Added Below Expectation -ve Value-Added Do these 3 students all have the same ability ?

35. GCSE as Baseline Same School - Spot the Difference ? Test as Baseline

36. GCSE as Baseline Test as Baseline

37. A2 Biology GCSE as Baseline Test as Baseline

38. A2 Biology Student B GCSE as Baseline Student A Student B Test as Baseline Student A

39. A2 Biology Student A Student B How well did these students perform ?

40. National or School Type Specific ?

41. Comparison to all schools Comparison to Independent Schools Only

42. Comparison to FE Colleges Only Comparison to all schools

43. Questions: • How does the unit of comparison used affect the Value-Added data and what implications does this have on your understanding of performance ? • Does this have implications for Self Evaluation ?

44. Thank You Robert Clark – robert.clark@cem.dur.ac.uk