Trainer: Geoff Hancock

1. Welcome to:How To Interpret the New PANDA and Write an Effective Standards and Achievement Section of the SEF. Trainer: Geoff Hancock

2. Session 1 The New Style PandA Explained

3. What the new Panda contains • The final version of the 2004 new-style PANDA report contains the following four sections: • Summary • Contextual information about the school, including attainment on entry • Contextual value added (CVA) information • Attainment information, including subject detail at KS4 and 16+

4. The basis of the new PANDA The new PANDA uses a wider range of data in a greater variety of ways than previous ones Fairer comparisons are sought Presentation is in a wider range of formats PANDAs comprise a series of subsections - each starting with page 1! (Please note and refer to the handwritten numbering on the PANDA used for this course)

5. Aims of the day: • to inform you about the types of data analyses in the PANDA report • to help you find the data you need quickly • to assist you in interpreting a school’s performance data • to demonstrate how the data inform inspection judgements • to enable you to pick out inclusion issues and pupils to follow up

6. Which inspection judgements do the data inform • The data in the PANDA report inform the judgements on: • standards • progress. • Taken together, these underpin the judgement ‘How well do learners achieve?’

7. What are the key features of the data? • The main differences from the previous PANDA report are: • more robust analyses based on statistical tests • removal of benchmark groups • use of contexts to account for pupils’ characteristics and consequently isolate school effectiveness • greater use of pupil-level data.

8. What are the key features of the data? • a quick overview through graphs, backed up by diagnostic information for groups and subjects that shows where strengths and weaknesses in standards and progress lie.

9. What’s in and what’s out. • overall information on standards in comparison with national averages. • provide contextual value added (CVA) scores based on individual pupils’ results. CVA calculations take into account factors that affect pupils’ achievements such as gender, age and ethnicity, as well as prior attainment, so offer a robust way of isolating and evaluating the progress brought about by the school.

12. The new-style PANDA report does not provide benchmark grades;

13. Pandas • uses significance tests to pick out only the performance we are 95% confident is different from the national average. • These tests identify different proportions of schools as significantly different from average for each set of attainment or CVA analyses. • For CVA analyses of overall average points score (APS) and for individual subject APS, roughly the: • top quarter of schools shows progress that is significantly above national average (sig+) • middle half of schools shows progress that is not significantly different from national average • lowest quarter of schools shows progress that is significantly below national average (sig-).

14. Contextual Value Added Indicators Free schools meals is a contextual factor. It can be a useful indicator but is may well give unreliable impressions if used to make precise distinctions on its own. Prior attainment is another contextual factor - though not always recognised as such. New PANDAs use a range of contextual factors when calculating the value-added or progress made by pupils. The key ones are: Gender Age Whether pupil joined the school with majority of pupils Ethnicity Several prior attainment indicators EAL In care/not in care Several SEN indicators IDACI score of pupils’ postcodes (Income of Deprivation Affecting Children Index) Shows the proportion of under-16s from low income households.

15. Contextual Value Added Indicators • For CVA analyses of overall average points score (APS) and for individual subject APS, roughly the: • top quarter of schools shows progress that is significantly above national average (sig+) • middle half of schools shows progress that is not significantly different from national average • lowest quarter of schools shows progress that is significantly below national average (sig-). • Consequently, the fact that a school has sig+ CVA distinguishes it from those with sig- or not significant CVA, but does not distinguish it from the other 25% of schools that also have sig+ CVA.

16. New graphs, new statistics, new analyses There is a great wealth of data presented in numerous forms. Some are familiar others are new. The best way to get an initial picture is to consider these examples: A Line graphs e.g. p3 B Bar graphs for results e.g. p4 C Comparative tables e.g. p5 and p9 (Note the use of shading and “significant change” arrows at foot of page) D Conversion Tables e.g. p6 and p10 (Note how many pupils go from Level 4b to Level 5+ in Maths; then look at English)

17. E Contextual Tables with distribution graphs e.g. p12 (How does pupil mobility/stability look?) F “Before and after” bar graphs for prior attainment e.g.p16 (Please note small print at top right of each box) G “Snake curve” graph for CVA e.g.p 19 and 25 H Quadrant graphs e.g.p 19 and 25 J Multiple CVA charts e.g.p 21 and 27 K Cumulative Distribution graphs e.g.p 35 L Summary Tables with RPI ratings e.g.p 40 (Please note the 3 lines of small print at top)

18. Attainment • Analyses of attainment shown by APS, much higher proportions of schools have results significantly different from average. • For overall APS, this is between 65% and 85% of schools. • Consequently, the fact that a school has sig- overall attainment does not distinguish it from the other roughly 30% to 40% of schools with sig- attainment. • Tables 18 and 19 in the Reference booklet show the proportions of schools significantly different from average for each key stage, for both APS and level threshold analyses.

19. Graphs of attainment results • The graphs of attainment results do not show the confidence interval. • For attainment, the size of the confidence interval depends upon the variation between individual pupils’ results as well as the size of the cohort. • A significance test is carried out to find whether the confidence interval would fall completely above or below the national average.

20. Session 2How to Interpret standards at Key Stages Standards means the ‘attainment’ of pupils – it compares the attainment reached in a particular school to National percentages

21. Judging Standards • When evaluating standards and progress you need to consider the: • results in relation to the national average • educational importance of the results • statistical significance • variation between key stages, subjects, groups and individual pupils.

22. Attainment on Entry Judgements on targets, progress and provision need to be informed by prior attainment data. The prior attainment of pupils who have completed the key stage in the past and are likely to have left the school will have had an impact on the results for the past five years. It does not necessarily reflect the attainment on entry of pupils currently in the school. It is important to describe the attainment on entry of all current pupils and any difference with previous cohorts for which standards and achievement are already known.

23. What does this page tell you?

24. Prior attainment: From the prior attainment of pupils in KS3 the trend for Years 7 to 9 in 2004 is shown by the difference in average point scores (APS) which is for Y9 (+0.6), Y8 (+1.0) and Y7 (+0.9). On average this equates to an APS difference of approximately (+0.83), which is classed as average (since it falls within 1.25 points of the national average). Overall, attainment on entry to Key Stage 3 for Years 7 to 9 in 2004 is average.

25. Key Stage 4\Prior attainment What does this page tell you?

26. For Years 11 and 10 in 2004 we see the KS2 and 3 APS on entry. This also allows us to see if attainment on entry to KS4 is higher or lower than would be expected, ie an indication of whether sufficient value has been added in KS3. It also allows you to see the attainment on entry of the last Y10 (now current Y11) attainment on entry at both Key Stages, which is helpful in re-evaluating targets in comparison with previous performance. Specifically: From Key Stage 2 attainment data, a broadly average attainment on entry, though with more students than average at Level 4 (27-30 average points). From Key Stage 3 results, an above average attainment on entry to KS4 with significantly higher than national Level 6, though slightly less Level 7s than national. The KS3 prior attainment for the 2004 KS4 results give the following differences from national, Y11 (KS3 APS diff = +2.0) above average, for a cohort who entered KS3 average (ie +1.1 points above national – within tolerance). For Y10 in 2004 (KS3 APS diff +3.2) well above average for a cohort who entered KS3 broadly average (+0.4 points, equates to within tolerance). This suggests for the Years 10 and 11 in 2004 there was an average intake to KS3. However, for Year 10 in 2004 they entered KS4 well above average, higher than the Year 11 in 2004 who entered KS4 above average.

27. Standards, trends and targets • To make a preliminary judgement on the standards reached by pupils • To interpret the graphs of standards at thresholds and identify potential inclusion issues • To identify trends in standards over the last five years • To pick out the main strengths and weaknesses in standards • To evaluate whether targets are adequately challenging.

28. Inspectors must • Inspectors must form a judgement for the school overall on ‘the standards reached by learners’. • ‘Inspectors should evaluate: • the standards learners reach as indicated by their test and examination results, and other available evidence, taking account of: any significant variations between groups of learners, subjects, courses and key stages; trends over time; and comparisons with all schools • whether learners achieve their targets and whether the targets are adequately challenging.’

29. Steps to take when reading tables and graphs deciding judgements on standards and progress Firstly, do your data indicators appear in each column for grade 4 to be present? Secondly, do your data indicators appear in each column for grade 1 to be present? Thirdly, do your data indicators appear in each column for grade 2 to be present?

30. The simplified approach • When evaluating standards and progress you need to consider the: • Results in relation to the national average • Educational importance of the results • Statistical significance • Variation between key stages, subjects, groups and individual pupils

31. What do they mean? • The pale box means that value is significantly higher than average and is also represented in other tables as ‘Sig+’ • The darker box means that value is significantly lower than average and is also represented in other tables as ‘Sig-’

32. Evaluating standards at Key Stage 3 (3 year summary)

33. Fig.1.a

34. Fig.1.b

35. Overall in 2003 English was: Above the national average at Level 5+ compared to all schools. Overall in 2004 English was: In line with the national average in terms of average points score. (from fig 1a) In line with the national average at Level 5+ compared to all schools (from fig 1b) In line with the national average at Level 6+ compared to all schools (from fig 1b) At level 7+ English was below average, though not significantly (from fig 1b). From 2000 to 2003 the attainment of English at KS3 has been significantly above the national, until 2004 when it fell in line with the national

36. What does this page tell you? Here we see five highlighted groups. Those groups which are not highlighted are in line with the national average. Of the five highlighted groups, four are significantly higher than average, ie all pupils CVA in 2003, non-FSM and non-SEN, along with prior attainment only (AAT) in 2004 for non-SEN students. The only group here to be significantly lower was the FSM group in terms of prior attainment only. The question would then arise as to why these groups were better or worse than average.

37. Session 3 Achievement: How to identify achievement (progress) for the subjects, the school and individual groups of learners Progress is the new name for achievement. This is a grade for the rate at which pupils make progress in their understanding and is relative to what they could do when they started

38. Grade Descriptors • Outstanding (1) - Progress is at least good in all or nearly all respects and is exemplary in significant elements, as reflected in contextual value added measures. • Good (2) - Learners meet challenging targets and, in relation to their capability and starting points, they achieve high standards. Most groups of learners, including those with learning difficulties and disabilities, make at least good progress and some may make very good progress, as reflected in contextual value added measures. Learners are gaining knowledge, skills and understanding at a good rate across all key stages. Most subjects and courses perform well, and some better than this, with nothing that is unsatisfactory. • Satisfactory (3) - Progress is inadequate in no major respect, and may be good in some respects, as reflected in contextual value added measures. • Inadequate (4) - A significant number of learners do not meet targets that are adequately challenging.

39. Inspectors must evaluate: • whether learners achieve their targets and whether the targets are adequately challenging • how well learners progress relative to their starting points and capabilities, with any significant variations between groups of learners (making clear whether there is any underachievement generally or among particular groups who could be doing better).’

40. The PandA: CVA and relative attainment Schools in this quadrant are performing above the national average in attainment and in contextual value added Schools in this quadrant are performing below the national average in attainment and above in contextual value added Schools in this quadrant are performing below the national average in attainment and in contextual value added Schools in this quadrant are performing above the national average in attainment but below in contextual value added

41. KS2-4 overall CVA Cohort = 72, CVA score = 975 What does this figure tell us?

42. That the CVA is inadequate as the plot is on the steeper part of the plotted line and couldn’t be considered satisfactory, as the confidence interval does not cross the median line

43. KS2-4 overall CVA Cohort = 87, CVA score = 1017 What does this figure tell us?

44. That the CVA is good: its lower confidence interval is above the median line and so would not be satisfactory. It is not likely to be deemed outstanding, as the plot is not on the steeper part of the curve.

45. Value added measures KS2-3 What does this page specifically mean?

46. Achievement: KS2 to 3 CVA The CVA shows English value added to be significantly lower than average achievement, ie CVA in English at KS3 is inadequate in 2004 The contextual value added indicator shows value added to be broadly average, though close to good: ie satisfactory CVA in maths to KS3 The contextual value added indicator shows value added to be significantly higher than average achievement, ie in 2004 the CVA for science is good This page further confirms what figure 36 told us, ie that the School CVA in 2004 is satisfactory, at the 46th percentile of all schools In 2004: (considering KS2 to KS3 CVA) English: Significantly lower than average achievement, at the 73rd percentile of all schools nationally, ie in 2004 the CVA for English is inadequate Maths: Average achievement, at the 41st percentile of all schools nationally, ie in 2004 the CVA for maths issatisfactory Science: Significantly higher than average achievement, improved to be in the top 18% of all schools nationally, ie in 2004 the CVA for science is good All pupils, based only on prior attainment are broadly average, placing the school at the 38th percentile of schools nationally. ie satisfactory CVASchool CVA: Satisfactory, at the 46th percentile of all schools

47. KS2-3 conversions Note also the tables of conversion rates in English, mathematics and science from Key Stages 2 to 3 and from Key Stages 3 to 4. What does Fig 38 specifically tell you?

48. Conversion to Levels 5+/6+ Maths converts the most of the lower ability students to L5+, whereas science converts the least. Science also has the weakest conversion of middle ability students to a Level 5+.However, English and maths have the better conversion rate for middle ability students, with maths being best. Of the three core subjects, 26% of the students who entered English at the beginning of KS3 at Level 5+ remained at or fell below that level, compared to 0% in maths and 11% in science. In maths 100% of the higher ability students reached Level 6+ compared to 80% in science, ie maths was better at converting higher ability students to Level 6+, with English converting the least.

49. KS2 to 4 CVA What does this page tell you?

50. Progress from KS2-KS4 In 2003 (considering KS2 to KS4 CVA): English: Significantly lower than the national average, being in the bottom 23% of all schools nationally Maths: In line with the national average, at the 45th percentile of all schools nationally Whole school: Significantly lower than the national average, in the bottom 17% of all schools nationally KS2 to 4 CVA – whole school KS4 subjects based on contextual value added, KS3 to KS4 in 2004: English: broadly average, ie in 2004 the CVA for English is satisfactory Maths: significantly above average, with a sig+ indicator, ie good CVA close to outstanding