James G Ladwig Newcastle Institute for Research in Education

1. The impact of teacher practice on student outcomes in Te Kotahitanga James G Ladwig Newcastle Institute for Research in Education

2. Main points of Presentation • First task: exploring the observational data to: • See if the data confirms the theoretical ideas about how pedagogy ‘works’ initially asking ‘what does the data say about how the various elements of pedagogy fit together?’ • See if we can use this data as a measure of pedagogy to examine the effects of high quality pedagogy on student outcomes • Second task: exploring the link between pedagogy and outcomes • Here we’re reliant on existing testing data (warts and all) • Here we’re also limited in the options we have for making that linking match the reality of what students experience (in different classes, subjects, with different teachers, etc…) • Third task: offer some interpretive suggestions of the implications for Te Kotahitanga schools and teachers

3. Initial elements • Learning relations • Engagement: based on counts (thus a direct measure of a representative selection of students in a given lesson) • Cognitive level: global indicator, single item on observational instrument • Discursive interactions: based on counts and categories (of same selected representative students) • Caring Relations (each global ratings, aka ‘the second page’): • Culturally Appropriate • Culturally Responsive • Caring • Well Managed • High Expectations for Performance • High Expectations for Behaviour

4. Modelling the relations • Since Engagement and ‘Discursive’ are counts, and since the ‘cognitive’ indicators is a single measure, ‘Learning Relations’ indicators are best taken as direct measures (you see them directly). • However, the Caring relation indicators could be taken as individual measures of something bigger but less directly observed, aka they could be indicators of a ‘latent concept’ • All six are significantly correlated with each other, so we wanted to see if these modelled as a ‘latent construct’ was worth the effort. • If so, you can combine them into one measure of ‘Caring’

5. So, is ‘Caring Relations’ a good construct? • Yes – very good ‘fit’ measures with slight modification • The ‘well managed’ item isn’t needed for this exercise, and • Good reliability (h =.84 for the technically inclined)

6. So what can this show us? • In this case, consider how Caring relations relate to the amount of ‘Discursive interaction’: • We don’t see high levels of discursive interactions without high level of Caring relations, but • Having high levels of Caring relations does not guarantee high degrees of discursive interaction

7. The reality is, of course, more complex • Modelling all of the indicators can be done, but gets pretty complicated.. • Importantly, though, note direct effects from specific indicators of Caring relations and the feedback loops

8. Making the link to outcomes….

9. Maths gains (using available data from 5 schools) 90.5% of variance within schools 9.5% between schools Where is the difference in student gains? (in Te Kotahitanga schools)

10. Taking ‘Discursive’ interaction as an example (2009 observational data): 16.2% of the variance is between schools 83.8% of the variance is between teachers And what of the variance of pedagogy?

11. What do we make of these? • First, keeping in mind that the biggest differences in both student learning gains AND in the quality of pedagogy are not between different schools. • In fact most of these school differences are not statistically significant – meaning we can’t tell they are really very different at all… BUT.. • The differences in gains scores and the quality of pedagogy within schools ARE quite important and, in my view, the place to start any school reform initiative (i.e. the main focus of Te Kotahitanga is very much on target). • This is all the more true if the model of pedagogy employed in Te Kotahitanga is linked to improved gains.

12. Some caveats before the punchline • First, keep in mind that the observation data was not gathered for this analysis – but was for PD purposes. • Also keeping in mind that not all school have comparable data (nor have they all test in comparable time schedules) • This means it has not be sampled in a manner that is readily linked to outcomes (observations are not directly tracked to specific classes of particular groups of students). • Also keep in mind that students change classes and subjects and teachers over the course of a school year…

13. So how did I do it? • I have had to come at this from the ground up – by taking data from whatever classes the students were in that were also observed. • This means that there is a fair bit of looseness in that matching. • Luckily, for most students a lot of classes were observed (around 9 over the year, on average) • So we can consider the average of all the classes as an estimate of each students pedagogical experience. • Given this… what’s the link can we find at this point?

14. Who experienced what quality of pedagogy?

15. And the punchline: Maths Gains by Pedagogy

16. So far so good • I should note this is a preliminary analysis and there is much more to do, including: • Examining the link with gains in Reading • Examining ways to better match the pedagogy data with students and classes • For now, though, it seems very clear that the focus on improving the quality of students’ pedagogical experiences is rightfully at the core of our work. • For teachers, this suggests the large effort to improve from good to very good is well worth the effort. • For schools, this suggests the effort to provide students with more consistently good quality pedagogy (getting less variance within schools by moving more classes ‘up’) is well worth the effort.