Formative evaluation of teaching performance

1. Formative evaluation ofteaching performance Dylan Wiliam (@dylanwiliam) INEE seminar, Mexico City, 5 December 2013 www.dylanwiliam.org

2. Outline • Education matters, for individuals and society • Teaching quality is the crucial variable • Teaching quality is not the same as teacher quality • Predicting who will be good teachers is almost impossible • Evaluating teacher quality is inherently difficult • Professional development is the key to teacher quality • Feedback is more complicated than generally assumed • Formative evaluation of teaching performance • Strategies for formative evaluation • Validity of formative evaluation of teaching • Implementing formative evaluation of teaching

3. Education matters:for individuals and society

4. What is the purpose of education? • Four main philosophies of education • Personal empowerment • Cultural transmission • Preparation for citizenship • Preparation for work • All are important • Any education system is a (sometimes uneasy) compromise between these four forces

5. Raising achievement matters • For individuals: • Increased lifetime earnings • Improved health • Longer life • For society: • Lower criminal justice costs • Lower healthcare costs • Increased economic growth: • Net present value to Mexico of a 25-point increase on PISA: US$5 trillion • Net present value to Mexico of getting all students to 400 on PISA: US$26 trillion (Hanushek & Woessman, 2010)

6. Teaching quality is the crucial variable

7. We need to focus on classrooms, not schools • In most countries, variability at the classroom level is much greater than that at school level. • As long as you go to school, it doesn’t matter very much which school you go to. • But it matters very much which classrooms you are in.

8. Within-school variation: 64% Between school variation not explained by social background: 18% Within schools Between school variation explained by social back-ground of students: 5% Between school variation explained by social back-ground of schools: 16% Between schools McGaw(2008)

9. Teaching quality is not the same as teacher quality

10. Teaching quality/teacher quality • Teaching quality depends on a number of factors • The time teachers have to plan teaching • The size of classes • The resources available • The skills of the teacher • All of these are important, but the quality of the teacher seems to be especially important

11. Teacher quality • Take a group of 50 teachers all teaching the same subject: • In the classroom of the best teacher, students learn in six months what students taught by the average teacher will take a year to learn. • In the classroom of the least effective teacher, students will take two years to learn the same amount (Hanushek& Rivkin, 2006) • And in the classrooms of the best teachers, students from disadvantaged backgrounds learn as much as others (Hamre & Pianta, 2005)

12. The “dark matter” of teacher quality • Teachers make a difference • But what makes the difference in teachers? • In particular, can we predict student progress from: • Teacher qualifications? • Value-added? • Teacher observation?

13. Predicting who will be good teachers is almost impossible

14. Teacher qualifications and student progress Harris and Sass (2007)

15. Evaluating teacher quality is inherently difficult

16. Framework for teaching (Danielson 1996) • Four domains of professional practice • Planning and preparation • Classroom environment • Instruction • Professional responsibilities • Links with student achievement (Sartain, et al. 2011) • Domains 1 and 4: no impact on student achievement • Domains 2 and 3: some impact on student achievement

17. A framework for teaching (Danielson, 1996) • Domain 2: The classroom environment • 2a: Creating an environment of respect and rapport • 2b: Establishing a culture for learning • 2c: Managing classroom procedures • 2d: Managing student behavior • 2e: Organizing physical space • Domain 3: Instruction • 3a: Communicating with students • 3b: Using questioning and discussion techniques • 3c: Engaging students in learning • 3d: Using assessment in instruction • 3e: Demonstrating flexibility and responsiveness

18. Observations and teacher quality So, the highest-rated teachers are 30% more productive than the lowest rated But the best teachers are 400% more productive than the least effective Sartain, Stoelinga, Brown,Luppescu, Matsko, Miller, Durwood, Jiang, and Glazer (2011)

19. We don’t know much about teaching… • We cannot predict how good a teacher will be • We cannot tell good teaching when we see it • Expert ratings of teaching • Student ratings of teaching • We cannot evaluate teaching with test scores

20. Traditional approaches to improving teaching • Two main approaches • Removing ineffective teachers • Rewarding good teachers • Problems • Consume large amounts of management time • Technically difficult to do well • Create competition between teachers • Differentially effective according to task complexity

21. The story so far • Improving student achievement is a priority for every country • Improving student achievement requires improving teacher quality • Improving teacher quality requires investment in serving teachers

22. Professional development is the key to teacher quality

23. General conclusions about expertise • Elite performance is the result of at least a decade of maximal efforts to improve performance through an optimal distribution of deliberate practice • What distinguishes experts from others is the commitment to deliberate practice • Deliberate practice is • an effortful activity that can be sustained only for a limited time each day • neither motivating nor enjoyable—it is instrumental in achieving further improvement in performance

24. Expertise • According to Berliner (1994), experts: • Excel mainly in their own domain • Often develop automaticity for the repetitive operations that are needed to accomplish their goals • Are more sensitive to the task demands and social situation when solving problems • Are more opportunistic and flexible in their teaching than novices • Represent problems in qualitatively different ways than novices • Have faster and more accurate pattern recognition capabilities • Perceive meaningful patterns in the domain in which they are experienced • Begin to solve problems slower but bring richer and more personal sources of information to bear

25. Effects of experience in teaching Mathematics Reading Rivkin, Hanushek and Kain (2005)

26. Implications for education systems • Pursuing a strategy of getting the “best and brightest” into teaching is unlikely to succeed • Currently all teachers slow, and most actually stop, improving after two or three years in the classroom • Expertise research therefore suggests that they are only beginning to scratch the surface of what they are capable of • What we need is to persuade those with a real passion for working with young people to become teachers, and to continue to improve as long as they stay in the job. • There is no limit to what we can achieve if we support our teachers in the right way

27. Feedback is generally more complex than generally assumed

28. Important caveats about research findings • Educational research can only tell us what was, not what might be. • Moreover, in education, “What works?” is not the right question, because • everything works somewhere, and • nothing works everywhere, which is why • in education, the right question is, “Under what conditions does this work?”

29. Effects of formative assessment Standardized effect size: differences in means, measured in population standard deviations

30. Understanding meta-analysis • A technique for aggregating results from different studies by converting empirical results to a common measure (usually effect size) • Standardized effect size is defined as: • Problems with meta-analysis • The “file drawer” problem • Variation in population variability • Selection of studies • Sensitivity of outcome measures

31. Effects of feedback • Kluger & DeNisi (1996) review of 3000 research reports • Excluding those: • without adequate controls • with poor design • with fewer than 10 participants • where performance was not measured • without details of effect sizes • left 131 reports, 607 effect sizes, involving 12652 individuals • On average, feedback increases achievement • Effect sizes highly variable • 38% (50 out of 131) of effect sizes were negative

32. Getting feedback right is hard

33. Kluger and DeNisi’s conclusions… These considerations of utility and alternative interventions suggest that even an FI [feedback intervention] with demonstrated positive effects on performance should not be administered whenever possible. Rather, additional development of FIT [feedback intervention theory] is needed to establish the circumstance under which positive FI effects on performance are also lasting and efficient and when these effects are transient and have questionable utility. This research must focus on the processes induced by FIs and not on the general question of whether FIs improve performance—look at how little progress 90 years of attempts to answer the latter question have yielded. (p. 278)

34. Formative evaluation of teaching performance

35. The evidence base for formative assessment • Fuchs & Fuchs (1986) • Natriello (1987) • Crooks (1988) • Bangert-Drowns, et al. (1991) • Dempster (1991, 1992) • Elshout-Mohr (1994) • Kluger & DeNisi (1996) • Black & Wiliam (1998) • Nyquist (2003) • Brookhart (2004) • Allal & Lopez (2005) • Köller (2005) • Brookhart (2007) • Wiliam (2007) • Hattie & Timperley (2007) • Shute (2008)

36. Assessment for learning/formative assessment “Assessment for learning is any assessment for which the first priority in its design and practice is to serve the purpose of promoting students’ learning. It thus differs from assessment designed primarily to serve the purposes of accountability, or of ranking, or of certifying competence. An assessment activity can help learning if it provides information that teachers and their students can use as feedback in assessing themselves and one another and in modifying the teaching and learning activities in which they are engaged. Such assessment becomes “formative assessment” when the evidence is actually used to adapt the teaching work to meet learning needs.” (Black, Harrison, Lee, Marshall & Wiliam, 2004 p. 10)

37. Theoretical questions • Need for clear definitions • So that research outcomes are commensurable • Theorization and definition • Possible variables • Category (instruments, outcomes, functions) • Beneficiaries (teachers, learners) • Timescale (months, weeks, days, hours, minutes) • Consequences (outcomes, instruction, decisions) • Theory of action (what gets formed?)

38. Formative assessment: a new definition “An evaluation of teacher performance functions formatively to the extent that evidence of teacher performance that is elicited by the assessment is interpreted by leaders, teachers, or their peers to make decisions about the professional development of the teacher that are likely to be better, or better founded, than those that would have been taken in the absence of that evidence.”

39. Formative evaluation involves the creation of, and capitalization upon, moments of contingency in the regulation of teachers’ learning processes • Kinds of regulation (Perrenoud, 1998) • Proactive • Interactive • Retroactive • Agents • Leaders (external regulation) • Peers (co-regulation) • Teachers (self-regulation)

40. Strategies of formative evaluation

41. Unpacking formative assessment of teaching • Engineering effective situations, tasks and activities that elicit evidence of development • Clarifying, sharing and understanding learning intentions • Providing feed-back that moves learners forward • Activating teachers as learning • resources for one another • Activating teachers as ownersof their own learning

42. Validity of formative evaluation

43. Validity: an evolving concept • Evolution of the idea • A property of a test • A property of students’ results on a test • A property of the inferences drawn on the basis of test results • For any test: • some inferences are warranted • some are not • “One validates not a test but an interpretation of data arising from a specified procedure” (Cronbach, 1971; emphasis in original) • No such thing as a valid assessment!

44. Validating formative evaluation • An assessment is a procedure for making inferences: • about what the learner knows (summative) • about what to do next (formative) • Summative inferences are validated by consistency of meanings across different readers • Formative inferences are validated by the consequences for learners

45. Implementing formative evaluation of teaching performance

46. A model for teacher learning • Content, then process • Content (what we want teachers to change): • Evidence • Ideas (strategies and techniques) • Process (how to go about change): • Choice • Flexibility • Small steps • Accountability • Support

47. Choice

48. A strengths-based approach to change • Talent development requires attending to both strengths and weaknesses • The question is how to distribute attention between the two: • For novices, attention to weaknesses is likely to have the greatest payoff • For more experienced teachers, attention to strengths is likely to be more advantageous

49. Flexibility

50. Tight, but loose • Two opposing factors in any school reform • Need for flexibility to adapt to local circumstances • Need to maintain fidelity to the theory of action of the reform, to minimise “lethal mutations” • The “tight but loose” formulation: • … combines an obsessive adherence to central design principles (the “tight” part) with accommodations to the needs, resources, constraints, and affordances that occur in any school or district(the “loose” part), but only where these do not conflict with the theory of action of the intervention.