Dylan Wiliam ETS dwiliam@ets

1. Assessing instructional and assessment practice:What makes a lesson formative?CRESST conference, September 2004UCLA Sunset Village, CA Dylan Wiliam ETS dwiliam@ets.org www.dylanwiliam.net

2. What makes a lesson formative? • Moving from “I know it when I see it” • Expanding ideas of formative assessment • Feedback • Elicitation and feedback • The student’s role • Systems thinking

3. Preliminary assumptions • The primary purpose of educational research is the improvement of education. • The purpose of education is the improvement of student achievement • The improvement of student achievement will be achieved primarily through changes in what happens in classrooms • The role of the teacher is not to teach but to create situations in which students learn

4. Functions of assessment • For evaluating institutions • For describing individuals • For supporting learning • Monitoring learning • Whether learning is taking place • Diagnosing (informing) learning • What is not being learnt • Forming learning • What to do about it

5. Effects of feedback • Kluger & DeNisi (1996) • Review of 3000 research reports • Excluding those: • without adequate controls • with poor design • with fewer than 10 participants • where performance was not measured • without details of effect sizes • left 131 reports, 607 effect sizes, involving 12652 individuals • Average effect size 0.4, but • Effect sizes very variable • 40% of effect sizes were negative

6. Effects of formative assessment • Several major reviews of the research • Natriello (1987) • Crooks (1988) • Black & Wiliam (1998) • Nyquist (2003) • All find consistent, substantial effects

7. Kinds of feedback (Nyquist, 2003) • Weaker feedback only • Knowledge or results (KoR) • Feedback only • KoR + clear goals or knowledge of correct results (KCR) • Weak formative assessment • KCR+ explanation (KCR+e) • Moderate formative assessment • (KCR+e) + specific actions for gap reduction • Strong formative assessment • (KCR+e) + activity

8. Effects of formative assessment (HE)

9. Formative assessment • Classroom assessment is not (necessarily) formative assessment • Formative assessment is not (necessarily) classroom assessment

10. Formative assessment Assessment for learning is any assessment for which the first priority in its design and practice is to serve the purpose of promoting pupils’ learning. It thus differs from assessment designed primarily to serve the purposes of accountability, or of ranking, or of certifying competence. An assessment activity can help learning if it provides information to be used as feedback, by teachers, and by their pupils, in assessing themselves and each other, to modify the teaching and learning activities in which they are engaged. Such assessment becomes ‘formative assessment’ when the evidence is actually used to adapt the teaching work to meet learning needs. Black et al, 2002

11. Bad items/good items 1 Which is larger: 0.33 or 1/3? • 0.33 is larger than 1/3 • 1/3 is larger than 0.33 • They are the same • You need more information to be sure

12. Bad items/good items 2 What is the general rule for the following pattern: 3, 7, 11, 15, …? • 3 + n • n + 4 • 3n + 4 • 4n - 1 • 4n + 3

13. Bad items/good items 3 If f + g = 8, then f + g + h = ? • 9 • 12 • 15 • 16 • 8 + h

14. Bad items/good items 4 Which of the following statements are true ? • A square has two 90° angles • A square is a trapezoid • A kites are squares • A trapezoid is a rectangle • A rectangle is a trapezoid

15. Bad items/good items 5 Ice-cubes are added to a glass of water. What happens to the level of the water as the ice-cubes dissolve? A) The level of the water drops B) The level of the water stays the same C) The level of the water increases D) You need more information to be sure

16. Bad items/good items 6 What is the pH of gasoline? A) Less than zero B) Zero C) Greater than zero but less than 7 D) 7 E) Greater than 7 F) None of the above

17. Bad items/good items 7 In what year did World War II begin? A) 1919 B) 1937 C) 1938 D) 1939 E) 1941 F) 1942

18. Good formative items are different… • Can have more (or less!) than one correct answer • Items need to be generative • of learning • of insights into learning • of insights into how to promote learning • Increasing ability may reduce your chance of getting the item correct • Distractors must be explicitly connected to incorrect or incomplete conceptions (facets) • Item responses must provide clues to effective action • Some items support both summative and formative functions well; most do not

19. How do we make good items? • Good items are not universal • In some domains, facets of thinking are reasonably universal (e.g. physics) • In most domains, facets of thinking are dependent on instruction • So there is no ‘universal’ set of facets • And we have to rely on teachers to do it for themselves • Good items have to be used • Requires a model of effective, scalable teacher professional development

20. Why research hasn’t changed teaching • The nature of expertise in teaching • Aristotle’s main intellectual virtues • Episteme: knowledge of universal truths • Techne: ability to make things • Phronesis: practical wisdom • What works is not the right question • Everything works somewhere • Nothing works everywhere • What’s interesting is “under what conditions” does this work? • Teaching is mainly a matter of phronesis, not episteme

21. Countdown 3 25 1 4 9 Target number: 127

22. Aspects of formative assessment

23. A formative lesson • Grade 9 Algebra lesson • Weather balloon set off, rising at 8fs-1 • 2s later, a flare is launched, initial velocity 80fs-1 • Students complete a table and a graph to answer three questions: • Maximum height of flare • How long to max height • Average velocity of flare

24. Regulation of learning • Teaching as engineering learning environments • Key feature: well-regulated • Teaching vs. learning • Regulation of activity vs. regulation of learning • Long feedback cycles vs. variable feedback cycles • Quality control vs. quality assurance in learning

25. Regulation of learning • Proactive (upstream) regulation • Planning regulation into the learning environment • Planning for evoking information • Interactive (downstream) regulation • ‘Negotiating the swiftly-flowing river’ • ‘Moments of contingency’ • Tightness of regulation (goals vs horizons) • Retrospective regulation • Structured reflection (e.g. lesson study)