What kinds of assessment promote learning?

1. What kinds of assessment promote learning? Dylan Wiliam Director, Learning and Teaching Research Center ETS NARST Annual International Conference: San Francisco, CA; April 2006

2. Overview of presentation • Why raising achievement is important • Why investing in teachers is the answer • Why assessment for learning should be the focus • How we can put this into practice

3. What do we need students to learn? ...the model that says ‘learn while you are at school the skills that you will apply during your lifetime’ is no longer tenable. These skills will be obsolete by the time you get into the workplace and need them, except for one skill – the skill of being able to learn. It is the skill of being able, not to give the right answer to questions about what you were taught in school, but to make the right response to situations that are outside the scope of what you were taught in school. We need to produce people who know how to act when they are faced with situations for which they were not specifically prepared. (Papert, 1998)

4. The four Rs of “learning power” • Resilience • Absorption, managing distractions, noticing, perseverance • Resourcefulness • Questioning, making links, imagining, reasoning, capitalizing • Reflectiveness • Planning, revising, distilling, meta-learning • Reciprocity • Interdependence, collaboration, empathy and listening, imitation (Claxton 2002)

5. Preparation for future learning (PFL) • Cannot be taught in isolation from other learning • Students still need the basic skills of literacy, numeracy, concepts and facts • Learning power is developed primarily through pedagogy, not curriculum • We have to change the way teachers teach, not what they teach

6. Successful education The test of successful education is not the amount of knowledge that a pupil takes away from school, but his [sic] appetite to know and his capacity to learn. If the school sends out children with the desire for knowledge and some idea how to acquire it, it will have done its work. Too many leave school with the appetite killed and the mind loaded with undigested lumps of information. The good schoolmaster [sic] is known by the number of valuable subjects which he declines to teach. (Sir Richard Livingstone, President of Corpus Christi College, Oxford, 1941)

7. Different approaches to upper secondary schooling • 20th century education • US: mass systems • Europe: elite systems • 21st century education challenge • US: increase quality • Europe: scale up quality

8. Raising achievement matters • For individuals • Increased lifetime salary • Improved health • For society • Lower criminal justice costs • Lower health-care costs • Increased economic growth

9. Where’s the solution? • Structure • Small high schools • K-8 schools • Alignment • Curriculum reform • Textbook replacement • Governance • Charter schools • Vouchers • Technology

10. It’s the classroom • Variability at the classroom level is up to 4 times greater than at school level • It’s not class size • It’s not the between-class grouping strategy • It’s not the within-class grouping strategy • It’s the teacher

11. Teacher quality: • A labor force issue with 2 solutions • Replace existing teachers with better ones? • No evidence that more pay brings in better teachers • No evidence that there are better teachers out there deterred by certification requirements • Improve the effectiveness of existing teachers • The “love the one you’re with” strategy • It can be done • We know how to do it, but at scale? Quickly? Sustainably?

12. Learning power environments • Key concept: • Teachers do not create learning • Learners create learning • Teaching as engineering learning environments • Key features: • Create student engagement (pedagogies of engagement) • Well-regulated (pedagogies of contingency)

13. Why pedagogies of engagement? • Intelligence is partly inherited • So what? • Intelligence is partly environmental • Environment creates intelligence • Intelligence creates environment • Learning environments • High cognitive demand • Inclusive • Obligatory

14. Motivation: cause or effect? high arousal Flow anxiety challenge control worry relaxation apathy boredom low low competence high (Csikszentmihalyi, 1990)

15. Why pedagogies of contingency? • For evaluating institutions • For describing individuals • For supporting learning • Monitoring learning • Whether learning is taking place • Diagnosing (informing) learning • What is not being learnt • Forming learning • What to do about it

16. Effects of formative assessment • Several major reviews of the research • Natriello (1987) • Crooks (1988) • Kluger & DeNisi (1996) • Black & Wiliam (1998) • Nyquist (2003) • All find consistent, substantial effects

17. Cost/effect comparisons

18. Effects of feedback • Kluger & DeNisi (1996) • Review of 3000 research reports • Excluding those: • without adequate controls • with poor design • with fewer than 10 participants • where performance was not measured • without details of effect sizes • left 131 reports, 607 effect sizes, involving 12652 individuals • Average effect size 0.4, but • Effect sizes very variable • 40% of effect sizes were negative

19. Kinds of feedback (Nyquist, 2003) • Weaker feedback only • Knowledge of results (KoR) • Feedback only • KoR + clear goals or knowledge of correct results (KCR) • Weak formative assessment • KCR+ explanation (KCR+e) • Moderate formative assessment • (KCR+e) + specific actions for gap reduction • Strong formative assessment • (KCR+e) + activity

20. Effect of formative assessment (HE)

21. Key processes • Establishing where the learner is in her/his learning • Establishing where she/he is going • Establishing how to get there.

22. Aspects of formative assessment

23. Five key strategies… • Clarifying and understanding learning intentions and criteria for success • Engineering effective classroom discussions that elicit evidence of learning • Providing feedback that moves learners forward • Activating students as instructional resources for each other • Activating students as the owners of their own learning

24. …and one big idea • Use evidence about learning to adapt instruction to meet student needs

25. Keeping Learning on Track (KLT) • A pilot guides a plane or boat toward its destination by planning a route, taking constant readings and making careful adjustments in response to wind, currents, weather, etc. • A KLT teacher does the same: • Plans a carefully chosen (possibly differentiated) route ahead of time (in essence building the track) • Takes readings along the way • Changes course as conditions dictate

26. Regulation of learning • Proactive (upstream) regulation • Planning regulation into the learning environment • Planning for evoking information • Interactive (downstream) regulation • ‘Negotiating the swiftly-flowing river’ • ‘Moments of contingency’ • Tightness of regulation (goals vs. horizons) • Retrospective regulation • Structured reflection (e.g., lesson study)

27. Types of formative assessment • Long-cycle • Focus: between units • Length: four weeks to one year • Medium-cycle • Focus: within units, between lessons • Length: one day to two weeks • Short-cycle • Focus: within lessons • Length: five seconds to one hour

28. Putting it into practice

29. Why research hasn’t changed teaching • Misunderstanding nature of teacher expertise • Leaving teachers to “translate into practice” • Failure to attend to both content and process

30. The nature of expertise in teaching • Aristotle’s main intellectual virtues • Episteme: knowledge of universal truths • Techne: ability to make things • Phronesis: practical wisdom • What works is not the right question • Everything works somewhere • Nothing works everywhere • What’s interesting is “under what conditions” does this work? • Teaching is mainly a matter of phronesis, not episteme

31. Kinds of knowing • Learning theories are not nested • Constructivist theories explain misconceptions better than associationist theories, but • Associationist theories explain acquisition of number facts better than constructivist theories • Each new theory provides better accounts of things the previous theory explained badly, but • Tends to explain poorly the things the previous theory explained well. • The best descriptions of what happens when you come to know something depends on what it is you are learning • For some kinds of learning, telling works • Teaching is not one of them

32. Klein & Klein (1981) Six video extracts of a person delivering cardio-pulmonary resuscitation (CPR) 5 of the video extracts are students 1 of the video extracts is an expert Videos shown to three groups Students, experts, instructors Success rate in identifying expert: Experts: 90% Students: 50% Instructors: 30%

33. Countdown 3 25 1 4 9 Target number: 127

34. Expertise Positioning Low Middle High Random positioning 4 3.5 3 Actual position 4 8 16 Chess (Newell & Simon, 1973)

35. Teacher expertise • Resembles expertise in other domains (Berliner, 1994) • Experts excel mainly in their own domain • Experts often develop automaticity for the repetitive operations that are needed to accomplish their goals • Experts are more sensitive to the task demands and social situation when solving problems. • Experts are more opportunistic and flexible in their teaching than novices • Experts represent problems in qualitatively different ways than novices. • Experts have fast and accurate pattern recognition capabilities. Novices cannot always make sense of what they experience. • Experts perceive meaningful patterns in the domain in which they are experienced. • Experts begin to solve problems more slowly, but bring richer and more personal sources of information to bear on the problem that they are trying to solve.

36. Knowledge creation and transmission After Nonaka & Tageuchi, 1995

37. A model for teacher learning • Content (what we want teachers to change) • Evidence • Ideas (strategies and techniques) • Process (how to go about change) • Choice • Flexibility • Small steps • Accountability • Support

38. Content: strategies and techniques • Distinction between strategies and techniques • Strategies define the territory of AfL (no brainers) • Teachers are responsible for choice of techniques • Allows for customization/ caters for local context • Creates ownership • Shares responsibility • Key requirements of techniques • embodiment of deep cognitive/affective principles • relevance • feasibility • acceptability

39. Design and intervention Our design process cognitive/affective insights synergy/ comprehensiveness set ofcomponents Teachers’ implementation process set of components synergy/ comprehensiveness cognitive/affective insights

40. Practical techniques questioning • Key idea: questioning should • cause thinking • provide data that informs teaching • Improving teacher questioning • generating questions with colleagues • closed v open • low-order v high-order • appropriate wait-time • Getting away from I-R-E • basketball rather than serial table-tennis • ‘No hands up’ (except to ask a question) • class polls to review current attitudes towards an issue • ‘Hot Seat’ questioning • All-student response systems • ABCD cards, Mini white-boards, Exit passes

42. Questioning in science: discussion Ice-cubes are added to a glass of water. What happens to the level of the water as the ice-cubes melt? • The level of the water drops • The level of the water stays the same • The level of the water increases • You need more information to be sure

43. Questioning in science: diagnosis The ball sitting on the table is not moving. It is not moving because: • no forces are pushing or pulling on the ball. • gravity is pulling down, but the table is in the way. • the table pushes up with the same force that gravity pulls down • gravity is holding it onto the table. • there is a force inside the ball keeping it from rolling off the table Wilson & Draney, 2004

44. Questioning in science: diagnosis Which of these is living? • A river • An acorn • Lichen • A Xerox machine • A broken-off branch of a tree

45. Joule Kilogram Newton Pascal Watt Energy Force Mass Pressure Weight Work Questioning in science: diagnosis

46. Dinosaur extinction • Why did dinosaurs become extinct? • A) Humans destroyed their habitat • B) Humans killed them all for food • C) There was a major change in climate

47. Save the ozone layer What can we do to preserve the ozone layer? • Reduce the amount of carbon dioxide produced by cars and factories • Reduce the greenhouse effect • Stop cutting down the rainforests • Limit the numbers of cars that can be used when the level of ozone is high • Properly dispose of air-conditioners and fridges

48. Hinge Questions • A hinge question is based on the important concept in a lesson that is critical for students to understand before you move on in the lesson. • The question should fall about midway during the lesson. • Every student must respond to the question within two minutes. • You must be able to collect and interpret the responses from all students in 30 seconds

49. Practical techniques: feedback • Key idea: feedback should • cause thinking • provide guidance on how to improve • Comment-only grading • Focused grading • Explicit reference to rubrics • Suggestions on how to improve • ‘Strategy cards’ ideas for improvement • Not giving complete solutions • Re-timing assessment • (eg two-thirds-of-the-way-through-a-unit test)

50. Practical techniques: sharing learning intentions • Explaining learning intentions at start of lesson/unit • Learning intentions • Success criteria • Intentions/criteria in students’ language • Posters of key words to talk about learning • eg describe, explain, evaluate • Planning/writing frames • Annotated examples of different standards to ‘flesh out’ assessment rubrics (e.g. lab reports) • Opportunities for students to design their own tests