Raising Educational Achievement: Why it Matters, What Has Been Tried, and Why It Hasn’t Worked

1. Raising Educational Achievement:Why it Matters, What Has Been Tried, and Why It Hasn’t Worked Dylan Wiliam Ohio Innovative Learning Environments Conference August 2011 www.dylanwiliam.net

2. Raising achievement matters • For individuals • Increased lifetime salary • Improved health • Longer life • For society • Lower criminal justice costs • Lower health-care costs • Increased economic growth • Net present value to the US of a 25 point increase on PISA: $40 trn • Net present value to the US of getting all students to 400 on PISA: $70 trn

3. Impact of education on health Proportion of adults reporting good health, by level of education (OECD, 2010)

4. Which of the following categories of skill is disappearing from the work-place most rapidly? • Routine manual • Non-routine manual • Routine cognitive • Complex communication • Expert thinking/problem-solving

5. …but what is learned matters too… Autor, Levy & Murnane, 2003

6. …now more than ever… Source: Economic Policy Institute

7. The world’s leading manufacturers

8. Which jobs are off-shoreable?

9. How flat is the world? Percentage crossing national boundaries • Physical mail: • Telephone minutes: • Internet traffic: • First generation immigrants: • University students: • People, ever in their lives: • Goods and services:

10. Percentage crossing national boundaries • Physical mail: • Telephone minutes: • Internet traffic: • First generation immigrants: • University students: • People, ever in their lives: • Goods and services: Responses • 1% • 5% • 10% • 20% • 50%

11. Mostly round; some flat bits (Ghemawat, 2011) Percentage crossing national boundaries • Physical mail: 1 • Telephone minutes: 2 • Internet traffic: 17 • First generation immigrants: 3 • University students: 2 • People, ever in their lives: 10 • Goods and services: 10

12. There is only one 21st century skill So the model that says learn while you’re at school, while you’re young, the skills that you will apply during your lifetime is no longer tenable. The skills that you can learn when you’re at school will not be applicable. They will be obsolete by the time you get into the workplace and need them, except for one skill. The one really competitive skill is the skill of being able to learn. It is the skill of being able not to give the right answer to questions about what you were taught in school, but to make the right response to situations that are outside the scope of what you were taught in school. We need to produce people who know how to act when they’re faced with situations for which they were not specifically prepared. (Papert, 1998)

13. Successful education? The test of successful education is not the amount of knowledge that a pupil takes away from school, but his appetite to know and his capacity to learn. If the school sends out children with the desire for knowledge and some idea how to acquire it, it will have done its work. Too many leave school with the appetite killed and the mind loaded with undigested lumps of information. The good schoolmaster is known by the number of valuable subjects which he declines to teach. The Future of Education (Livingstone, 1941 p. 28)

14. Reflection • What is the most interesting/surprising/challenging thing you have heard so far? • See if you can see get consensus in your group…

15. Written examinations “They have perverted the best efforts of teachers, and narrowed and grooved their instruction; they have occasioned and made well nigh imperative the use of mechanical and rote methods of teaching; they have occasioned cramming and the most vicious habits of study; they have caused much of the overpressure charged upon schools, some of which is real; they have tempted both teachers and pupils to dishonesty; and last but not least, they have permitted a mechanical method of school supervision.” (White, 1888 p. 517-518)

16. The Lake Wobegon effect revisited Lake Wobegon is a place where “all the women are strong, all the men are good-looking, and all the children are above average.” (Garrison Keillor, 1985)

17. The Macnamara Fallacy (Handy, 1994 p. 219) • The first step is to measure whatever can be easily measured. • This is OK as far as it goes. • The second step is to disregard that which can’t easily be measured or to give it an arbitrary quantitative value. • This is artificial and misleading. • The third step is to presume that what can’t be measured easily really isn’t important. • This is blindness. • The fourth step is to say that what can’t be easily measured really doesn’t exist. • This is suicide.

18. Goodhart’s law (Campbell’s law) • All performance indicators lose their meaning when adopted as policy targets: • Inflation and money supply • Railtrack’s performance targets • National Health Service waiting lists • National or provincial school achievement targets • The clearer you are about what you want, the more likely you are to get it, but the less likely it is to mean anything

19. Where’s the solution? • Structure • Smaller/larger high schools • K-8 schools/ “All-through” schools • Alignment • Curriculum reform • Textbook replacement • Governance • Charter Schools • Vouchers • Technology • Computers • Interactive white-boards • Staffing • Teachers’ aides

20. School effectiveness • Three generations of school effectiveness research • Raw results approaches • Different schools get different results • Conclusion: Schools make a difference • Demographic-based approaches • Demographic factors account for most of the variation • Conclusion: Schools don’t make a difference • Value-added approaches • School-level differences in value-added are relatively small • Classroom-level differences in value-added are large • Conclusion: An effective school is a school full of effective classrooms

21. USA Within schools Between schools OECD PISA data from McGaw, 2008

23. Between-school differences are small • In the United States: • 74% of the variation in student achievement is within schools, and • 26% of the variation in student achievement is between schools • But, around two-thirds of the “between-school” variation is caused by differences in the students attending that school, which means that • 8% of the variability in student achievement is attributable to the school, so • 92% of of the variability in achievement is not attributable to the school • This means if 15 students in a class reach proficiency in the average school: • 17 students will do so at a “good” school (1sd above mean) • 13 students will do so at a “bad” school (1sd below mean)

24. It’s the classroom… • In the USA, variability at the classroom level is at least 4 times that at school level • As long as you go to school, it doesn’t matter very much which school you go to • But it matters very much which classrooms you are in… • It’s not class size • It’s not the between-class grouping strategy • It’s not the within-class grouping strategy

25. … and specifically, it’s the teacher… Barber & Mourshed, 2007

26. Teachers make the difference • The commodification of teachers has received widespread support • From teacher unions (who understandably resist performance-related pay), because • It doesn’t work (Springer et al., 2010) • It cannot be done fairly in principle. • From politicians (who are happy that the focus is on teacher supply, rather than teacher quality) • But has resulted in the pursuit of policies with poor benefit to cost

27. Impact of background on development (Feinstein, 2003)

28. Meaningful differences • Hour-long samples of family talk in 42 US families • Number of words spoken to children by adults by the age of 36 months • In professional families: 35 million • In other working-class families: 20 million • In families on welfare: 10 million • Kinds of reinforcements: positive negative • professional 500,000 50,000 • working-class 200,000 100,000 • welfare 100,000 200,000 (Hart & Risley, 1995)

29. Teacher quality matters • To see how big the difference is, take a group of 50 teachers • Students taught by the most effective teacher in that group of 50 teachers learn in six months what those taught by the average teacher learn in a year • Students taught by the least effective teacher in that group of 50 teachers will take two years to achieve the same learning (Hanushek, 2006) • And furthermore: • In the classrooms of the most effective teachers • students from disadvantaged backgrounds learn at the same rate as those from advantaged backgrounds • students with behavioral difficulties learn at the same rate as those without behavioral difficulties (Hamre & Pianta, 2005)

30. Improving teacher quality takes time… • A classic labor force issue with 2 (non-exclusive) solutions • Replace existing teachers with better ones • Help existing teachers become even more effective

31. The ‘dark matter’ of teacher quality • Teachers make a difference • But what makes the difference in teachers?

32. Improving teacher quality takes time… • Replace existing teachers with better ones? • Increasing the quality of entrants to exclude the lowest performing 30% of teachers would in 30 years, increase average teacher quality by 0.5 standard deviations. • Cumulatively, one extra student passing a test per class every three years…

33. …so we have to help existing teachers improve… • Improve the effectiveness of existing teachers • The “love the one you’re with” strategy • It can be done • Provided we focus rigorously on the things that matter • Even when they’re hard to do

34. Teachers do improve, but slowly… Leigh, A. (2007). Estimating teacher effectivenessfrom two-year changes in student test scores.

35. People like neuroscience • Descriptions of 18 psychological phenomena • Examples: “the curse of knowledge” • Designed to be comprehensible without scientific training • Each phenomenon was given four possible explanations • Basic (without neuroscience) • Good explanation (provided by the researchers) • Bad explanation (e.g., circular reasoning) • Enhanced (with neuroscience explanation) • Good explanation • Bad explanation • Added neuroscience did not change the logic of the explanation • Participants randomly given one of the four explanations • Asked to rate this on a 7-point scale (-3 to +3).

36. The curse of knowledge • Researchers created a list of facts that approximately 50% of adults would know • Participants were asked which of the facts they knew • For each of the facts, participants were asked what proportion of the adult population would be able to answer the question correctly • Participants gave consistently higher estimates for those items they knew

37. Sample explanations

38. Seductive allure (Weisberg et al., 2008)

39. Brains recognizing words Group-level activations for recognition of words versus a baseline condition (Miller, et al., 2002)

41. Dissociation in the brain representation of Arabic numbers between native Chinese speakers and native English speakers (Tang et al., 2008)

42. Pareto analysis • Vilfredo Pareto (1848-1923) • Economist and philosopher associated withthe 80:20 rule • Pareto improvement • A change that can make at least one personbetter off without making anyone else worseoff. • Pareto efficiency/Pareto optimality • An allocation of resource is Pareto efficient orParetooptimal when there are no more Paretoimprovements • Obstacles to Pareto improvements • The political economy of reform • It is very hard to stop people doing valuable things in order to give them time to do even more valuable things

43. Cost/effect comparisons

44. Relevant studies • Fuchs & Fuchs (1986) • Natriello (1987) • Crooks (1988) • Banger-Drowns, et al. (1991) • Kluger & DeNisi (1996) • Black & Wiliam (1998) • Nyquist (2003) • Dempster (1991, 1992) • Elshout-Mohr (1994) • Brookhart (2004) • Allal & Lopez (2005) • Köller (2005) • Brookhart (2007) • Wiliam (2007) • Hattie & Timperley (2007) • Shute (2008)

45. Feedback has complex effects • 264 low and high ability grade 6 students in 12 classes in 4 schools; analysis of 132 students at top and bottom of each class • Same teaching, same aims, same teachers, same classwork • Three kinds of feedback: scores, comments, scores+comments [Butler(1988) Br. J. Educ. Psychol., 58 1-14]

46. Responses What do you think happened for the students given both scores and comments? • Gain: 30%; Attitude: all positive • Gain: 30%; Attitude: high scorers positive, low scorers negative • Gain: 0%; Attitude: all positive • Gain: 0%; Attitude: high scorers positive, low scorers negative • Something else [Butler(1988) Br. J. Educ. Psychol., 58 1-14]

47. Feedback is not always effective • 200 grade 5 and 6 Israeli students • Divergent thinking tasks • 4 matched groups • experimental group 1 (EG1); comments • experimental group 2 (EG2); grades • experimental group 3 (EG3); praise • control group (CG); no feedback • Achievement • EG1>(EG2≈EG3≈CG) • Ego-involvement • (EG2≈EG3)>(EG1≈CG) [Butler (1987) J. Educ. Psychol.79 474-482]

48. Feedback should feed forward • 80 Grade 8 Canadian students learning to write major scales in Music • Experimental group 1 (EG1) given • written praise • list of weaknesses • workplan • Experimental group 2 (EG2) given • oral feedback • nature of errors • chance to correct errors • Control group (CG1) given • no feedback • Achievement: EG2>(EG1≈CG) [Boulet et al. (1990) J. Educational Research84 119-125]

49. Good feedback leaves learning with the learner • ‘Peekability’ (Simmonds & Cope, 1993) • Pairs of students, aged 9-11 • Angle and rotation problems • class 1 worked on paper • class 2 worked on a computer, using Logo • Class 1 outperformed class 2 • ‘Scaffolding’ (Day & Cordón, 1993) • 2 grade 3 classes • class 1 given ‘scaffolded’ response • class 2 given solution when stuck • Class 1 outperformed class 2

50. Effects of feedback • Kluger & DeNisi (1996) • Review of 3000 research reports • Excluding those: • without adequate controls • with poor design • with fewer than 10 participants • where performance was not measured • without details of effect sizes • left 131 reports, 607 effect sizes, involving 12652 individuals • On average feedback does improve performance, but • Effect sizes very different in different studies • 40% of effect sizes were negative