Encouraging Experimental Evidence in Education & Social Sciences - RCTs in the Social Sciences Conference

1. What works…and who listens?Encouraging the experimental evidence base in education and the social sciences • RCTs in the Social Sciences • 9th Annual Conference • Professor Carole Torgerson • School of Education • Durham University • carole.torgerson@durham.ac.uk • September 11th 2014

4. First known RCT “Five seniors, each of whom had a good scholarship record, pleasing personality, excellent health and fine social environment, were chosen to act as personnel counselors for the members of the freshman class, who at the end of the first eight weeks of school happened to be delinquent in scholarship in the School of Mechanical Engineering in I929-30. The 220 delinquent freshmen were divided into two groups by random sampling.” [italics added] • [Walters, 1931]

5. US legislation: Institute of EducationSciences “Scientifically valid educational evaluation employs experimental designs using random assignment, when feasible, and other research methodologies that allow for the strongest possible causal inferences when random assignment is not feasible.” [page 5] [italics added]

6. Education Endowment Foundation (EEF) • “...all EEF-funded projects are independently and rigorously evaluated...The impact of projects on attainment will be evaluated where possible, using randomised controlled trials.” [italics added]

7. Every Child CountsThe independent evaluation • [Torgerson et al, 2013]

8. Distribution of schools

9. Early reading policy

10. Two alternative designs

11. House of Commons Education and Skills Committee, 18th July 2005 “…in conducting his review, Jim Rose will have the opportunity to draw on the findings of an independent systematic literature review of phonics use in the teaching and application of reading and spelling which we have commissioned from Professor Greg Brooks and Carole Torgerson. This delivers on the public commitment we made…in 2003 to publish an analysis of existing research on phonics teaching methodologies. The aim … is to identify what is known from existing literature about how effective different approaches to phonics teaching are in comparison with each other, including the specific area of analytic versus synthetic phonics.”

12. Meta-analysis: Forest plot

13. Publication bias: Funnel plot • Effect Size

14. Publication bias: Funnel plot • Effect Size

15. Publication bias: Funnel plot • Missing trials? • Effect Size

16. Synthetic versus analytic phonics

17. Policy decision • “I am clear that synthetic phonics should be the first strategy in teaching all children to read.” Ruth Kelly, Secretary of State for Education and Skills Times Mar. 21st 2006 • Sir Jim Rose • Times Mar. 21st 2006 • “The case for synthetic phonics is overwhelming.”

18. “Synthetic phonics does look promising,” says Carole Torgerson…one of the report's authors. “We found it had a moderate effect compared with whole-language approaches, but the evidence base for this conclusion was 12 relatively small trials, only one of which was UK-based. This would be an ideal time to do a national evaluation by implementing systematic synthetic phonics in some schools and not in others and then comparing the two.” Economist Mar. 26th 2006

19. “When the UK government recently introduced the ‘synthetic phonics’ method of teaching young children to read, they were told by Carole Torgerson…that they could easily bolster the slim evidence base by randomising which schools joined the programme first.” Financial Times Mar.18th 2010 “In 2007 the Government introduced a new reading strategy for primary schools based on synthetic phonics, which matches sounds to groups of letters. Professor Torgerson urged ministers to start a randomised trial: the introduction of phonics would have been staggered, with schools chosen at random to start it one year or the next. Every child would have received the intervention, but it would have been possible to compare outcomes and establish whether phonics really works.” Times Sept. 24th 2011

20. Evidence of impact? • ‘KS1 results cast doubt on phonics’ • http://www.tes.co.uk/article.aspx?storycode=6021536

21. Study characteristics of educational trials 1990-2002 • Torgerson et al, British Educational Research Journal 2006, 31:6.

22. Annual conference: Randomised controlled trials in the social sciences

23. Thank you for listening! Rose, J. (2006) Independent review of the teaching of early reading, London: Department for Education and Skills Torgerson, C.J.,Wiggins, A., Torgerson, D.J., Ainsworth, H., Hewitt, C. (2013) Every Child Counts: Testing policy effectiveness using a randomized controlled trial, designed, conducted and reported to CONSORT standards, Journal of Research in Mathematics Education:15(2) Torgerson DJ, Torgerson CJ The design of randomised trials in health, education and the social sciences. Palgrave Macmillan,Basingstoke, 2008 Torgerson, C.J., Brooks, G. & Hall, J. (2006) Literature review of phonics use. London: Department for Education and Skills Torgerson, C. Systematic Reviews, Continuum, London and New York, 2003 Walters, J.E. (1931) The Journal of Higher Education: 2(8), 1931