Judith Bennett, University of York, UK

1. Using evidence to inform practice in science teaching: the promise, the practice, and the potential Judith Bennett, University of York, UK

2. What policy-makers want … We need to be able to rely on … social scientists to tell us what works and why, and what types of policy initiatives are likely to be most effective. David Blunkett, Secretary of State for Education, 2000

3. Some key questions • What constitutes good evidence? • How might such evidence be gathered? • How might such evidence be used to inform: • the development curriculum intervention programmes • classroom practice • policy decisions in science education?

5. The promise • There has been considerable debate over the last fifteen years about the nature of educational research, and a drive to improve it quality through adopting a more scientific approach. • The promise is that this will provide a much sounder evidence base to inform decision making.

6. Reviews of research Reviews of research are undertaken for a variety of purposes: • an entity in themselves as an expert review paper in a journal • part of a bid for research funding • a section in a research thesis

7. Systematic reviews … 1 Systematic review methods involve: • developing systematic search strategies for reports of research studies based on specific criteria • coding the studies against pre-specified and agreed characteristics • generating an overview or map of the area • looking in detail at specific aspects of studies • synthesising the findings and making judgements about the quality of the evidence presented

8. Systematic reviews … 2 • In the UK, systematic reviews have been sponsored by the Government, who funded: • the establishing of the Evidence, Policy and Practice Initiative Centre (EPPI Centre) • the Training and Development Agency for Schools (TDA) • Reviews have been undertaken using methods developed by the EPPI-Centre • The bulk of the reviews were undertaken in the first half of the 2000s

9. Some Review Groups

10. Systematic reviews in science education • The impact of context-based and science-technology-society (STS) approaches on students understanding of, and attitudes to, school science • The use and effects of small-group discussion work in school science teaching • the impact of ICT on science teaching

11. What are context-based approaches? • Contexts and applications of science are used as the starting point for developing scientific understanding • Contexts can be social, economic, environmental, technological and industrial applications of science

13. Why look at this topic? • Advocates of context-based approaches argue that: • there are improvements in both understanding of science and attitudes to science as a result of their use • one outcome is increased uptake of science subjects post-16 • Those who are less persuaded argue that that: • students do not acquire a systematic grasp of scientific ideas and so develop a poorer overall understanding

14. The review • 2,500 studies identified • 61 met the inclusion criteria (one of which was that the study was some form of evaluation) • 24 experimental designs • 44 reported on attitudes to science, 41 reported on understanding of science ideas • 35 focused on science, 13 on chemistry, 10 on physics and 3 on biology • data gathered through test results, questionnaires and interviews

15. Making judgments about quality • focus of the study (understanding and/or attitude, with these as explicit independent variables) • research design (including sample size and matching of control and experimental groups) • the nature of the data collected (pre and post intervention, or post intervention) • reliability and validity of the data collection methods and tools • reliability and validity of data analysis • range of outcome measures • 17 studies were judged to be of good quality or better and included in the detailed review - all experimental designs focusing on effects on understanding and/or attitudes

16. Selected review findings • Attitudes to school science almost always improved • Attitudes to science also improved, but not as much as to school science • Both boys and girls demonstrated more positive attitudes, with the biggest change being for girls • The majority of studies found that context-based approaches provided as good a development of understanding as more conventional approaches • Some evidence that understanding was improved • Some small increases in numbers taking science subjects • Some evidence of difficulties in measuring effects

18. Why look at small-group discussions? • Many people involved in teaching and curriculum development in science believe that small-group discussions are an important tool in science teaching, motivating students and enhancing their learning • They are advocated in: • course aimed at developing scientific literacy • constructivist approaches in science teaching • developing techniques to enhance assessment for learning • Their introduction into science lessons challenges the established pedagogy of science teaching, and places new demands on teachers • Many of the studies are qualitative studies

19. The review • 2,300 studies identified • 94 met the inclusion criteria (one of which was that the study was some form of evaluation) • just over half comprised case studies • 28 experimental designs • 69 reported on understanding of science ideas • most looked at some form of ‘collaborative learning’ skills • the majority focused on biology and physics topics • data mainly gathered via video-tapes and audio-tapes of discussions and interviews, with some use made of questionnaires and test results

20. Making judgments about quality • justification for choice of sample • details of data collection methods and instruments • measures for increasing trustworthiness of the data collected (e.g. corroborating evidence, multiple perspectives, depth, detail and richness of data) • the approach to analysis • the generation of criteria for effectiveness or impact • how evaluative judgements were reached • relatability of the findings • 12 studies of good quality or better and included in the detailed review

21. Selected review findings • Small group discussions used in a wide variety of ways, and the term often used very loosely • Students often struggled to formulate and express coherent arguments and demonstrated a low level of engagement with tasks • Very clear evidence of teachers and students needing to be given explicit teaching in the skills associated with development of arguments and characteristics associated with effective group discussions • Groups functioned better when the stimulus used to promote discussion involves both internal and external conflict • Methodologically, there were two very contrasting approaches to analysis • Grounded theory • Use of existing analysis tools

22. Issues • What does ‘what works’ mean? • Experimental studies - how desirable and possible are they? • ‘Review distortion’? • Does systematic = objective? • Lack of standardisation of instruments • Developer or evaluator? • ‘Ground clearing’, or a secure base for decision-making? • Communicating the findings to different audiences

23. The potential – for educational research • A knowledge of systematic review methods is likely to enhance any review of research • More informed discussion about when experimental methods might be appropriate • Help identify under-researched areas • Points to the need to give very careful consideration to the generation of a resource bank of instruments • To consider how research findings are best communicated to different audiences

24. The potential – for policy and practice • Hmmmm …. • Not yet there with the means of providing a secure evidence base to inform policy decisions (if that is really what policy-makers want) • If we can garner good evidence, we do know quite a bit about how to help teachers draw on it in their teaching …

25. “Teachers will not take up attractive-sounding ideas, albeit based on extensive research, if these are presented as general principles which leave entirely to them the task of translating them into everyday practice - their classroom lives are too busy for this to be possible … What they need is a variety of living examples of implementation, by teachers with whom they can identify, and from whom they can derive both the conviction and confidence that they can do better, and see concrete examples of what doing better means in practice.” Black, P. and Wiliam, D. (1998) Inside the black box: raising standards through classroom assessment. London: King’s College. A final word …

26. More information … Judith Bennett: judith.bennett@york.ac.uk • The detailed review reports: http://eppi.ioe.ac.uk(in REEL - the Research Evidence in Education Library) • The summary booklet: www.york.ac.uk/educ/ResearchPaperSeries(Paper 12 and Paper 21) • Bennett, J., Lubben, F. and Hogarth, S. (2007) Bringing science to life: a synthesis of the research evidence on the effects of context-based and STS approaches to science teaching. Science Education, 91 (3), 347-370. • Bennett, J., Hogarth, S., Lubben, F., Campbell, B. and Robinson, A. (2010) Talking Science: The research evidence on the use of small group discussions in science teaching. International Journal of Science Education, 32 (1), 69-95.