Assessment of skills and competences in science

1. Assessment of skills and competences in science Wynne Harlen

2. OECD Skills Strategy • “In the context of the OECD Skills Strategy, the concepts of ‘skill’ and ‘competence’ are used interchangeably. By skill (or competence) we mean: the bundle of knowledge, attributes and capacities that enables an individual to successfully and consistently perform an activity or task, whether broadly or narrowly conceived, and can be built upon and extended through learning.” Towards an OECD Skills Strategy (OECD 2011 p7 foot note)

3. Agenda • Starting point: aims of science units and courses (Access and Higher) • Inquiry as a means to developing scientific literacy • Research into development of understanding through inquiry • Challenges to assessing understanding and skills (particularly context sampling error ) • Examples of questions assessing skills and understanding in science • Alternatives to external papers • Pros and cons of using teachers’ assessment.

4. Unit and course outcomes Unit assessment • “Learners who complete this Unit will be able to: • apply skills of scientific inquiry and investigation in the context of the unit (eg physics of dynamics and astrophysics) • demonstrate knowledge and understanding related to the content of the units (eg physics of dynamics and astrophysics) “

5. Scientific literacy as an aim • “The Course will therefore encourage learners to become scientifically literate citizens, while developing their literacy and numeracy skills. It will also develop learners’ investigative and experimental skills in a (physics) context. Learners will recognise the impact (physics) makes on their lives, the environment and society. Through this Course, learners can develop relevant skills for learning, for use in everyday life and across all sectors of employment.”

6. A note on scientific literacy (SL) • Scientific literacy as a metaphor in which the traditional sense of literacy assumes another meaning (Bybee). • SL refers to being well-educated and well-informed in science. It does NOT mean being able to read and write about science or being able to understand science vocabulary. • Development of SL: scientific illiteracy --> nominal SL --> functional SL --> conceptual and procedural SL --> multidimensional SL • Through scientific inquiry students’ advance their understanding of scientific concepts and the nature of science, that is, procedural literacy.

7. The role of inquiry in science education • Through inquiry students progressively developing key scientific ideas through learning how to investigate and build their knowledge and understanding of the world around. They use skills employed by scientists such as raising questions, collecting data, reasoning and reviewing evidence in the light of what is already known, drawing conclusions and discussing results. (based on IAP 2011)

8. Is it necessary to assess inquiry skills? • Yes, if inquiry skills and competencies are seen as key outcomes in their own right, not just a means to developing understanding (of science and about science) • Skills needed for continued learning – • Students cannot learning in school everything they will need to know in adult life. What they must acquire is the prerequisites for successful learning in future life….Students must become able to organise and regulate their own learning, to learn independently and in groups and to overcome difficulties in the learning process. This requires them to be aware of their own thinking and learning strategies and methods. (OECD 1999)

9. How does this development of understanding happen? New experience/question Alternative ideas Bigger idea Possible explanation Existing idea Prediction Plan and conduct investigation Interpret data Conclusion

10. Does inquiry ‘work’? • Do inquiry-based activities lead to understanding more than do more direct teaching approaches? • Inquiry synthesis project Minner, D.D., Levy, A.J., & Century, J. (2010). Inquiry-based science instruction – what is it and does it matter? Results from a research synthesis years 1984-2002. Journal of Research in Science Teaching, 47 (4), 474-496

11. Inquiry Synthesis Project • Synthesis of 138 studies • 30 experimental; 35 quasi-experimental; 73 non-experimental • Most (105) conducted in the United States • Amount of instruction delivered in the interventions varied widely (mean ~ 12 lessons) • Most (56%) did not report if the outcome measurement instruments were new or existing • Generally low rigour regarding reliability and validity of the measures used

12. Inquiry Synthesis Project findings • The majority of studies (51%, n=71) showed positive impacts of some level of inquiry science instruction on student content learning and retention • The extent of ‘inquiry saturation’, or specific elements of inquiry, or specific components of instruction was not statistically significantly associated with better learning outcomes • Active thinking was a significant predictor of learning outcomes • Drawing and thinking about conclusions from data was a marginally significant predictor of learning outcomes • 5 studies (of the 6 that were reviewed) of the effect of hands-on activities on student learning showed a statistically significant difference • 6 studies (of the 9 that were reviewed) of the effect of giving students greater responsibility showed a statistically significant difference

13. Course assessment • “Learners will draw on, extend and apply the skills they have learned during the Course. • This will be assessed within • a question paper (external paper) • and a case study (stimulus-based assessment of ability to analyse and draw conclusions) • requiring demonstration of the breadth of knowledge, skills and understanding acquired from across the Units and of how they can be applied in unfamiliar contexts and/or integrated ways.”

14. Challenges for assessment • To show that the skills, knowledge and understanding can be used in ‘unfamiliar contexts’ • The need for authentic situations, real contexts • Complexity in presenting the context • Bias of familiarity with the context • Preference for ‘unfamiliar contexts’ in order to assessment application of understanding and capabilities • Context sampling error • Impact on reliability and validity

15. Sources of error in assessment of understanding • Requires application of knowledge, in a context • How to choose – what are ‘unfamiliar contexts’? • Lack of knowledge or lack of ability to apply? • Context sampling error (Wiliam 2001).

16. The context effect in assessment of inquiry skills • (Brown et al 1996) • ‘Little evidence of the generalisability of skills assessment across science subjects’ • Need for continued assessment of skills within subject domains • High context sampling error when few contexts can be used. • Possible solutions: • Find contexts likely to be unfamiliar to all students • Increase contexts by assessing smaller, specific aspects of inquiry • Use the range of contexts encountered in the course for moderated teacher’s assessment (e.g. portfolio).

17. Important differences between assessing populations and assessing individuals • PISA and national monitoring programmes • Can minimise the context sampling error by using a bank of items and random allocation of items to students • But, restricted in the form of the item on paper or on screen • For population, high reliability (accuracy) but lower validity • Assessing individuals • Using examinations and tests, same items of all, limited range of contexts • Large context sample error; low validity • Using performance in regular and special inquiry-based tasks raises validity • Higher reliability tends to lower validity.

18. Cooling Relative Importance Heating Carbon dioxide Methane Particles Particle effects on clouds known effect possible effect What Human activities contribute to climate change?Read the following information and answer the questions which follow. The burning of coal, oil and natural gas, as well as deforestation and various agricultural and industrial practices, are altering the composition of the atmosphere and contributing to climate change. These human activities have led to increased concentrations of particles and greenhouse gases in the atmosphere. The relative importance of the main contributors to temperature change is shown in the figure

19. Bars extending to the right of the centre line indicate a heating effect. Bars extending to the left of the centre line indicate a cooling effect. The relative effect of ‘Particles’ and ‘Particle effects on clouds’ are quite uncertain: in each case the possible effect is somewhere in the range shown by the light grey bar. Figure 1 shows that increased concentrations of carbon dioxide and methane have a heating effect. Increased concentrations of particles have a cooling effect in two ways, labelled ‘Particles’ and ‘Particle effects on clouds’. • Use the information in Figure 1 to support the view that priority should be given to reducing the emission of carbon dioxide from the human activities mentioned. • Use the information in Figure 1 to support the view that priority should not be given to reducing the emission of carbon dioxide from the human activities mentioned. Questions

20. Early immunisation • As early as the 11th century , Chinese doctors were manipulating the immune system. By blowing pulverised scabs from a smallpox victim into their patients’ nostrils. They could often induce a mild case of the disease that prevented a more severe onslaught later on. In the 1770s people rubbed their skins with dried scabs to protect themselves from the disease. These primitive practices were introduced into England and the American colonies. In 1771 and 1772, during a smallpox epidemic, a Boston doctor named Zabdiel Boylston tested an ideas that he had. He scratched the skin on his six year old son and 285 other people and rubbed pus from small pox scabs into the wound. All but six of his patients survived. • Questions: • 1. What idea might Zabdiel Boylston have been testing? • 2. Give two other pieces of information that you would need to decide how successful Boylston’s approach was

21. Chocolate • A newspaper article (Daily Mail on March 30, 1998) recounted the story of a 22 year old student, named Jessica, who has a ‘chocolate diet’. She claims to remain healthy, and at a steady weight of 50Kg, whilst eating 90 bars of chocolate a week and cutting out all other food, apart from one ‘proper meal’ every five days. A nutrition expert commented: “I am surprised someone can live with a diet like this. Fats give her energy to live but she is not getting a balanced diet. There are some minerals and nutrients in chocolate, but she is not getting enough vitamins. She could encounter serious health problems in later life.”

22. Imaginary animal

23. Threats to validity • Communicating an unfamiliar context makes demands on reading, interpretation of representations • The extent to which the assessment specific skills equates to assessment of capability in inquiry and investigations • Other features of context (beyond familiarity) may affect engagement • What was once a ‘novel’ context can become familiar once used • Science fiction a questionable context for assessing understanding and skills in science.

24. Some advantages of assessment by teachers • Potential for the full range of goals to be included as teachers collect evidence as part of their normal work with students • Can relieve the pressure, on students and teachers, of terminal tests and examinations • Teachers can use information about students formatively as well as summatively • Can release resources (time and other costs) for alternative use

25. Some disadvantages • Teachers’ judgements often perceived as being unreliable • Increase in work load for teachers • Can lead to the same distortion of teaching as testing if the results are used for high stakes accountability. .

26. Dimensions of approaches to assessment by teachers 2 1 Greater specification of tasks 3 4 Assessment criteria more detailed

27. A portfolio system (eg Queensland) • School-based • Constructed over all units in the course • Selected and assessed by reference to strict criteria • Full range of evidence – projects, assignments, observations of practical performance, field work • Mandatory criteria • Up-dated to include most recent relevant evidence • Five levels of achievement • Moderated.

28. A formative function • Assessment of any kind should ultimately improve learning (Harlen, 2010) • The collection of data during the year informs the teacher of the areas of more and less progress of students (formative use of summative information) • The practice of assessment for learning can inform the judgement of achievement at a particular time (summative use of formative information).

29. References Brown, CR., Moor J., Silkstone, B.E., and Botton, C. (1996) The construct validity and context dependency of teacher assessment of practical skills in some pre-university level science examinations, Assessment in Education, 3 (3) 377-391. Bybee, R., Fensham, P and Laurie, R (2009) Scientific literacy and contexts in PISA 2006 science. Journal of Research in Science Teaching 46 (8) 862-864. Harlen, W. (2001) The assessment of scientific literacy in the OECD/PISA project. Studies in Science Education,Vol 36, 79-104 Harlen, W. (2005)Trusting teachers’ judgment: research evidence of the reliability and validity of teachers’ assessment used for summative purposes. Research Papers in Education 20 (3) 245-270 Harlen, W. (2010) What is quality teacher assessment? In Gardner et al Developing Teacher Assessment. Maidenhead: Open University McGraw-Hill IAP (InterAcademies Panel) (2011) Taking IBSE into Secondary Education. Report on a conference held in York, October 2010. London: Welcome Trust. Minner, D.D., Levy, A.J., & Century, J. (2010). Inquiry-based science instruction – what is it and does it matter? Results from a research synthesis years 1984-2002. Journal of Research in Science Teaching, 47 (4), 474-496 Wiliam, D. (2001) Reliability, validity and all that jazz, Education 3-13 (3) 17-21