On Science Inquiry and Formative Assessment

1. On Science Inquiry and Formative Assessment Richard J. Shavelson Stanford Education Assessment Lab Keynote Address NSTA Science Assessment Conference Research & Practical Approaches for Classroom Teachers, School Administrators, & School Districts Chicago, Illinois November 9, 2005

2. Thanks To NSF And Project Officers Over The Years! Indicator systems for monitoring the health of science and mathematics education Dick Berry AERA Grants Program Dick Berry Larry Suter New technologies for assessing science performance Susan Snyder �Romance� study Larry Suter Janice Earle Capital�Teacher formative assessment project Barry Sloane Center for Assessment and Evaluation of Student Learning (CAESL) Janice Earle

3. The �Romance� StudyCollaborative Research Rich Shavelson Maria Araceli Ruiz-Primo Carlos Ayala Yue Yin Erin Furtak Noah Feinstein Marjorie Lucks Jim Vanides Don Young Frank Pottenger Paul Brandon Miki Tomita Rachel Haynes Alice Taum Mary Gray

4. Goals: Develop Students� Understanding Of The natural world by Moving students from na�ve conceptions of the natural world into scientifically justifiable conceptions (�conceptual change�) Getting students to provide evidence to support their conceptions and their general applicability How science creates and warrants new knowledge

5. Potential Of Formative Assessment A set of potentially powerful pedagogical ideas and acts Links teaching, learning and assessment for changing conceptions that include: Creating/capitalizing on teachable moments and Providing immediate feedback on how to improve understanding

6. Challenges Many teachers are not prepared to incorporate formative assessment into their teaching practice ... perhaps for good reason: Conflict between their summative (grading) and formative (teaching) roles View that assessment is not part of teaching and learning Need to change beliefs and conceptions about assessment, content and pedagogy to integrate assessment into teaching Inconsistent findings on the impact of enhancing teachers� formative assessment student achievement However, we need to succeed if the Black & Wiliam research review is correct, and some lessons are provided here

7. Formative & Summative Assessment

8. Types of Formative Assessment

10. Romance Project Synopsis Study tests Black and Wiliam�s (1998) contention that: If feedback based on formative assessment is closely connected to instruction and provides information about how to improve performance, this feedback would produce a large positive effect on students� learning. A knowledge-type framework was used to create formative (formal embedded) and summative (end-of-unit) assessments, for a unit on buoyancy The Romance project culminated in a small randomized trial that tests Black and Wiliam�s contention focusing on: Student learning Teacher implementation (especially formative feedback to students)

11. The �Romance� Study:Research Framework and Questions Embedded a suite of formative assessments into a nationally used science-inquiry curriculum First twelve investigations focus on explaining �Why things sink and float?� Tested, in a small study, Black and Wiliam�s contention that formative assessment significantly improves student learning

13. Our Initial Conception of Embedded Assessment Built assessments according to a knowledge theory of science achievement Embed (�reliable�) assessments at critical joints in unit Provide information to teachers & immediate feedback to students Observe our success in pilot study!

14. Embedded Assessment Suites� Theory OK/Practice Unwieldy! Assessment suite took too long to implement Feedback delayed in spite of �rules of thumb� for reducing time Information overloaded and at times conflicted for teacher to use immediately Teachers did not know how to use information to close gap Teachers used embedded assessments in their summative assessment scripts�something outside teaching and learning When teachable moment arrived, formative feedback from teachers was absent

15. Reflective (Embedded) Lessons(To Avoid Assessment Script!)

16. Types of Reflective Lessons Reflective Lessons Type I � Focus on the use of evidence to support students� conceptions and/or models of why things sink and float: Interpreting and evaluating a graph Predict-observe-explain activity an event related to sinking and floating Short answer question �Why Do Things Sink and Float?� and Predict-observe activity an event related to sinking and floating Reflective Lessons Type II � Focus on checking students� progress in their conceptual understanding of the key concepts across the twelve investigations with a concept map:

17. SEAL Reflective Lesson Timeline

18. Sequence of Each Reflective Lesson

19. Development Framework for Pedagogical Actions

20. Experimental Group: Preparation For Formative Assessment Teaching Five full days of �training� Teachers iterated through each Reflective Lesson: Participate as �students� taught by staff Discuss RL as teacher colleagues Practice teaching students in laboratory school while being observed by colleagues and staff mentor Reflect with colleagues and mentor on implementation of RL Follow up during their classroom teaching of unit

21. Some Training Challenges Too much material in short period of time Assistance implementing Reflective Lessons throughout the unit provided but teachers too busy unless on we�re on site Beliefs about what constituted inquiry teaching and learning overwhelmed the training program

22. Results

25. Concluding Thoughts I As we know, when any new reform idea such as formative assessment, comes along, there�s: A lot of hype Teachers are expected to pick up the �tools� and implement the new ideas perfectly on the first try In reality, new ideas take a great deal of time to go beyond the talk and become not only practice, but also �belief��reform is about conceptual change!

26. Concluding Thoughts II Formative assessment holds promise�our teachers who actually taught in ways closest to the vision were more effective than those who held na�ve conceptions of inquiry teaching (e.g., never tell students the answer!) They were in both the experimental and control groups They varied in their experience They believed (our inference from observation and discussion) that students, with appropriate guidance, could construct conceptions of sinking and floating at the top level of the developmental model

27. Concluding Thoughts III Bottom line� Formative assessment links teaching, learning and assessment There�s nothing wrong with the conception We�all of us here today�need to understand what it takes to put it into practice effectively and then test it out!

28. Thank You!