Tools to Measure Student Progress

1. Tools to Measure Student Progress Session presented at the CAESL conference, San Jose, May 17, 2007 presented by CAESL Fellows Diana Bernbaum, Brent Duckor, & Rachael Tan Berkeley Evaluation & Assessment Research (BEAR) Center University of California, Berkeley

2. The Problem • National and state policy push • NAEP science • NAGB Standards • Myriad of rich, engaging curricula • LHS-FOSS-ASK • Emerging landscape of tests and assessments • Authorities (teachers, curriculum developers, testing companies and consultants) • Quality (reliability, validity, usability) • Scalability (beyond classroom or a few schools) • The quality of any system (policy, curriculum & assessments) depends on the quality of the inferences and evidence © BEAR Center

3. Observation Interpretation NRC Assessment Triangle Cognition Design a Clear Chain of Reasoning 3. What observations are required to generate the evidence ? 2. What evidence is required to draw the inferences ? 1. What are the inferences one wishes to draw? © BEAR Center 3

4. The Proposed Framework • BEAR Assessment System Principles • Assessments should be based on a developmental perspective of student learning. • Assessments should be clearly aligned with the goals of instruction. • Assessments must produce valid and reliable evidence of what students know and can do. • Assessment data should provide information that is useful to teachers and students to improve learning outcomes. © BEAR Center

5. Focus questions • How can we know what our students know and can do in K-6 science? • How can we as curriculum developers who work closely with teachers assure ourselves that our measures of student progress are meaningful, consistent, and fair? © BEAR Center

6. Today’s Agenda • Put on your “teacher hat” • Guiding Questions • What do student responses to this problem tell you about what students know? • How would you as a teacher organize/make sense of students’ responses? • What are the critical junctures that help describe students’ level of understanding? • How can a Performance Map be used to help understand student progress over time? © BEAR Center

7. LHS-FOSS Curriculum & ASK project • LHS-FOSS curricular domains • 35 modules for K-8 • 4 strands/content areas: Life Science, Physical Science, Earth Science, and Scientific Reasoning and Technology • Scope of ASK project within LHS-FOSS • 4 year project (focused on 3-6th grades) • Develop assessments for 16 modules • Goals: • define learning progressions • to field test items • validate assessment tools and techniques • Purposes: • To enhance student learning • To support teacher effectiveness • To provide accountability data in science education © BEAR Center

8. Today’s presentation focuses on • Physical sciences strand • Mixtures and Solutions (MX) module • 4 investigations • Separating mixtures • Saturation • Concentration • Chemical reactions © BEAR Center

9. Activity: Background • Concentration refers to the amount of material (solute) dissolved in a given amount of liquid (solvent). • The more material dissolved in a given amount of liquid, the more concentrated the solution. © BEAR Center

10. Activity: Making and Comparing Salt Solutions • Salt Concentration – Parts 1-3 • Materials: • Three plastic cups • Spoon • Syringe • Balance • Stirring stick • Water container © BEAR Center

11. Activity: Making and Comparing Salt Solutions, cont. • Part 1 • Follow instructions to make two salt solutions. • Part 2 • Use the balance to compare the two solutions. In each drawing circle the solution that is heavier. • Part 3 • Make predictions about how a new solution will compare to the solutions you made in Part 1. • Make the new solution and summarize the results. • Response Sheet © BEAR Center

12. Activity: Guiding Questions • While working in your groups, think about the following: • What do student responses to this problem tell you about what students know? • Observations during activities • Written work • Predictions • Summaries • Response Sheet • Time: 20 minutes © BEAR Center

13. Understanding Student Responses • Student responses from Sheet No. 17 • Understanding Concentration • Guiding Question: How would you as a teacher organize/make sense of students’ responses? • Not to assign a grade – to give students feedback and to use information to adjust instruction © BEAR Center

14. Categorizing Student Responses • Can you categorize students based on their responses to this item? • Summarize group results • Ranking • Qualitative Descriptors • What kind of formative feedback would you give? • Time: 10 minutes © BEAR Center

15. Guiding questions • What are the salient features that help describe students’ level of understanding in each category? • What are the differences among categories? • What places a response in one category rather than another? © BEAR Center

16. How did we create the Categories? • Describe levels of student understanding based on experience with module and item focus. • Building Relationships: Beyond what students learned in class, synthesizing material, applying knowledge to new situations • Making Connections: Connecting pieces of information learned in class • Recognition: Easier factual questions - multiple choice, basic definitions • Notions: Cannot answer Recognition-type questions. © BEAR Center

17. How did we create the Categories? • Statistical analyses of student work • Item difficulty, qualitatively and quantitatively • Do they support our original descriptions? • Adjust scoring guides, descriptions © BEAR Center

18. Performance Categories for Concentration • Building Relationships • Can apply understandings about solubility to new contexts. • Making Connections • Can compare the concentration of solutions using the ratio of the solute to the solvent. • Recognition • Can define concentration. Needs to consider both the amount of solvent and solute of solutions when comparing concentration. • Notions • Knows when equal amounts of solutions are placed on a balance, the more concentrated solution will be heavier. Needs to learn about concentration of solutions. © BEAR Center

19. Group Performance on One Assessment © BEAR Center

20. Guiding Question How can a Performance Map be used to help understand student progress over time? • Many items measure the same concept and are administered at multiple time points. • Before instruction, during instruction, and after instruction Why use a Performance Map? • To interpret student work in the context of instructional goals © BEAR Center

21. Student Performance on Concentration Activity © BEAR Center

22. Progress Map - Tara © BEAR Center

23. Progress Map - Chance © BEAR Center

24. Detail Reports • Diagnostic Report by Item • Students listed according to how they scored on each item and activity Item: MX46 1 2 3 ------------ ------------ ------------ Rhonda Tara Tsotsi Xiaoxia Andrea Pacey Carey Miguel Yong 1 - Took the wrong focus. 2 - Considers only the amount of solvent or solute when comparing the concentration of two or more solutions. 3 - Can compare the concentration of solutions using the ratio of the solute to the solvent. © BEAR Center

25. Detail Reports • Ability Estimates by Level • Students listed according to their current level of understanding on each concept Key Concept – Level: Concentration – Making Connections Can compare the concentration of solutions using the ratio of the solute to the solvent. Name -------- Andrea Pacey Tsotsi Level: 9 Students 18.4% of valid © BEAR Center

26. Conclusion • Review Answers to Guiding Questions • Student responses to multiple problems provide information about what students know about a given topic. • It is difficult to organize/make sense of students’ responses without an existing framework, like a Performance Map. • A Performance Map can be used to measure student progress over time. © BEAR Center

27. Discussion Questions • What are the challenges that a practitioner would face in using this system? • Can other tools, along with Performance Maps, be used to measure what students know and can do in K-6 science? © BEAR Center