AERA April 2005

1. Models and Tools for Drawing Inferences from Student Work:The BEAR Scoring Engine Cathleen Kennedy & Mark Wilson University of California, Berkeley AERA April 2005

2. Overview • Features of “complex” tasks. • How PADI addresses complex task features. • The “big” assessment picture and where inferences are drawn (in measurement models implemented in Scoring Engine). • An example of the PADI and Scoring Engine views. • Next steps: Wizard to guide designers in developing solid chain of reasoning. 2 © BEAR Center, 2005

3. Example from FOSS 3 © BEAR Center, 2005

4. Example from FOSS Two measures: Physics (speed) Mathematics 5 responses: Equation choice Fill-in Numbers Fill-in Units Calculate Units in Answer Are the responses dependent? 4 © BEAR Center, 2005

5. “Complex” Task Features • Multiple measures of interest • Content & inquiry • Multiple aspects of inquiry • Response dependencies • Common stimulus • Sequence of steps 5 © BEAR Center, 2005

6. Complex Measurement Requires Clear Chain of Reasoning Inferences one wishes to draw (cognition vertex) Evidence required to draw the inferences (interpretation vertex) Observations required to generate evidence (observations vertex) Inferences are then interpretable in the context of the purpose of the assessment 6 © BEAR Center, 2005

7. PADI Addresses “Complex” Task Features PADI Approach: Multidimensional IRT measurement model. Well-defined evaluation phases model response dependencies (rather than ignoring them). • Multiple measures of interest • Content & inquiry • Multiple aspects of inquiry • Response dependencies • Common stimulus • Sequence of steps (within task) 7 © BEAR Center, 2005

8. Task Specifications Student Database Delivery System Assessment System Architecture Design Implementation Design System Students Design Team Scoring Engine 8 © BEAR Center, 2005

9. Task Specifications Chain of Inferential Reasoning Assessment Purpose Student Database Assessment Evidence Design System Delivery System Students Design Team Scoring Engine Inferences about what students know and can do 9 © BEAR Center, 2005

10. FOSS Example: PADI View Two student model variables: Physics (speed) Mathematics 6 observable variables: Equation choice (Physics) Fill-in Numbers (Physics) Fill-in Units (Physics) Calculate (Math) Units in Answer (Physics) Bundled physics items 10 © BEAR Center, 2005

11. FOSS Example : Bundling Rules • Defined in PADI Design System • Template (task specification) • Activity • Evaluation Procedure • Evaluation Phase • Implemented in Delivery System 11 © BEAR Center, 2005

12. FOSS Example : Bundling Rules 12 © BEAR Center, 2005

13. FOSS Example : Scoring Engine View Two student model variables: Physics (speed) Mathematics 2 observable variables: Math score Bundled Physics score “Between Item” MD: Each observable variable provides evidence of one SMV. Scoring Engine returns two proficiency estimates per student to the Assessment Delivery System. 13 © BEAR Center, 2005

14. Next Steps Develop Measurement Model Design Wizard • Evaluate design needs of users (how do they do it now, what would work better?) • Guide thinking from the “assessment purpose” standpoint • Align inferences, evidence and observations 14 © BEAR Center, 2005