Using the PADI Design System to Examine the Features of a NAEP Performance Assessment

1. AERA April 2005 Using the PADI Design System to Examine the Features of a NAEP Performance Assessment Kathleen C. Haynie Kathleen Haynie Consulting Andrea A. Lash, Geneva D. Haertel, Edys S. Quellmalz, Angela Haydel DeBarger SRI International

2. 2 Background • Use of 21st century collaborative workplace tools is on the rise • Technologically rich “dispersed” collaboration requires (Olson & Olson, 2000): • Common ground • Coupling of work • Technological readiness • Collaborative readiness

3. 3 PADI Project • Seeks to improve assessment of inquiry in science learning • Networked collaboration • Development and use of Web-based tool

4. 4 Central Process The central process for project strands working with the PADI Design System can be stated as follows: The interaction between assessment materials (e.g., assessment tasks) and the Evidence-Centered Design framework (Mislevy, Steinberg, & Almond, 2002) is mediated by various tools, representational forms, and representations utilized by collaborative workgroups.

5. 5 PADI Assessment Design

6. 6 NAEP Performance Assessment • Goals: • To reverse engineer a performance task from a large-scale, national assessment • To use the PADI System as an analytical tool for understanding task features • In reverse engineering and analyzing a performance assessment task via PADI, what types of knowledge were created?

7. 7 Reverse Engineering & Analysis Process 1. Selection of a Set of Items (July – August) 2. Exploration of the Floating Pencil Task (August – December) 3. Development of a Task Specification (December – March)

8. 8 The Student Model • We selected a set of items for analysis • NSES inquiry standards, PADI Design Patterns • Defined our Student Model • Exploration of multiple student models • Choice of NAEP framework • Refined the Student Model

9. 9 The Evidence Model • We explored evaluation & measurement issues via the PADI representational forms • Evaluative Sub-Model based on NAEP rubric • Multiple levels of conditional dependencies • MRCML psychometric model • Clarified our Measurement Model • Based closely on NAEP practices • Elements of PADI

10. 10 The Task Model • We explored attributes of the Floating Pencil task • Content-lean, inquiry-constrained • Use of lab materials; High verbal demand • Constructed Task Model • Sought to define a ‘Family of Tasks’

11. 11 So What? • Process: Floating Pencil team engaged in collaboration using technological workplace tools • Results: • Analysis of the Floating Pencil task • A ‘trace’ on PADI: Floating Pencil Task Specification • The impact of our work on PADI Design System • Next Steps: • Task family and TMVs • Empirical data analysis • Technical report on Floating Pencil work • Consideration of alternative student models