Overview of the PADI Design System

1. AERA April 2005 Overview of the PADI Design System Robert Mislevy, University of Maryland Geneva Haertel, SRI International Robert Murphy, SRI International

2. 2 The Problem • Creating good science inquiry tasks is hard. • How to do it both soundly and efficiently? • What’s the link with science standards? • What’s the psychological perspective represented? • How to design assessments for different content areas & ages of students? • How do you score it? • How to sort out confounded evidence? • How to leverage the use of technology?

3. 3 The PADI Solution (1) • Conceptual frameworks and representational (“epistemic”) forms… • Design patterns for assessing inquiry • Task-design templates • Assessment delivery system • Object model for task design • Software tools to aid the design process • Optional scoring and calibration engine to sort out evidence

4. 4 The PADI Solution (2) • Libraries of exemplars… • Design Patterns for assessing inquiry • Task templates and pointers to tasks • Worked-through applications • FOSS, BioKIDS; GLOBE and others • Evaluation studies • FOSS, BioKIDS

5. 5 Project Objectives To produce a conceptual framework and collection of design tools and resources for supporting the design of assessments of science inquiry. • Formulate a design framework for science inquiry that is extensible, shareable, and practical • Lay out the framework in a open-system object model • Develop supporting software to create and work with design patterns and task templates • Develop an optional scoring and calibration engines for the sorting out of evidence • Provide an initial set of high quality exemplars to highlight the elements of a coherent assessment argument. • Test out the utility of the framework, object model, and tools with applications (GLOBE, FOSS, BioKids etc.)

6. 6 Scope of PADI project • Not an authoring system. Conceptual framework and object model provide infrastructure around which tailored authoring systems can be built. • Not a delivery system. Shared conceptions, representations, and object definitions enhance the efficiency of delivery system design by providing a common infrastructure (i.e. delivery system should be consistent with principles of ECD). • Does not set out to define inquiry based on a single perspective. Structures for expressing assessment arguments (Design Patterns) and instantiating them in tasks (Task Templates) independent of the psychological perspective.

7. Design Evidence- Models centered design (Conceptual) "Scoring engine" Delivery System Architecture Calibration RunTime Design Tools (Specific incarnation) Design Three-tiered Object Use Cases Software architecture Model 7 Main Components of PADI PADI Science Inquiry Other examples Science Cog Psych PADI Libraries Standards Literature Applications Exemplars Design Patterns Templates Globe BioKIDS FOSS Conceptual Conceptual models models Conceptual Application models Application Design Design Application Design Implementation Implementation Implementation Evaluation study

8. 8 Evidence Centered Design • The assessment problem: To draw inferences about student proficiencies based on limited observations. • Evidence-Centered Design: Defines elements of the assessment argument and relationships between them. • What does it mean to know and do inquiry? • What constitutes evidence of knowing? • How can that evidence be elicited from students? • What are the appropriate statistical techniques for making valid student inferences about what students know from what students do? • PADI is implementing a special case of ECD to the study of science inquiry.

9. Design Evidence- Models centered design (Conceptual) Delivery "Scoring System engine" Architecture Calibration RunTime 9 Evidence-Centered Design (1)

10. Design Evidence- Models centered design (Conceptual) "Scoring Evidence Models Task Models Student Model(s) engine" Delivery Evidence Stat System Rules model Architecture Feature s 1. xxxxx Calibration RunTime 2. xxxxx xxxxx 3. 10 Assessment Design Models (2)

11. The Four-Process Delivery System Design Evidence- Models centered design (Conceptual) Presentation ActivitySelection "Scoring Process Process engine" Administrator Examinee Delivery Task/ Evidence System Composite Architecture Library Calibration RunTime Task Level Evidence Feedback Summary Feedback Evidence Accumulation Identification Process Process (Summary (Response Scoring) 11 Processing) Assessment Design Models (3)

12. The Four-Process Delivery System Design Evidence- Models centered design (Conceptual) Presentation ActivitySelection "Scoring Process Process Administrator engine" Examinee Delivery Task/ Evidence System "Scoring Composite Architecture Library engine" Calibration RunTime Task Level Calibration RunTime Feedback Evidence Summary Feedback Evidence Accumulation Identification Process Process (Summary (Response 12 Scoring) Processing) Assessment Design Models (3a)

13. Science Inquiry Other examples Science Cog Psych PADI Standards Literature Applications 13 Paradigms for Assessing Science Inquiry

14. Design Tools (Specific incarnation) Design Three-tiered Object Use Cases Software architecture Model 14 Software Support & Architecture for Designing Assessment Objects

15. PADI Applications PADI Applications Globe BioKIDS FOSS Conceptual Conceptual models models Conceptual Application Application models Design Design Application Design Implementation Implementation Implementation 15

16. BioKIDS FOSS Conceptual Conceptual models models Application Application Design Design Implementation Implementation Evaluation study 16 Evaluation Studies

17. Evidence- centered design Science Inquiry PADI Libraries Design Tools Applications (Specific incarnation) Design Patterns Exemplars Templates 17 Libraries of Assessment Objects

18. The PADI Solution • PADI focus is on ... • Design framework (Design patterns, templates, delivery system) • Assessing inquiry in science • PADI focus is not on ... • Tasks, per se (although will be examples, through FOSS, BioKIDS, other exemplars) • Authoring systems • Delivery systems • (exceptions: scoring and calibration engine, GradeBook)