The Critical Role of Design Patterns in Large-Scale Assessment

1. The Critical Role of Design Patterns in Large-Scale Assessment Robert J. Mislevy & Min Liu University of Maryland Geneva Haertel SRI International DR K-12 grant #0733172, “Application of Evidence-Centered Design to State Large-Scale Science Assessment.” This material is based upon work supported by the National Science Foundation under Grant No. DRL- 0733172. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

2. Three design challenges Hard-to-assess standards in the domain of interest e.g., inquiry science skills Efficient and valid design and development of complex tasks e.g., scenarios, simulations Accessibility of tasks for diverse learners varying perceptual and expressive capabilities valid use of assistive technology, modifications, alternative assessments

3. Design Patterns in Architecture Design Patterns in Software Engineering Design Patterns in Literature Design Patterns

4. Motivation for Assessment Design Patterns • In-between structure, to connect... • Thinking about science learning & inquiry • Technical elements of measurement & delivery • Narrative, not technical, contents • Some Design Patterns from PADI • Model-Based Reasoning • Model Formation; Evaluation; Model Revision; Use • Observational & Experimental Investigations • Systems Thinking

5. Motivation for Assessment Design Patterns • They lay out a design space for developers • Choices, connections, examples • Things to be aware of (e.g., research on Universal Design for Learning) • Can improve both Efficiency + Validity • Attributes reflect assessment argument structure

6. Assessment Arguments What complex of knowledge, skills, or other attributes should be assessed? What behaviors or performances should reveal those constructs? What tasks or situations should elicit those behaviors? Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13-23. Student Model Evidence Model Task Model Mislevy, R.J., & Haertel, G. (2006). Implications for evidence-centered design for educational assessment. Educational Measurement: Issues and Practice, 25, 6-20.

7. The Structure of Assessment Design Patterns Student Model Evidence Model Task Model

8. How Design Patterns Support Thinking about the Assessment Argument Student Model The Rationale provides background into the nature of the Focal KSAs, and the kinds of things that people do in what kinds of situations that evidence it. E.g., overview, research links, examples. Evidence Model Task Model

9. How Design Patterns Support Thinking about the Assessment Argument Student Model The design pattern is organized around Focal KSAs. They will be involved in the Student Model, although there may be other KSAs that are included in the target of inference (e.g., Model Revision—but what models, what context?). Evidence Model Task Model Associated with Characteristic Features of Tasks.

10. How Design Patterns Support Thinking about the Assessment Argument Student Model Evidence Model Additional KSAs play multiple roles. You need to think about which ones you really DO want to include as targets of inference (validity) and which ones you really DON’T (invalidity). Task Model

11. How Design Patterns Support Thinking about the Assessment Argument The Additional KSAs you DO want to include as targets of inference are part of the claim. E.g., knowing Mendel’s laws as well as being able to formulate a model in an investigation. Student Model Evidence Model Task Model Connected with Variable Features of Tasks.

12. How Design Patterns Support Thinking about the Assessment Argument Student Model Evidence Model The Additional KSAs you DON’T want to include as targets of inference introduce construct-irrelevant reasons for poor performance. (Especially important for assessing special populations – UDL & accommodations.) Task Model Connected with Variable Features of Tasks & Work Products.

13. How Design Patterns Support Thinking about the Assessment Argument Student Model Evidence Model The Characteristic Features of Tasks help you think about critical features of the tasks situation you need to get evidence about the Focal KSAs. Task Model

14. How Design Patterns Support Thinking about the Assessment Argument Variable Features of Tasks also help you think about data concerning the situation – but now to influence difficulty … Student Model or to bring in or reduce demand for Additional KSAs to avoid alternative explanations. Evidence Model Task Model

15. How Design Patterns Support Thinking about the Assessment Argument Some Variable Features of Tasks help you match features of tasks and background / knowledge / characteristics of students: Interests, familiarity, previous instruction. Student Model Evidence Model Task Model

16. How Design Patterns Support Thinking about the Assessment Argument Potential Work Products help you think about what you want to capture from a performance –product, process, constructed model, written explanation, etc. Student Model Evidence Model Task Model Can also call attention to demand for Additional KSAs, & avoid alternative explanations for poor performance

17. How Design Patterns Support Thinking about the Assessment Argument Potential Observations are possibilities for the qualities of Work Products – i.e., the data concerning the performance. Student Model Evidence Model Task Model

18. How Design Patterns Support Thinking about the Assessment Argument Student Model Evidence Model And Potential Rubrics are algorithms/rubrics/rules for evaluating Work Products to interpret evidence from the student’s performance. Task Model

19. Current Catalog of Design Patterns • ECD/PADI related projects have produced over 100 Design Patterns • Domains include science inquiry, science content, mathematics, economics, model-based reasoning • Span grades 3-16+ • Organized around themes, models, and processes, not surface features or formats of tasks • Support the design of scenario-based, multiple choice, and performance tasks

20. Current Catalog of Design Patterns

21. For more information… PADI: Principled Assessment Design for Inquiry http://padi.sri.com Links to NSF & IES follow-on projects Lots of Tech Reports, interactive online examples Bob Mislevy home page http://www.education.umd.edu/EDMS/mislevy/ Links to papers on ECD Cisco applications

22. Now for the Good Stuff … • Examples of design patterns with content • Different projects • Different grain sizes • Different users • How they are being used to tackle pervasive challenges of large-scale assessment. • How they evolved to suit needs of users • Same essential structure, but • Representations, language, emphases, and affordances tuned to users and needs