On Layers and Objects in Assessment Design

1. On Layers and Objectsin Assessment Design Robert Mislevy, University of Maryland Michelle Riconscente, University of Maryland

2. Evidence-Centered Design • Conceptual design framework • Broad applicability • Based on • Evidentiary reasoning principles • Demands of assessment production & delivery • Assures • Validity of assessment • Coordination among collaborators (Mislevy, Almond, & Lukas, 2003)

3. Defining Assessment • Reasoning from the particular things students do, say, or make… • to inferences about their knowledge, skills & abilities.

4. Some Fundamentals • Evidentiary Reasoning • Assessment as Argument

5. Evidentiary Reasoning C unless A since W on account of supports so R B D Toulmin (1958)

6. Example I C unless A since W on account of supports so R B D Toulmin (1958)

7. Example II C John caused the car accident and Marg was seriously injured. W1: Since an intoxicated driver is generally presumed to be at fault in an accident…. W2: Since the impact of a 2000lb. auto moving at 50mph on a human will generally cause serious injury… unless A Marg was also intoxicated… since W on account of supports so R B D • John’s BAC was .13. • Marg was unconscious and bleeding. • John’s speech was slurred. • Marg had the right of way. B1: State Code: drunk driving at .10 BAC and common law doctrine of negligence per se. B2: Emergency medical records at Wishard Hospital indicate this type of collision will result in serious condition. Toulmin (1958)

8. Assessment as Argument Inferences C • Inferences • Observations needed to ground them • Situations that will evoke them • Chain of reasoning connecting them since since W Observations D Situations

9. Some Fundamentals • Evidentiary Reasoning • Assessment as Argument • Knowledge Representations (KRs) • Common Language

10. Knowledge Representations • Directions from Campus to the White House • http://www.mapquest.com/directions/main.adp?go=1&do=nw&un=m&2tabval=address&cl=EN&ct=NA&1tabval=address&1y=US&1a=&1c=&1s=&1z=&1ah=NXXzjLrv%252bgGeW7JeQ4fDrUNya%252b4DtKsjxheMTbVThQJ3lE9WrNx7DAJWKfgZBsxDNU8KMBJ3MT8wsLFk85Pi6jLRqQHxBFYMlklq5k82hIODOViBiLYDxDz3sk%252bXxGlUcWB2Xec5x21sAggO4xXFyg%253d%253d&2y=US&2a=1600+Pennsylvania+Avenue&2c=Washington&2s=DC&2z=&2ah=&idx=0&id=417d7541-00047-01632-400c2551&aid=417d7541-00048-01632-400c2551 • OR ?

11. Some Fundamentals • Evidentiary Reasoning • Assessment as Argument • Knowledge Representations (KRs) • Common Language • Layers • Leveraging Varied Expertise • Common Structures

12. A Layered Approach • Leveraging Varied Expertise • e.g., Housing Developments • Common Structures • e.g., Architecture, Software Design • Iterate through Assessment Argument • narrative  technical specifications Inferences Observations Situations

13. ECD Layers Domain Analysis • What is important about this domain? • What tasks are central to proficiency in this domain? • What KRs are central to this domain? Domain Modeling • Represent key aspects of the domain in terms of assessment argument. • What tools and materials do we need to implement this kind of assessment? Conceptual Assessment Framework Assessment Delivery • How do we move to implementation with actual examinees?

14. Domain Analysis Domain Analysis • Valued work • Task features • Representational forms • Performance outcomes • Valued knowledge • Knowledge structure and relationships • Knowledge-task relationships

15. ECD Layers Domain Analysis Domain Modeling Conceptual Assessment Framework Assessment Delivery

16. Domain Modeling • Express the content of the domain analysis in terms of the assessment argument. • Takes narrative form • PADI KR: Design Patterns Domain Modeling Inferences Observations Situations

17. Design Patterns • Identify in narrative form: • Knowledge, skills, and abilities (KSAs) • Observations to support inference • Features of task situations that elicit target KSAs • Related content or inquiry standards • Do not provide a concrete design or implementation of an assessment task

19. ECD Layers Domain Analysis Domain Modeling Conceptual Assessment Framework Assessment Delivery

20. Conceptual Assessment Framework • Move from narrative to more technical specifications. • Express assessment argument in terms closer to implementation  ECD Models Conceptual Assessment Framework

21. From Argument to Models STUDENT MODEL Inferences EVIDENCE MODEL Observations Situations TASK MODEL (Mislevy, Almond, & Lukas, 2003)

22. From Argument to Models STUDENT MODEL EVIDENCE MODEL TASK MODEL What are we measuring? How do we measure it? • How do we measure it? • Quality • Relevance Where do we measure it? (Mislevy, Almond, & Lukas, 2003)

23. Conceptual Assessment Framework • Move from narrative to more technical specifications. • Express assessment argument in terms closer to implementation  ECD Models • Create pre-blueprintsfrom which we will eventually generate blueprints and operational assessments. • PADIKR: Task Templates

24. Task Templates • Support the specification of technical details • Link CAF components • Serve as pre-blueprints: abstractions of multiple assessment tasks • Become task specifications when all template components are specified

27. Summing Up • ECD: assessment design as development of an assessment argument • Coordinate work across expertise • Use of layers • Knowledge Representations • Illustrated with PADI work • In addition: • Explication of reasoning behind assessment design decisions • Identification of reusable elements and pieces of infrastructure

28. An ECD “Flight Check” • Conceptual design framework • Broad applicability • Based on • Evidentiary reasoning principles • Demands of assessment production & delivery • Assures • Validity of assessment • Coordination among collaborators (Mislevy, Almond, & Lukas, 2003)