The Application of ECD to the Redesign of Advanced Placement Exams

1. The Application of ECD to the Redesign of Advanced Placement Exams CCSSO June 2012 Maureen Ewing, Kristen Huff, Amy Hendrickson, Pamela Kaliski

2. Why ECD for AP? Need targets of measurement and curricular requirements No guesswork on part of the test user, test maker or test taker about what is valued in the domain Lack of ambiguity and use of task models helps ensure stable measurement Enhances validity argument

3. Three Critical ECD Phases

4. Experts with Disciplinary Knowledge Unpacked the Content omains • Big ideas(level 1) • Enduring understandings (level 2) • Supporting understandings(level 3) 1 2 2 2 3

5. Experts with Disciplinary Knowledge Unpacked the Domains • Identify Skills or reasoning processes • Articulate definition of skill (i.e., observable evidence) 1 2 2 2 3

6. Inputs to Domain Analysis • National and state standards • Latest research on student learning and assessment • Curriculum Surveys • Other relevant documents • Expert judgment of panelists

7. Three Critical (re)Design Phases

8. Writing Claims and Evidence • Claims start with the phrase “The student can” to reinforce the focus on what the student knows and can do • Claims are comprised of content and/or skills that were identified in the domain analysis • Evidence starts with the phrase “the work is characterized by…” • Evidence includes nouns and adjectives (no verbs)

9. If it’s unobservable, it isn’t ‘evidence’ Examples of non-evidence: “a deep awareness of the importance of theorems” “understanding of differences and similarities between derivativesand integrals” What would a student have to do, say, or write in order to prove to you that he/she has a deep awareness of the importance of theorems? The answer to this question is the evidence. As evidence is written, necessary edits to the claims may be uncovered; iteration is key!

10. Achievement Level Descriptions – ALDs • Describe what students in each performance category should reasonably know and be able to do • Relate directly to the knowledge, skills, and abilities a student should know and be able to demonstrate to be classified into a particular score category • Distinguish clearly from one performance category (e.g., AP Grade 2) to the next (e.g., AP Grade 3)

12. Three Critical (re)Design Phases

13. Achievement Level 3

14. Task Models: Take-home messages • Claims and evidence are “fodder” for item • Stimuli • Prompt • Response options • Rubrics • Many “like” items can be produced from same claim / evidence pair Construct relevance as the target of measurement is the claim/evidence Item generation; Comparability

15. Form Assembly Specifications - Considerations • Multiple inputs • Domain Model • Experts’ ratings of importance of content and skills • Psychometric criteria • Structure of the domain • Claims: skills-based versus integration of skills and content • Content relationships • Skill relationships • Content and skill relationships

16. Challenges of ECD • Iterative nature of work • Sometimes viewed as “doing it wrong” the first time, an annoyance, but it IS the process • Resource and time intensive (6 years and counting), especially development of task models based on iterative pilots • Steep learning curve – difficult to think in terms of observable evidence • Defining appropriate grain size for each piece

17. Lessons Learned and Ongoing Work • We need to develop an evaluation process that specifically addresses the iterations between C&E, ALDs, etc • Documents, captures, thinks about the iteration - this is the real work of figuring out the construct about how students learn as well as ECD • We need to figure out the grain size issues and how everything fits together