Frameworks for Considering Item Response Demands and Item Difficulty Kristen Huff College Board

1. NCSA Session: Theory and Research on Item Response Demands: What Makes Items Difficult? Construct-Relevant?June 20, 2010 Detroit Frameworks for Considering Item Response Demands and Item Difficulty Kristen Huff College Board Steve Ferrara CTB/McGraw-Hill

2. Thesis A coherent and comprehensive understanding of the interaction between items and examinees, and the controllable features of items that elicit predictable interactions is needed to: • Design items that do a better job of measuring what we’re interested in knowing • Design tests that are better suited to facilitate valid inferences about student performance • More generally, bridge the gap between large-scale assessment and teaching & learning

4. Objective To conduct research (& review past research) that informs the development of a framework, or conceptual structure, of item response demands. Research questions: How much do existing item response demand schemas cover current achievement constructs? What are the features of items that influence item difficulty? What student responses are triggered by different item features, and how do these responses influence item difficulty?

5. feature feature feature feature

6. Reading comprehension # words reading level overlap between key and text Math number of variables graphics fractions vs whole numbers feature feature feature feature

9. Framework

11. Framework

14. Conclusions & Points of Discussion • Mapping the landscape of response demands, item features and the interaction between the two is messy, difficult work • Achieving “coherent and comprehensive” frameworks needs to be higher priority in research • Start now using draft in operational testing programs • Need more effective and easier ways to gauge opportunity to learn for many reasons, but also to inform the work we are recommending