Using Construct Maps to Improve Standard Setting Processes

1. Using Construct Maps to Improve Standard Setting Processes Adam E. Wyse Michigan Department of Education Bureau of Educational Assessment and Accountability Measurement Research and Psychometrics

2. Session Goals • Provide a conceptual framework to view common standard-setting methods based on construct maps • Explain what a construct maps are and how they can used to improve standard setting processes. • Discuss how construct maps have been used in several standard setting applications and how they might be used in the future. • Outline the policy implications of using construct maps and how they might be used to improve score reporting from a state perspective.

3. What is a construct map? • A construct map is a graphical or tabular display that shows the relationship between an underlying achievement construct and other examinee and item data from that assessment. • Three basic requirements: • A sound definition of the content underlying the assessment • The construct can be represented by an underlying continuum and examinees and items can be ordered along that continuum • A measurement model can be used to show the relationships between the achievement continuum and the examinee or item data

4. Theoretical Construct Map

5. Empirical Construct Map

6. Construct Maps and their relationship to Common Criticisms of Standard Setting Transparency Rater Inconsistency Score/Item Gaps Range Restriction

7. Uses of Construct Maps in Standard Setting Set cut scores and provide feedback information Tools to evaluate standard setting and collect validity evidence Synthesize data and decide on final cut scores

8. Contact Me • WyseA@michigan.gov