Unraveling the Mysteries of Setting Standards and Scaled Scores

2. Unraveling the Mysteries of Setting Standards and Scaled Scores • Julie Miles PhD, 10.27.2011

3. Overview of the Session • 1. What is Standard Setting? • Basic Vocabulary • Definition • Performance Level Descriptions • Threshold Descriptions • When Does It Occur? • Methods Used in Virginia • 2. The Connection to Scaled Scores • Converting Raw Scores to Scaled Scores • Example Conversion • 3. From Scaled Scores to Equated Forms • How Are Scaled Scores Connected to Equating? • The Basics of Equating • Recap of How It All Comes Together Presentation Title runs here l 00/00/00

4. 1 What Is Standard Setting? Presentation Title runs here l 00/00/00

5. What is Standard Setting?Basic Vocabulary • Content Standards: the content and skills that students are expected to know and be able to do. • Performance Levels (Achievement Levels, Performance Categories): Labels for levels of student achievement (e.g., below basic, basic, proficient and advanced). • Performance Level Descriptors (PLDs): Descriptions of the competencies associated with each level of achievement. • Cut Scores (Performance Standards): Scores on an assessment that separate one level of achievement from another. Presentation Title runs here l 00/00/00

6. What is Standard Setting?Definition • A judgmental process which has a variety of steps and includes relevant stakeholders throughout. Steps in this process typically include: • Identifying the relevant knowledge and skills to be taught and assessed at each grade/content area to support the goals of the state • Defining the expectations associated with each Performance Level • Convening a committee of educators to provide content-based recommendations for cut scores at each grade or subject area • Review of cut score recommendations and adoption by the State Board of Education Presentation Title runs here l 00/00/00

7. What is Standard Setting?Performance Level Descriptors (PLDs) • Define the knowledge, skills, and abilities (KSAs) that are expected of the students to gain entry into specific performance levels (e.g., Proficient or Advanced) • The main goal of standard setting is to quantify or operationalize the Performance Level Descriptors. Presentation Title runs here l 00/00/00

8. What is Standard Setting?Threshold Descriptions (TDs) • Define what students who are “just over the threshold” in a performance level (e.g., a student scoring a 400 or 401 or 500 or 501) should be able to demonstrate in terms of KSAs. • These are the borderline or minimally qualified students in terms of performance Presentation Title runs here l 00/00/00

9. What is Standard Setting?When Does It Occur? Presentation Title runs here l 00/00/00

10. What is Standard Setting?Methods Used in Virginia • Virginia predominantly uses “Modified Angoff” (SOL and VMAST),“Body of Work” (VAAP), and “Reasoned Judgment” (VGLA) methods. All methods typically have similar components: • Overview of standard setting • Review of test blueprint and performance level descriptions • Creation of the threshold descriptions • Overview of actual test administered to students • Three rounds of judgments by committee: • MC Tests: should a ‘just-barely’ student get the item correct 2 out of 3 times? • VGLA: how many points should a ‘just-barely’ student earn on this SOL? • VAAP: which performance level does a COE represents? • Final round results in cut score recommendations that are provided to the SBOE. • The number of correct answers needed to gain entry into each performance level. Presentation Title runs here l 00/00/00

11. 2 The Connection to Scaled Scores Presentation Title runs here l 00/00/00

12. The Connection to Scaled ScoresConverting Raw Scores to Scaled Scores • The recommendations for a cut score from standard setting are in a raw score metric. But this is not helpful from year-to-year. • Student ability is different from student to student • Test forms change from year-to-year (and within year) • A raw score of 36 on a slightly easier test does not indicate the same level of achievement as a raw score of 36 on a slightly more difficult test. Need a metric that is stable from year-to-year! • This is where I earn my keep  • The metric is based on item response theory (IRT) and it is called “theta.” This theta value (associated with raw score) is converted to a scaled score that remains stable from year-to-year so that 400 is comparable to 400 regardless of the student, year, or form. Presentation Title runs here l 00/00/00

13. The Connection to Scaled ScoresExample Conversion to Scaled Scores • Algebra II • where θa is the value of theta (2.616) corresponding to the raw score (45) at the pass/advanced level and θp is the value of theta (.6416) corresponding to the raw score (30) at the pass/proficient level. • Solving for a yields: • And substituting the values of theta corresponding to the raw score cuts gives: • Solving for b yields: • And substituting the values of θp and a gives Presentation Title runs here l 00/00/00

14. 3 From Scaled Scores to Equated Forms Presentation Title runs here l 00/00/00

15. From Scaled Scores to Equated FormsHow are Scaled Scores Connected to Equating? • When a test is built, the item difficulties (in the Rasch metric) are known from the field test statistical analyses. • The tests are built to Rasch difficulty targets for the overall test and all reporting categories based on the standard setting form. • Even though an attempt is made to construct test forms of equal Rasch-based difficulty from form to form and year to year, there will be small variations in difficulty. • When building tests, the IRT model makes it possible to estimate the raw score that corresponds to a scale score of 400. • Each core form of a test is equated to the established scale so that the scores indicate the same level of achievement regardless of the core form taken. Presentation Title runs here l 00/00/00

16. From Scaled Scores to Equated FormsThe Basics of Equating • Common-Item Nonequivalent Groups Design The common-item set is constructed as a “mini version” of the total test. Presentation Title runs here l 00/00/00

17. From Scaled Scores to Equated FormsThe Basics of Equating Presentation Title runs here l 00/00/00

18. Recap of How It All Comes Together  Scores Test is Equated Test is Scaled Cut Scores are adopted by SBOE Cut Scoresare Recommended Test Is Developed Presentation Title runs here l 00/00/00

19. Questions?julie.miles@pearson.com