Download
1 / 19
190 likes | 293 Views
Unraveling the Mysteries of Setting Standards and Scaled Scores. Julie Miles PhD, 10.31.2013. Overview of the Session. 1. What is Standard Setting? Basic Vocabulary Definition Performance Level Descriptions Threshold Descriptions When Does It Occur? Methods Used in Virginia
E N D
Unraveling the Mysteries of Setting Standards and Scaled Scores • Julie Miles PhD, 10.31.2013
Overview of the Session • 1. What is Standard Setting? • Basic Vocabulary • Definition • Performance Level Descriptions • Threshold Descriptions • When Does It Occur? • Methods Used in Virginia • 2. The Connection to Scaled Scores • Converting Raw Scores to Scaled Scores • Example Conversion • 3. From Scaled Scores to Equated Forms • How Are Scaled Scores Connected to Equating? • The Basics of Equating • Recap of How It All Comes Together Unravelling the Mysteries...
1 What Is Standard Setting? Unravelling the Mysteries...
What is Standard Setting?Basic Vocabulary • Content Standards: the content and skills that students are expected to know and be able to do. • Performance Levels (Achievement Levels, Performance Categories): Labels for levels of student achievement (e.g., below basic, basic, proficient and advanced). • Performance Level Descriptors (PLDs): Descriptions of the competencies associated with each level of achievement. • Cut Scores (Performance Standards): Scores on an assessment that separate one level of achievement from another. Unravelling the Mysteries...
What is Standard Setting?Definition • A judgmental process which has a variety of steps and includes relevant stakeholders throughout. Steps in this process typically include: • Identifying the relevant knowledge and skills to be taught and assessed at each grade/content area to support the goals of the state • Defining the expectations associated with each Performance Level • Convening a committee of educators to provide content-based recommendations for cut scores at each grade or subject area • Review of cut score recommendations and adoption by the State Board of Education Unravelling the Mysteries...
What is Standard Setting?Performance Level Descriptors (PLDs) • Define the knowledge, skills, and abilities (KSAs) that are expected of the students to gain entry into specific performance levels (e.g., Proficient or Advanced) • The main goal of standard setting is to quantify or operationalize the Performance Level Descriptors. Unravelling the Mysteries...
What is Standard Setting?Threshold Descriptions (TDs) • Define what students who are “just over the threshold” in a performance level (e.g., a student scoring a 400 or 401 or 500 or 501) should be able to demonstrate in terms of KSAs. • These are the borderline or minimally qualified students in terms of performance Unravelling the Mysteries...
What is Standard Setting?When Does It Occur? Unravelling the Mysteries...
What is Standard Setting?Methods Used in Virginia • Virginia predominantly uses “Modified Angoff” (SOL and VMAST),“Body of Work” (VAAP), and “Reasoned Judgment” (VGLA) methods. All methods typically have similar components: • Overview of standard setting • Review of test blueprint and performance level descriptions • Creation of the threshold descriptions • Overview of actual test administered to students • Three rounds of judgments by committee: • MC Tests: should a ‘just-barely’ student get the item correct 2 out of 3 times? • VGLA: how many points should a ‘just-barely’ student earn on this SOL? • VAAP: which performance level does a COE represents? • Final round results in cut score recommendations that are provided to the SBOE. • The number of correct answers needed to gain entry into each performance level. Unravelling the Mysteries...
2 The Connection to Scaled Scores Unravelling the Mysteries...
The Connection to Scaled ScoresConverting Raw Scores to Scaled Scores • The recommendations for a cut score from standard setting are in a raw score metric. But this is not helpful from year-to-year. • Student ability is different from student to student • Test forms change from year-to-year (and within year) • A raw score of 36 on a slightly easier test does not indicate the same level of achievement as a raw score of 36 on a slightly more difficult test. Need a metric that is stable from year-to-year! • This is where my team earns their keep • The metric is based on item response theory (IRT) and it is called “theta.” This theta value (associated with raw score) is converted to a scaled score that remains stable from year-to-year so that 400 is comparable to 400 regardless of the student, year, or form. Unravelling the Mysteries...
The Connection to Scaled ScoresExample Conversion to Scaled Scores • Algebra II • where θa is the value of theta (2.616) corresponding to the raw score (45) at the pass/advanced level and θp is the value of theta (.6416) corresponding to the raw score (30) at the pass/proficient level. • Solving for a yields: • And substituting the values of theta corresponding to the raw score cuts gives: • Solving for b yields: • And substituting the values of θp and a gives Unravelling the Mysteries...
3 From Scaled Scores to Equated Forms Unravelling the Mysteries...
From Scaled Scores to Equated FormsHow are Scaled Scores Connected to Equating? • When a test is built, the item difficulties (in the Rasch metric) are known from the field test statistical analyses. • The tests are built to Rasch difficulty targets for the overall test and all reporting categories based on the standard setting form. • Even though an attempt is made to construct test forms of equal Rasch-based difficulty from form to form and year to year, there will be small variations in difficulty. • When building tests, the IRT model makes it possible to estimate the raw score that corresponds to a scale score of 400. • Each core form of a test is equated to the established scale so that the scores indicate the same level of achievement regardless of the core form taken. Unravelling the Mysteries...
From Scaled Scores to Equated FormsThe Basics of Equating • Common-Item Nonequivalent Groups Design The common-item set is constructed as a “mini version” of the total test. Unravelling the Mysteries...
From Scaled Scores to Equated FormsThe Basics of Equating Unravelling the Mysteries...
Recap of How It All Comes Together Scores Test is Equated Test is Scaled Cut Scores are adopted by SBOE Cut Scoresare Recommended Test Is Developed Unravelling the Mysteries...
