350 likes | 521 Views
New England Common Assessment Program (NECAP). Setting Performance Standards. Purpose. Provide data to establish the following three cut scores for Reading, Grades 3-8 and Writing, Grades 5 and 8: Proficient/Proficient with Distinction Partially Proficient/Proficient
E N D
New EnglandCommon Assessment Program(NECAP) Setting Performance Standards
Purpose • Provide data to establish the following three cut scores for Reading, Grades 3-8 and Writing, Grades 5 and 8: • Proficient/Proficient with Distinction • Partially Proficient/Proficient • Substantially Below Proficient/Partially Proficient
What is Standard Setting? • Set of activities that result in the determination of threshold or cut scores on an assessment • We are trying to answer the question: • How much is enough?
What is Standard Setting • Data collection phase • Policy/Decision making phase
Many Standard Setting Methods • Angoff • Body of Work • Bookmark
Choice of Method is Based on Many Factors • Prior usage/history • Recommendation/requirement by some policy making authority • Type of assessment
Choice of Method is Based on Many Factors • Weighing all these factors, it was determined that the methods to be used are: • Reading: Bookmark • Writing: Body of Work • They are both well-established procedures that have been successfully used on many assessments • They have produced defensible results
Choice of Method is Based on Many Factors • Bookmark is appropriate for assessments that consist primarily of multiple-choice items but also include some constructed-response items • Body of Work method works well for assessments that consist primarily of constructed-response items
Standard Setting vs. Standards Validation • Standard setting • Generally three rounds: • Round 1: individual ratings • Round 2: table group discussion of Round 1 results, followed by second round ratings • Round 3: whole room discussion of Round 2 results, followed by final ratings
Standard Setting vs. Standards Validation • Standards validation • Starting cut points are provided to panelists at the beginning of the process • Two rounds: • Round 1: table group discussion of starting cut points, followed by first round ratings • Round 2: whole group discussion of Round 1 results, followed by second round ratings • The process you will be following will be closer to a standards validation process
Details for Standards Validation using the Bookmark and Body of Work Procedures
Starting Cut Points • Starting cuts for Reading and Writing were determined using teacher judgment data: • At the time of testing, teachers were asked to categorize each student into an achievement level category, based on classroom performance (not test performance) • Using these categorizations as well as the students’ test scores, Measured Progress staff calculated starting cut points
Starting Cut Points • Analyses of teacher judgment data indicate: • Good participation rates • Teachers appear to have been conscientious in completing the task • In other words, available evidence supports the validity of the starting cuts
Starting Cut Points • However, standard setting is also a critical part of the process because: • the teachers based their judgments on classroom performance, not performance on NECAP • different teachers have different conceptions of what it means to be, for example, Partially Proficient • they did not have the benefit of discussion with their colleagues from across the three states • etc.
Starting Cut Points • Both sources of information are important. Therefore: • You are free to recommend changing the starting cuts, but we don’t expect to see major changes • The changes you recommend should be based on the test content and the Achievement Level Descriptions
The Bookmark Procedure(Reading) • The Bookmark procedure is a standard setting method that uses a book of items (ordered from easiest to hardest) • Panelists place bookmarks in that book of items
What is the bookmark procedure? • For purposes of NECAP standard setting, you will be provided with starting cut points and will either validate those cuts, or recommend modifications to them
A Technical Detail regarding the Bookmark Method • Note that the ordered item cut point for a given cut does not equal the raw score a student must obtain to be categorized into the higher achievement level • For example, the Substantially BelowProficient/ Partially Proficient starting cut for grade 3 reading is between ordered items 6 and 7; however, a student must obtain at least 20 points on the test in order to be classified into the Partially Proficient level
How to Place a Bookmark • A few concepts you will need to know: • The starting cut points • The achievement level descriptions • ‘Borderline’ students • What knowledge, skills, and abilities (KSAs) are needed to answer each question
How to Place a Bookmark • Start at the beginning of the ordered item book • Evaluate whether at least 2 out of 3 students demonstrating skills at the ‘borderline’ of Partially Proficient would correctly answer item 1 • Moving through the book, make this evaluation of each item • The bookmark should go where you no longer think 2 out of 3 Partially Proficient ‘borderline’ students would correctly answer the question. • You may decide that the starting cuts are in the right place, or you might recommend moving them.
How to Place a Bookmark • In the example, the bookmark would go between items 8 and 9 • However, it won’t be that easy; there will be gray areas • Place one bookmark for each cut score • You will have the opportunity to discuss your bookmark placements and change them if desired
How to Place a Bookmark • To place your bookmarks you will need to be familiar with the achievement level descriptions and the assessment items
How to Place a Bookmark • Don’t worry, we have procedures, materials and staff to assist you in this process.
The Body of Work Procedure (Writing) • A standard method that uses approximately 30 complete sets of student work, including their answers to open-ended items and a display of their responses to the multiple-choice items • Bodies of work cover the range of possible scores and are presented in order from lowest to highest total score
What is the Body of Work procedure? • You will classify each BOW into the achievement level in which you feel it belongs: • Proficient with Distinction • Proficient • Partially Proficient • Substantially Below Proficient
The Body of Work Procedure • Prior to beginning the process of rating the BOWs, you will: • take the assessment • become very familiar with the Achievement Level Descriptions • define, with your grade-level group, ‘borderline’ students • individually review the full set of BOWs
Making Your Ratings • Beginning with the first BOW, and referring to the Achievement Level Descriptions and the definition of ‘borderline’ students, determine into which achievement level the BOW should be categorized • Continue through each BOW in turn, assigning each to an achievement level • You may agree with the categorizations based on the starting cuts, or you may recommend changing some of them
What Next? • After this session, you will break into grade-level groups, where you will: • take the assessment to familiarize yourself with the test items • Reading groups: complete the item map, which is a document that will help you with the bookmark placement process • discuss the Achievement Level Descriptions and develop definitions of “borderline” Partially Proficient, Proficient, and Proficient with Distinction students
What Next? • You will: • discuss the starting cut points in table groups and do the first round of ratings • discuss the first round ratings as a whole room then do the second round of ratings
What Next? • After the second round of ratings, you will have an opportunity to provide feedback about the Achievement Level Descriptions • As the final step, we will ask you to complete an evaluation of the standard setting process