Setting Performance Standards Grades 5-7 NJ ASK

1. Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006

2. What do we mean by “Standard Setting”? • Content standards: specify the content that should be learned by students – given implementation of the state standards and an opportunity to learn. • Performance standards: specify the precise level of performance required on an assessment to achieve some category or outcome. How much must students know and be able to do?

3. Proficiency Levels on NJ ASK 100 - 199 250 - 300 200 - 249 Advanced Partially Proficient Proficient Proficient Proficient Advanced Cut Score Cut Score

4. NJ ASK Grades 5-7 Standard Setting • Phase 1 • 20 panelists on the Math Grades 5-7 & LAL Grade 7 panels • 19 panelists on the LAL Grades 5-6 panels • Used a Modified Angoff procedure to calculate cut scores in a 2-day workshop • Phase 2 • Representatives from Phase 1 and state and local policy makers reviewed results • Examined consistency across grades and subjects

5. The Angoff Procedure • Research-based procedure used since the early 1970s • The most commonly used standard setting method — used in many other state testing programs as well as on certification tests • Has undergone many modifications over the years and is often referred to as the Modified Angoff or Extended Angoff procedure

6. Angoff Procedure (con’t.) • Original Angoff procedure asked panelists to examine each item on a test and determine whether a student who was just barely Proficient would answer it correctly • A common modification is to ask “If we had 100 barely Proficient students in the room, how many of them would answer this item correctly?”

7. The Angoff Task • Review each item • For MC items, determine how many out of 100 just barely Proficient and just barely Advanced Proficient students should answer each item correctly • For OE items, determine the average score a just barely student at each performance level should receive

8. Validity Considerations • The process was well-planned – see standard setting proposal. • All policy, operational and technical decisions were reasonable and well-documented. • The judges were satisfied with the process and its results – see the evaluation results.

9. The Final Decision on Cut Scores – Reminder • Legitimate authority should have all the information they want or need to make an informed decision • At a minimum it should include: • Overall results – Phase 1 and Phase 2 • Variance in judgments and in measurement • Impact on subpopulations in the state • Impact on accountability • Documentation of the process (with as much detail as is necessary)

10. Ratings • Cut scores are determined over three rounds of ratings • Ratings were made independently • After each set of ratings, panelists saw their cut scores and the average cut scores • There was a discussion between each round

11. Calculating the Cut Score • Consider the MC ratings as probabilities • We sum the probabilities to obtain a cut score for the MC items and add the average scores for the OE items to get the total cut score Item Rating Possible 1 .85 1 2 .55 1 3 .40 1 4 1.5 3 Total= .85 + .55 + .40 + 1.5 = 3.3 points out of a possible 6 points

12. “Just Barely” Performance • The idea of just barely performance is key to the Angoff procedure • We spent time talking about what it means to be just barely Proficient and Advanced Proficient • Our discussions were driven by the NJ ASK LAL and Math Grades 5-7 Performance Level Descriptors (PLDs)

13. Phase 2 Meeting • Policymakers and stakeholders gathered to review the results of Phase 1 in light of NJ ASK program goals. • Reviewed results from Phase 1 along with standard deviation of judgments and impact data. • Compared results of Grades 3, 4 & 8 LAL and Math with Grades 5-7 LAL and Math and discussed the need for consistency across grades and content areas.

14. Phase 2 Results: Final Cut Score Recommendations • Committee produced cut score recommendations for NJDOE and the State Board:

15. Evidence the Results Were Technically Sound • Survey Results (see handout) • Standard deviation of judgments in each round:

16. Distribution of Students by SubgroupLAL Grade 5

17. Distribution of Students by SubgroupLAL Grade 6

18. Distribution of Students by SubgroupLAL Grade 7

19. Distribution of Students by SubgroupMath Grade 5

20. Distribution of Students by SubgroupMath Grade 6

21. Distribution of Students by SubgroupMath Grade 7

22. Final Recommended CutscoresLAL Grades 5-7 (with LAL Grades 3,4 &8)

23. Final Recommended CutscoresMath Grades 5-7 (with Math Grades 3,4 &8)

24. Key Statistics from Surveys