What (the Heck) Does “Proficient” Mean? Standard Setting on the MCAS Alternate Assessment

1. What (the Heck) Does “Proficient” Mean? Standard Setting on the MCAS Alternate Assessment Daniel J. Wiener Massachusetts Dept. of Education CCSSO - San Antonio, TX June 2003

2. Standard Setting: Important Decisions • Naming and defining “Performance Levels” • Which raw scores count for PL? • How will raw scores be combined to get a PL? • What combination of scores yields which PL?

3. MCAS-Alt Performance Levels:Describe how well student has learned the General Curriculum? • Awareness: Very little, if at all • Emerging: Just getting the hang of it • Progressing : Moving right along on challenging standards, but below grade level • Needs Improvement: Doing okay at grade level • Proficient: Doing well at grade level • Advanced: Excelling at grade level

4. Which Scores Count Toward the Performance Level? Based on stakeholders’ advice, “Parallel the General Assessment” Therefore, Rubric scores in: • Complexity, Accuracy, and Independence are combined to determine an overall Performance Level; • Not Self-Evaluation, Generalized Performance

5. Two Possible Ways to Combine Rubric Scores EITHER: 1) Calculate a total numerical score, based on individual rubric scores; then, set cut score for each Performance Level Example: Complexity X (Accuracy + Independence) = Total score

6. Two Possible Ways to Combine Rubric Scores (continued...) OR: 2) Describe the characteristics of student performance for all score combinations (no total numerical score) Example: What does a score of LC=3, DSC=4, Ind=3 indicate about a student’s performance?

7. Student Performance: A Balancing Act Complexity Accuracy Independence LOW HIGH

8. Performance Level: Awareness Complexity Accuracy Independence LOW HIGH

9. Performance Level: Awareness Complexity Accuracy Independence LOW HIGH

10. Performance Level: Emerging Complexity Accuracy Independence LOW HIGH

11. Performance Level: Emerging Complexity Accuracy Independence LOW HIGH

12. Performance Level: Progressing Complexity Accuracy Independence LOW HIGH

13. Analytical Rubric: Determine Performance Level from Sub-scores If Level of Complexity = 3 (‘entry points’), and Demonstration of Skills & Concepts (Accuracy) = Independence =

14. Calculating an Overall Performance Level • Average the Performance Level subscores in each content area StudentStrandPerformance Level #1#2#3 A Aw (1) - Aw (1) - Em (2) = Awareness (ave: 1.33) B Aw (1) - Em (2) - Em (2) = Emerging (ave: 1.67) CEm (2) - Pg (3) - NI (4) = Progressing (ave: 3.0)

15. So, What Is Proficient, Anyway? When the student: • Addresses challenging, achievable outcomes, based on academic standards • Demonstrates knowledge and skills with >50% accuracy and independence

16. Changes Since 2001 • Fine-tune the Analytical Rubric • We know more than we did in 2001 • Report “Incomplete” portfolios separately • Previously reported as Awareness (lowest score)