Setting Alternate Achievement Standards

1. Setting Alternate Achievement Standards Prepared by Sue Rigney U.S. Department of Education NCEO Teleconference March 21, 2005

2. Regulation on alternate achievement standards §200.1 “For students… with the most significant cognitive disabilities, who take an alternate assessment, a State may, through a documented and validated standards-setting process, define alternate achievement standards…”

3. Content v Achievement Standards

4. Achievement standard 4 Components • Label • Descriptor • Student work • Cut score(s)

5. Alternate assessment based on alternate achievement standards Should have: • Clearly defined structure • Guidelines for which students may participate • Clearly defined scoring criteria and procedures • Report format that clearly communicates student performance in terms of the academic achievement standards defined by the State

6. This is a real assessment • Based on academic content • Structured to permit observation of student achievement of a clearly defined construct • Scoring criteria are consistent with the construct • Aggregation of data is consistent with the construct

7. Essentials of the standard setting process • Define goals; purpose to be served by the cutscore • Select participants; qualifications depend on the decisions to be made using the cutscore, familiarity with population tested • Train participants; start with practice & feedback • Define the performance standard • Data collection procedures Kane, ed Cizek pp.64-69

8. Standard setting options • Test-centered • Angoff - estimates difficulty of items for hypothetical examinees • Bookmark - sets cut points within set of ordered item booklet • Examinee-centered • Contrasting groups - based on judgements about performance of examinees • Body of work • Profile (policy capturing) • Different procedures yield different results

9. Bookmark method • Arrange items in order of difficulty from easiest to hardest • 67% answer this item correctly • Descriptors provide general notion of levels • Judges identify boundaries between performance levels • Judges get feedback data and confirm cut points

10. Body of work method • Judges examine student responses to a variety of tasks • Collections of student work prepared for • Training • Range finding • Pinpointing • Data analysis

11. Generic Steps • Select large, representative panel of judges • Choose method, prepare agenda and training materials • Prepare achievement level descriptors • Train participants to use the method • Compile item ratings or other judgments, prepare summary information as feedback

12. More steps • Participants discuss initial summary information • Another round of ratings, compile information and discuss as in 5 & 6 • Final opportunity to review information and arrive at final recommended achievement standards • Evaluate standard-setting process • Assemble documentation of process and other evidence appropriate for validity

13. Document the procedures • Number of participants, how selected • Qualifications of participants • Qualifications of those designing methodology • Materials used • Instructions to participants • Frameworks developed by participants • Timeline, schedule of events, actual agenda

14. More documentation • Any deviations from intended procedures • Evidence of consistency/inconsistency of judgments • Between rates • Between rounds

15. Checklist for performance standards • Understandable and useful for stakeholders • Clearly differentiate among levels • Grounded in student work but not tied to status quo • Built by consensus • Focused on learning Handbook, p. 16

16. Peer Review looks for • Formal adoption by Board? • How linked with grade-level content? • Involvement of diverse stakeholders? • Alternate achievement standard available for each grade level? • Documentation of process used? • Results reported in terms of the alternate achievement standards?

17. Questions • What should alternate standards look like? Grade-by-grade? • How can group data reflect individual student growth? • What if the AA-AAS doesn’t meet “commonly accepted professional standards”? Will our state “pass” peer review?