What the heck does proficiency mean for students with significant cognitive disabilities?

1. Nancy Arnold, Washington Ron Cammaert, Riverside Publishing Dan Wiener, Massachusetts Ed Roeber, Measured Progress Rachel Quenemoen, NCEO What the heck does proficiency mean for students with significant cognitive disabilities?

2. “…would allow States to use a documented and validated standards-setting process to define academic achievement standards for students with the most significant cognitive disabilities, as defined in proposed Sec. 200.1(d)(2), who take an alternate assessment.” March 20, 2003 NPRM for Title I

3. Many states have completed a careful standards-setting process for their alternate assessment, just as they have done for their general assessments – these efforts are needed to produce alternate achievement standards Examples shown in NCEO reports (see Synthesis Reports 42, 47, 48, plus one more on its way)

4. “These [alternate achievement] standards must be aligned with the State's academic content standards and reflect professional judgment of the highest learning standards possible for those students.” March 20, 2003 NPRM for Title I

5. WHO are the students who participate in alternate assessment aligned to alternate achievement standards?

6. What do we measure? What does good learning look like for this small group of children? What is “achievement?”

7. Developmental approaches Functional approaches Academic approaches – access to the general curriculum, standards-based content, grade level contexts * Diane Browder, 2001 A brief history of educational goals for these students

8. 1.     Careful stakeholder and policymaker development of desired student outcomes for the population, reflecting the best understanding of research and practice, thoughtfully aligned to same content expected for all students. 2.     Careful development, testing, and refinement of assessment methods. 3.     Scoring of evidence of content aligned student work, according to professionally accepted standards, against criteria that reflect best understanding from research and practice. 4.     Standard-setting process to allow use of results in reporting and accountability systems. 5.     Continuous improvement of the assessment process. Development of Alternate AssessmentsQuenemoen, Rigney, & Thurlow, 2002

9. Alternate Assessments are works in progress • Alignment to content standards varies: • Reading and math skills in context of grade level curriculum contexts • Reading and math skills in functional contexts • Reading and math skills in isolation • Weak linkages to reading and math All of these exist within current state approaches

10. Alternate Assessment Strategies Thompson & Thurlow, 2001 IEP Analysis Performance event Combination Checklist Evidence/Portfolio [2 states undecided]

11. 1. Content Standards Linkage. 2. Independence. 3. Generalization. 4. Appropriateness. 5. IEP Linkage. 6. Performance.level of skill or mastery and multiple settings; progress and appropriateness; accuracy, mastery, progress, independence, multiple settings, multiple occasions, or multiple purpose. Case Studies: Common CriteriaQuenemoen, Thompson, & Thurlow, 2003

12. Validity of Alternate Assessments* Face Validity – Are scoring procedures consistent with important best practice indicators in the lives of students with significant disabilities? Concurrent Validity – Do scores correlate with other measures of students achievement and indices of quality programming at the school level? Predictive Validity – How well do scores predict post-school success? *Concepts developed by Harold Kleinert, U of KY

13. 1.     Careful stakeholder and policymaker development of desired student outcomes for the population, reflecting the best understanding of research and practice, thoughtfully aligned to same content expected for all students. 2.     Careful development, testing, and refinement of assessment methods. 3.     Scoring of evidence of content aligned student work, according to professionally accepted standards, against criteria that reflect best understanding from research and practice. 4.     Standard-setting process to allow use of results in reporting and accountability systems. 5.     Continuous improvement of the assessment process. Development of Alternate AssessmentsQuenemoen, Rigney, & Thurlow, 2002

14. Copies of the papers cited and presented are at: http://education.umn.edu/nceo or Search for NCEO Alternate Assessment Topic, Resources

15. What (the Heck) Does “Proficient” Mean? Standard Setting on the WAAS Nancy Arnold Washington Department of Public Instruction CCSSO - San Antonio, TX June 2003

16. Review of NCEO Synthesis Reports Detailed Study of Standard Setting for Alternate Assessment in Other States Review of Relevant Literature Standards for Educational and Psychological Testing (AERA, APA, NCME) Setting Performance Standards: Concepts, Methods, and Perspectives (Cizek) Validation of Process with National Technical AdvisoryCommittee and Advisory Panel Steps Taken to Determine Standard Setting Process

17. Selection of panelists Teachers, parents and administrators Stratified sample Portfolio experts and novices Selection of standard setting materials Performance descriptors Scoring patterns Exemplar portfolios Orientation and training of panelists Standard Setting Methodology

18. Determining alternate achievement performance descriptors Set cut scores in three rounds using scoring patterns revise cut scores using exemplar portfolios finalize cut scores using impact data Evaluate standard setting process Standard Setting Methodology (continued)