The MAAECF: Assessment Procedures, Supporting Data, Technical Considerations

1. E & R Assessments 2007 1 The MAAECF: Assessment Procedures, Supporting Data, & Technical Considerations Stephen N. Elliott Andrew T. Roach Vanderbilt University Georgia State University Presented at the 8th Annual Maryland Conference On Alternate Assessment

2. E & R Assessments 2007 2 Key Questions to be Addressed 1. What does the Mississippi Alternate Assessment of Extended Curriculum Frameworks (MAAECF) measure? 2. How do you conduct an alternate assessment using the MAAECF Rating Scale for students with significant cognitive disabilities? 3. What evidence is there to support the claim that the MAAECF yields reliable and valid results? 4. What are salient technical issues associated with this approach to alternate assessment? Note to Presenter: Simply read these 4 questions as a means of focusing participants attention and highlighting the practical purposes of the presentation. Every participant should be able to answer these 4 questions when they have completed the session.Note to Presenter: Simply read these 4 questions as a means of focusing participants attention and highlighting the practical purposes of the presentation. Every participant should be able to answer these 4 questions when they have completed the session.

3. E & R Assessments 2007 3 Technical Issues to Consider Issue #1. Teachers as reliable judges of student performance. Issue #2. Reliability of teachers’ ratings and the concept of independence. Issue #3. Use of standards-based IEPs to drive the development and collection of evidence for a meaningful sample of items. Issue #4. Amount and quality of classroom evidence needed for valid ratings of proficiency.

4. E & R Assessments 2007 4 Validity Evidence Plan

5. E & R Assessments 2007 5 MAAECF : A Comprehensive, Evidence-BasedRating Scale that Uses Teachers’ Judgments to Measure Student Achievement

6. E & R Assessments 2007 6 Assessment of the Mississippi Extended Curriculum Frameworks (MECF) Extended Content Standards - General statements that describe what students should understand and be able to do in reading/ language arts, mathematics, and science Competencies and Objectives - Specific statements of expected knowledge and skills necessary to meet a content standard requirement Note to Presenter: Read or paraphrase the following definitions of key terms. Resources to “show and tell” a little about: WI Content Standards document. Ideally have a couple of copies to pass around so participants can see examples of a content standard, a performance standard, and the 4 level proficiency framework. MECS document. Content standards - general statements that describe what students should understand and be able to do in various subject matter content areas such as mathematics or reading. Competencies and Objectives - specific statements of expected knowledge and skills necessary to meet a content standard requirement at a particular grade level. Proficiency standards - descriptive categories that describe the degree to which performance standards have been attained. In Mississippi, there are four levels of proficiency used to describe how well a student has done on a test designed to measure the state's content and performance standards. Also note that the state developed Mississippi Extended Content Standards (MECS) – The Mississippi Extended Content Standards (MECS) are organized according to a 4-level hierarchical structure with the most general level being a Content Area. This initial version of the MECS covers two content areas: Language Arts and Mathematics. In late 2005, the Extended Content Standards for the area of Science will be developed. Each Content Area is comprised of multiple Content Strands that subsume multiple Competencies, and in turn, each Competency can be further defined by specific Objectives. Figure 1 illustrates the relationship among these four levels of content. The collective MECS includes curriculum content that students with significant cognitive disabilities are expected to be exposed to during the course of their education. Note to Presenter: Read or paraphrase the following definitions of key terms. Resources to “show and tell” a little about: WI Content Standards document. Ideally have a couple of copies to pass around so participants can see examples of a content standard, a performance standard, and the 4 level proficiency framework. MECS document. Content standards - general statements that describe what students should understand and be able to do in various subject matter content areas such as mathematics or reading. Competencies and Objectives - specific statements of expected knowledge and skills necessary to meet a content standard requirement at a particular grade level. Proficiency standards - descriptive categories that describe the degree to which performance standards have been attained. In Mississippi, there are four levels of proficiency used to describe how well a student has done on a test designed to measure the state's content and performance standards. Also note that the state developed Mississippi Extended Content Standards (MECS) – The Mississippi Extended Content Standards (MECS) are organized according to a 4-level hierarchical structure with the most general level being a Content Area. This initial version of the MECS covers two content areas: Language Arts and Mathematics. In late 2005, the Extended Content Standards for the area of Science will be developed. Each Content Area is comprised of multiple Content Strands that subsume multiple Competencies, and in turn, each Competency can be further defined by specific Objectives. Figure 1 illustrates the relationship among these four levels of content. The collective MECS includes curriculum content that students with significant cognitive disabilities are expected to be exposed to during the course of their education.

7. E & R Assessments 2007 7 MECF Organizational Structure

8. E & R Assessments 2007 8 LA Strands and Competencies

9. E & R Assessments 2007 9 MECF Classroom Tasks/Activities to Help Teach and Generate Evidence

10. E & R Assessments 2007 10 Sample MAAECF Items and Rating Scale Format

11. E & R Assessments 2007 11 Essential Features of Evidence-Based Judgment Systems like the MAAECF Unbiased items. High-quality evidence about students’ knowledge & skills. Clear and objective scoring criteria. Procedures to ensure the reliability of the scores. Meaningful methods for communicating results of the assessment. Trained judges – all educators using the MAAECF must attend a workshop + pass a qualification test!

12. E & R Assessments 2007 12 MAAECF Honors “Teachers as Tests” The research-based concept of teachers as tests emphasizes that teachers collect a substantial amount of data about students and when they are provided a structure for collecting evidence and a method for quantifying and reporting this information, it can be used like other highly reliable test results. Key studies on the reliability of teachers’ judgments: Hoge & Coladarci (1989) Demary & Elliott (1998) Hurwitz, Elliott, & Braden (2007)

13. E & R Assessments 2007 13 What Does the MAAECF measure? The MAAECF (regardless of level) includes more than 100 items representing prerequisite knowledge and skills in core academic areas. Rating scale items are based on the objectives specified in the Mississippi Extended Curriculum Frameworks (MECF). Note that various levels of the MAAECF exist for students in the grade clusters of 3-5, 6-8, and 12. Teacher rates proficiency using a 4-point scale (0 = Non-Existent to 3 = Accomplished). The same item level rubric is used across all grade clusters. Results are reported as performance levels (Minimal, Basic, Proficient, or Advanced) in each content area. Definitions and cut scores for these levels vary by grade. The results are on a continuum that is very similar to the general education test’s performance levels. Presenter: Note that the MAA is a rating scale completed by an individual with knowledge of the student’s IEP goals, objectives, benchmarks, curriculum, knowledge, and skills. This person is usually a teacher. The MAA borrows from the science of behavior rating scales, the use of performance assessment scoring rubrics, and the fact that teachers have collected vast amounts of information about their students and can reliably report the information when provided a meaningful structure. Resources: (1) Hold up a copy of both the Academic Content Standards and the MECS and briefly explain their relationship. Then add a copy of the MAA and physically model how they are aligned or overlap in content (knowledge and skills) covered. (2) Hold up a copy of state proficiency standards.Presenter: Note that the MAA is a rating scale completed by an individual with knowledge of the student’s IEP goals, objectives, benchmarks, curriculum, knowledge, and skills. This person is usually a teacher. The MAA borrows from the science of behavior rating scales, the use of performance assessment scoring rubrics, and the fact that teachers have collected vast amounts of information about their students and can reliably report the information when provided a meaningful structure. Resources: (1) Hold up a copy of both the Academic Content Standards and the MECS and briefly explain their relationship. Then add a copy of the MAA and physically model how they are aligned or overlap in content (knowledge and skills) covered. (2) Hold up a copy of state proficiency standards.

14. E & R Assessments 2007 14 Item Count Per Scales & Grade Levels

15. E & R Assessments 2007 15 Mississippi Alternate Assessment and Accountability System

16. E & R Assessments 2007 16 MAAECF : Administration & Scoring Procedures

17. E & R Assessments 2007 17 The MAAECF Process Once the IEP Team decides that a student is eligible for an alternate assessment, a 5-step process must be followed: Step 1: Align MAAECF items with IEP goals, objectives, or other learning objectives. Step 2: Collect performance evidence for a sample of items for each strand using Evidence Worksheets. Step 3: Analyze and rate proficiency of all items. Presenter: Note that the student’s teacher identifies MAA items that align with the student’s IEP goals, objectives, or benchmarks. For each IEP aligned MAA item, at least two forms of evidence from the student’s classroom are collected. Elaborate on ratings – reference people to pages in MAA rating form where they can view the item proficiency criteria and the overall Performance Summary continua.Presenter: Note that the student’s teacher identifies MAA items that align with the student’s IEP goals, objectives, or benchmarks. For each IEP aligned MAA item, at least two forms of evidence from the student’s classroom are collected. Elaborate on ratings – reference people to pages in MAA rating form where they can view the item proficiency criteria and the overall Performance Summary continua.

18. E & R Assessments 2007 18 The MAAECF Process (continued) Step 4: Summarize proficiency scores and performance level decisions. A second educator completes the process by reviewing collected evidence, providing ratings on only items with evidence samples, and checking the accuracy of the performance level decisions. Step 5: Report results after the first rater and the individual who completes the reliability check have reached agreement. Presenter: Note that a second rater, usually another teacher, reviews the evidence collected to ensure that the original teacher’s rating are reliable. A proficiency rating (i.e., non-existent, emerging, developing/developed, and proficient/generalized) is assigned to each MAA item to describe student skill development. Items that are not relevant or possible given a student’s disability are rated as not applicable. An Overall Performance Level (i.e., Prerequisite Skill Level 1-4), for each domain is determined. Resource: Show an overhead of the Proficiency Scoring Criteria and also select one of the Content Area Developmental Continua to highlight the developmental and criterion-reference nature of the various Prerequisite Skills Levels. Connect the 4 PS levels to the Proficiency Levels for the MCT. It is important that you have overheads of pages 2, 3, and 4 from the MAA Rating Scale. It is also important to have overheads of a content area, such as mathematics, that is correctly completed. Finally, have an overhead of the summary score page (last page of MAA Rating Form). Presenter: Note that a second rater, usually another teacher, reviews the evidence collected to ensure that the original teacher’s rating are reliable. A proficiency rating (i.e., non-existent, emerging, developing/developed, and proficient/generalized) is assigned to each MAA item to describe student skill development. Items that are not relevant or possible given a student’s disability are rated as not applicable. An Overall Performance Level (i.e., Prerequisite Skill Level 1-4), for each domain is determined. Resource: Show an overhead of the Proficiency Scoring Criteria and also select one of the Content Area Developmental Continua to highlight the developmental and criterion-reference nature of the various Prerequisite Skills Levels. Connect the 4 PS levels to the Proficiency Levels for the MCT. It is important that you have overheads of pages 2, 3, and 4 from the MAA Rating Scale. It is also important to have overheads of a content area, such as mathematics, that is correctly completed. Finally, have an overhead of the summary score page (last page of MAA Rating Form).

19. E & R Assessments 2007 19 Team Work for Reliable Results:Summarizing the Roles of Raters 1 and 2 Rater 1

20. E & R Assessments 2007 20 Step 1: Alignment of Items Determine in which content areas a student will be assessed. Identify MAAECF items that align with the student’s IEP goals and objectives, or other learning goals. If there is not at least 1 IEP-aligned item for each competency (4 in LA, 5 in Math, and 4 in Science), then an item must be selected and evidence generated for that item. Presenter: Note that this alignment process requires both professional judgment and an understanding of the state’s academic standards. The goal is to identify and assess connections between MAA items and the student’s IEP goals, objectives, or benchmarks. The alignment process requires a person to be quite familiar with the student’s IEP and the items on the MAA rating scale. It is possible that an IEP goal, objective, or benchmark may align with more than one item on the rating scale. In addition, it is entirely likely than some IEP objectives will not align with any of the MAA items. Once a set of IEP-aligned items has been identified, it is the responsibility of the rater (usually the child’s main teacher) to collect examples of work samples, progress records, observations, etc. to provide a recent and representative sample of information to substantiate the rater’s evaluation.Presenter: Note that this alignment process requires both professional judgment and an understanding of the state’s academic standards. The goal is to identify and assess connections between MAA items and the student’s IEP goals, objectives, or benchmarks. The alignment process requires a person to be quite familiar with the student’s IEP and the items on the MAA rating scale. It is possible that an IEP goal, objective, or benchmark may align with more than one item on the rating scale. In addition, it is entirely likely than some IEP objectives will not align with any of the MAA items. Once a set of IEP-aligned items has been identified, it is the responsibility of the rater (usually the child’s main teacher) to collect examples of work samples, progress records, observations, etc. to provide a recent and representative sample of information to substantiate the rater’s evaluation.

21. E & R Assessments 2007 21 Step 2: Collect Evidence of Skills Teachers must collect evidence from 2 different categories for at least 1 item for each competency. Work Samples Tests Observations Interviews Video/Photo Audio Tape Presenter: Note that teachers conducting an alternated assessment can begin doing ratings and collecting evidence to support their ratings any time after the beginning of school up April 30th, the date that the Overall Performance Summary Report must be submitted to the district assessment coordinator. Provide more guidance about evidence – how much is enough, how one “piece” can be used for multiple items, it should be indigenous to the classroom (i.e., already exists) but can be developed in reaction to items. It also should used to communicate with parents/guardians. Stress that evidence must be Recent and Representative. Also note that one piece of evidence can be used to support ratings of several items. For example, in the case of April that we will learn more about, her teacher used a videotape of April doing work in the school store to document skills for several mathematics and social studies items. Presenter: Note that teachers conducting an alternated assessment can begin doing ratings and collecting evidence to support their ratings any time after the beginning of school up April 30th, the date that the Overall Performance Summary Report must be submitted to the district assessment coordinator. Provide more guidance about evidence – how much is enough, how one “piece” can be used for multiple items, it should be indigenous to the classroom (i.e., already exists) but can be developed in reaction to items. It also should used to communicate with parents/guardians. Stress that evidence must be Recent and Representative. Also note that one piece of evidence can be used to support ratings of several items. For example, in the case of April that we will learn more about, her teacher used a videotape of April doing work in the school store to document skills for several mathematics and social studies items.

22. E & R Assessments 2007 22 Characteristics of Good Evidence Recent (collected during the current school year & dated) Representative (several forms that are typical performances of knowledge and skills with classroom materials and instructional accommodations) Relevant (evidence that is clearly indicative of an item that is aligned with content standards, and that identifies the relevant item number(s) for the second rater and others) Reliable (increases the likelihood that 2 or more raters have the same understanding of the item by documenting the support provided to get the response from the student and indicating the overall accuracy of the student’s typical response)

23. E & R Assessments 2007 23 Evidence Collection Worksheets

24. E & R Assessments 2007 24 Quality Evidence Matters: Collected Evidence Should Meet or Exceed the Standard!

25. E & R Assessments 2007 25 Step 3: Criteria for Rating All Items 0 = NON-EXISTENT (Can’t do currently) 1 = EMERGING (Aware and starting to do) 2 = PROGRESSING (Can do partially and inconsistently) 3 = ACCOMPLISHED (Can do well and consistently) ***Examine descriptions of these Proficiency Rating Levels*** Presenter: Note that this slide lists some common criteria that IEP teams have used to describe students’ progress during annual reviews of their IEPs. These same criteria can be used to communicate about students’ performances on alternate assessments in content domains. When Showing this Slide: Read each of the four criteria listed and ask participants to provide some examples of ways to use each of the criteria. Encourage them to think about a student that they know well and how these criteria could be used to communicate about the students’ skills in reading or math. Note: It is important to discriminate between NA & non-existent. Presenter: Note that this slide lists some common criteria that IEP teams have used to describe students’ progress during annual reviews of their IEPs. These same criteria can be used to communicate about students’ performances on alternate assessments in content domains. When Showing this Slide: Read each of the four criteria listed and ask participants to provide some examples of ways to use each of the criteria. Encourage them to think about a student that they know well and how these criteria could be used to communicate about the students’ skills in reading or math. Note: It is important to discriminate between NA & non-existent.

26. E & R Assessments 2007 26 MAAECF Item Level Proficiency Rating Scale and Descriptive Criteria

27. E & R Assessments 2007 27 Step 4: Overall Proficiency Scores and Performance Continua Students earn an Individualized Proficiency Total Score for each content area. The total scores are used to guide the determination of which of the four Overall Performance Levels best describes the student’s achievement. Minimal Basic Proficient Advanced ***Examine detailed Performance Levels*** Presenter: Note that the score continuum is a downward extension of the four performance levels (i.e., minimal performance, basic, proficient, and advanced proficient) that result fro the MCT. Elaborate on what the characteristics are that differentiate a PS1 from a PS2, etc. * Frequency/accuracy * Amount of support * Quality of work * Generalization Reference the scoring continua. Finally, note that persons taking MCT would be expected to be functioning at or above PS4. Resource Material: Use a copy of the 8 level scoring criteria that was used to develop the MAA developmental continua to illustrate the complete continua for a content area that includes the MAA+WKCE.Presenter: Note that the score continuum is a downward extension of the four performance levels (i.e., minimal performance, basic, proficient, and advanced proficient) that result fro the MCT. Elaborate on what the characteristics are that differentiate a PS1 from a PS2, etc. * Frequency/accuracy * Amount of support * Quality of work * Generalization Reference the scoring continua. Finally, note that persons taking MCT would be expected to be functioning at or above PS4. Resource Material: Use a copy of the 8 level scoring criteria that was used to develop the MAA developmental continua to illustrate the complete continua for a content area that includes the MAA+WKCE.

28. E & R Assessments 2007 28 LA Performance Level Descriptor(with Cut Scores for Grades 3, 4, and 5)

29. E & R Assessments 2007 29 MAAECF Item Count & Score Ranges by Performance Level & Grade Clusters

30. E & R Assessments 2007 30 2nd Rater’s Responsibilities Review evidence of student’s knowledge and skills and then independently rate items with which they are aligned. This is done on a separate copy of the MAAECF. Review the 1st rater’s ratings of evidence-based items to determine general degree of agreement and then examine all item ratings to learn more about the student. These ratings provide additional evidence for the 2nd rater. Based on all of the evidence, independently select the overall Performance Level (Minimal, Basic, Proficient, or Advanced) that best characterizes the student’s current functioning. Double check 1st rater’s total score for the students in Language Arts, Math, and Science to ensure all items were rated and the total scores are correct. Compare their overall Performance Level determination to that of the 1st rater and decide if it is reliable using the Inter-Rater Reliability Estimate table of the MAAECF for Language Arts and Math. Settle any disagreements that result in unreliable decisions.

31. E & R Assessments 2007 31 Inter-Rater Agreement Table

32. E & R Assessments 2007 32

33. E & R Assessments 2007 33 Descriptive Statistics & Key Evidence to Support the Inferences Made From MAAECF Scores About Student Achievement

34. E & R Assessments 2007 34 Descriptive Statistics for MAAECF Math

35. E & R Assessments 2007 35 Frequency Distributions for Math

36. E & R Assessments 2007 36

37. E & R Assessments 2007 37 Inter-rater Agreement & Evidence Quality Study

38. E & R Assessments 2007 38 Quality of Evidence Samples Matter!

39. E & R Assessments 2007 39 Factor Analytic Evidence to Support Claims about the MAAECF’s Structure

40. E & R Assessments 2007 40 Evidence Based on Relations to Other Variables

41. E & R Assessments 2007 41 More Evidence Based on Relations to Other Variables

42. E & R Assessments 2007 42 Evidence based on Consequences of Testing

43. E & R Assessments 2007 43 Item and Scale Revision Plan in Progress for 2008

44. E & R Assessments 2007 44 MS CAARES Project Enhancements to the MAAECF

45. E & R Assessments 2007 45 Thank You Contact Information Stephen N. Elliott Vanderbilt University 616-322-2538 steve.elliott@vanderbilt.edu Andrew T. Roach Georgia State University 404-413-8176 aroach@gsu.edu