MEAP Writing Assessment Update

1. MEAP Writing Assessment Update Presented at the December 2010 meeting of the Michigan State Board of Education

2. Changes and Improvements • Moved from grades 3-8 to grades 4 & 7 • Added more points to improve reliability • 15, 20, and 23 points (in 2006, 2007, and 2008, respectively) • 2009 = Field tested new assessment • 2010 = 50 points (grades 4 & 7 only) • Added analytic rubrics • Scoring multiple facets of student writing samples independently • Gives more and targeted feedback to schools and students • Standardized the question asked of students on the “Peer Response to a Student Writing Sample” essay • Added Informational Writing to the assessment • Maintained Narrative Writing on the assessment • Divided the New Writing assessment over two days

3. New Writing Assessment Design • Essay in Narrative Style 15 points • Essay in Informational Style 15 points • Multiple Choice Response to Student Writing Sample #1 8 points • Multiple Choice Response to Student Writing Sample #2 8 points • Essay Response to Student Writing Sample 4 points • Total50 points • Total from essays 34 points • Percent of total from essays 68

4. Essay Response to Student WritingSample • Standardized Question • “What could the writer do to improve this sample?” • Improves consistency of scoring from year to year • Responsive to grade level content expectations regarding student understanding of the writing process, including such aspects as revision for sequence, flow, style, clarity, coherence, and consistency; proofing; editing; etcetera. • New Holistic Rubric • 4 points • Scored on whether the student adequately, and in an integrated manner, gave recommendations for improvement using relevant and specific details from the sample to support the recommendations, and an explanation of why their recommendations improve the sample

5. Essay Response to a NarrativeWriting Task Write about change: Your life changes as you get older. For example, you can now do things you could not do before. Think about things that have changed in your life. Write about change. Do only oneof the following: Write a story about a time when you had to make a change OR Write a story about someone you know that had to make a change OR Write a story in your own way about change Includes a checklist for the writers to check their work against

6. Essay Response to a NarrativeWriting Task, continued… • Responsive to grade level content expectations regarding • Creation of a narrative • Use of ideas consistent with narrative • Organization consistent with narrative • Personal style • Grammar and usage • Spelling

7. Essay Response to a NarrativeWriting Task, continued… • Analytic Rubric (15 points total—ideas doubled)

8. Essay Response to an InformationalWriting Task • Example prompts • Grade 4 • “Animals are interesting. Think about two animals. What do they look like? Where do they live? What do they eat? What else do you know about them? Compare two animals.” • Grade 7 • “What effect does watching television have on study habits?” • The prompts include checklists for high quality writing for the writers to check their work against

9. Essay Response to an InformationalWriting Task, continued… • Responsive to grade level content expectations regarding • Creation of an informational text • Use of ideas consistent with informational text • Organization consistent with informational text • Personal style • Grammar and usage • Spelling

10. Essay Response to an InformationalWriting Task, continued… • Analytic Rubric (15 points total—ideas doubled)

11. New MEAP Writing Reports • Goals for New Reports • Represent best practices in writing instruction and assessment • Provide rich feedback regarding student writing skills in order to… • Aid teachers in designing effective instruction for individual students • Assess curriculum needs • Help parents and guardians understand the writing progress of students

12. Comparing Elements of Old Versus New MEAP Writing Reports • New • Overall Writing Scale Score • Overall Writing Performance Level • Item scores for 16 Multiple Choice responses to student writing samples • Revised holistic (0-4) score for Essay response to student writing sample • Analytical score for Essay response to narrative writing task (15 points total) • (0-3) Ideas, doubled • (0-3) Organization • (0-3) Style • (0-3) Conventions • Analytical score for Essay response to informational writing task (15 points total) • (0-3) Ideas, doubled • (0-3) Organization • (0-3) Style • (0-3) Conventions • Total of 27 reported elements per student • Total of 50 points • Old • Overall Writing Scale Score • Overall Writing Performance Level • Item scores for 13 Multiple Choice responses to student writing samples • Holistic (0-4) score for Essay response to student writing sample • Total of 17 reported elements per student • Total of 23 points

13. Current Status and Next Steps • Students have taken the MEAP Writing Assessment • Assessments have been returned to the contractor (Measurement Inc.) for processing • Processing, scoring, and data file creation is nearly complete • All MEAP results (minus Writing) will be released in data files to schools as soon as data file creation is complete and validated (released last year in mid December) • Standard setting activity will occur for the new writing assessment in January, 2011 • Results of standard setting will be reviewed and validated by our Technical Advisory Committee of national measurement experts • Results of standard setting will be presented to the State Board of Education at the February, 2011 meeting • Pending SBE action at the February meeting, all reports of MEAP results will be finalized and released to schools and the public as soon as possible

14. MEAP Writing Standard Setting Plan • A “Modified Body of Work” procedure will be used for standard setting to identify cut scores on the writing test • Three cut scores will be set to identify four ranges on the score scale representing student achievement in writing as follows: • The process will cover three days with training on day 1, and multiple rounds of standard setting activities on days 2 and 3 • Roughly 20-25 panelists will be recruited to serve in each of grades 4 and 7 • Panelists will be representative of Michigan education stakeholders and will include teachers, administrators, parents, and other business or community leaders

15. Brief Overview of the Modified Body of Work Method to be Employed • Before Round 1 • Panelists will take the assessment • Panelists will review “Performance Level Descriptors” (PLDs) • Descriptions of what students in each of the four performance levels should know and be able to do • Panelists will become intimately familiar with the PLDs • All subsequent sorting of student work samples will be based on their match to specific PLDs • Panelists will receive training on the standard setting process

16. Round 1 • Panelists will review a large set of student work samples • Each work sample will include responses to all tasks on the assessment (or the students’ bodies of work) • The complete set of bodies of work will span the full range of scores observed on the assessment • The complete set of bodies of work will be strategically selected to have desirable statistical and qualitative features • Particular attention will be given to the students’ essays • Panelists will examine the student work samples, the scores that students received on those work samples, and their compatibility with the Performance Level Descriptors (PLDs) to provide recommendations for cut scores.

17. Between Round 1 and Round 2 • Statistical analyses will be run to identify the three group cut scores from the cut scores that each panelist recommended in round 1 • Feedback information will be provided that shows the cut score recommendations for each panelist and the group cut scores • Panelists will have the opportunity to discuss the feedback information prior to round 2

18. Round 2 • Panelists will again examine student work samples and the scores that student received of those work samples to provide cut score recommendations. However, in this round, the selected bodies of work… • Will not span the entire range of student achievement on the assessment • Will be targeted at, just below, and just above the three recommended cut scores resulting from round 1 • Panelists will also be provided with information on relationship of the scores on these new work samples to the work samples that they rated in round 1. • The targeted selection allows for fine tuning of the three cut scores

19. Between Round 2 and Round 3 • Statistical analyses will be run to identify the refined (round 2) cut score recommendations • Feedback information will again be provided that shows the cut score recommendations for each panelist and the group cut scores • Statistical analyses will also be run to provide impact data to the panelists (e.g., the percentage of students in each of the four categories and that would be at or above each of the three cut scores)

20. Round 3 • Panelists will receive impact information and get to see how the impact information is related to the student work samples that they that examined in rounds 1 and 2 • Vertical Articulation • Grade 4 and 7 panelists will meet together to make sure that the recommended cut scores are appropriately articulated across the two grades • Information on high school writing cut scores will also be incorporated to assure coherence between grade 4, 7, and 11 writing cut scores • Grade 11 writing cut score is already very similar in difficulty to the ACT College Readiness Benchmark in English • Panelists will be reminded that cut scores should be tied to the Performance Level Descriptors • Panelists will make their final recommendations regarding cut scores

21. See You After Standard Setting! • We look forward to bringing for your approval the recommended cut scores on this new and much more informative MEAP writing assessment

22. Contact Information • Joseph Martineau • Director • Office of Educational Assessment & Accountability • Michigan Department of Education • martineauj@michigan.gov • 517-241-4710