MCAS 2019 Preliminary Data and Discrepancy Reporting Overview

1. MCAS 2019 Preliminary Data and Discrepancy Reporting Overview July 2019

2. Presenters • Bob Lee, MCAS Chief Data Analyst • Scott Kelley, Reporting Specialist • Jodie Zalk, Test Administration Coordinator

3. Logistics for This Session • Due to this year’s early discrepancy reporting period, this session is only available as a recording, and is posted at: • MCAS Resource Center -- mcas.pearsonsupport.com/training • In the section entitled “Training Sessions Previously Offered This School Year” • Questions should be directed to the appropriate group, as follows: • Questions about this session or testing policy should be directed to Student Assessment at mcas@doe.mass.edu or 781-338-3625. • Questions about the Discrepancy Reporting Tool should be directed to the MCAS Service Center at mcas@measuredprogress.org or 800-737-5103.

4. 01 Data Deliveries and Early 2019 MCAS Results 02 What’s New 03 Discrepancy Reporting Window CONTENTS

5. Data Deliveries and Early 2019 MCAS Results 01

6. 2019 Data Delivery Milestones • April 29: Grade 10 ELA machine scored results. • June 11: MCAS-Alt Portfolio Feedback Forms in DropBoxes • June 18: ELA Grade 10 data in PearsonAccessnext (no scaled scores or SGP; released essays on June 24) • July 1: Grades 3–8 ELA and Math; Grades 5 & 8 STE (ELA released essays; no scaled scores or SGP) • July 8: Grade 10 Math (no scaled scores or SGP) • July 24 in DropBoxes: Raw scores for grades & subjects (.csv only; no rosters, scaled scores, or SGP) • July 29 in DropBoxes: Preliminary participation data (aggregated by subgroup & subject)

7. 2019 Data Delivery Milestones (cont’d) • July 31 in Edwin Analytics: Raw scores for all grades and subjects (curriculum & item analysis interactive reports; no scaled scores or SGP) • August 23 in DropBoxes: Full preliminary results for all grades and subjects (includes scaled scores, achievement levels, SGP; .csv data files only; no rosters) • August 30 in Edwin Analytics: Full preliminary results for all grades and subjects (includes scaled scores, achievement levels, SGP) • Mid-September: Official embargoed MCAS rosters and .csv files in DropBoxes; interactive reports in Edwin Analytics • Late September: Public release of official MCAS results • Official state, district and school-level results on profiles.doe.mass.edu • Parent/Guardian reports received by superintendents

8. Analyzing Early Release Raw Data • Excel Template: Statewide item information, state average points and percent correct per item (item difficulty) available for next-generation MCAS tests at: • PearsonAccessnext in Support>Documentation • MCAS Resource Center in Released Items (See link below.) • Compare school & district points/percent correct to state averages. • Instructions for using the templates are available in both places. • Released items for next-generation tests are available on the MCAS Resource Center. • http://mcas.pearsonsupport.com/released-items/ • Includes all items and item descriptions for ELA, Math, and STE operational items

9. Using the item analysis template (from PAN / Resource Center)

10. Using the item analysis template: pre-formatted graph

11. What’s New 02

12. Next-Generation Grade 10 ELA Test Design vs. Legacy Next-Generation Grade 10 ELA Test Design Legacy Grade 10 ELA Test Design ELA Test Designs www.doe.mass.edu/mcas/tdd/ela.html?section=testdesign

13. Next-Generation Grade 10 Math Test Design vs. Legacy Legacy Grade 10 Math Test Design Next-Generation Grade 10 Math Test Design Math Test Designs www.doe.mass.edu/mcas/tdd/math.html?section=testdesign

14. Next-Generation Grades 5 & 8 STE Test Design vs. Legacy Legacy Grades 5 & 8 STE Test Design Next-Generation Grades 5 & 8 STE Test Design STE Test Designs www.doe.mass.edu/mcas/tdd/sci.html?section=testdesign

15. 2019 vs. 2018 ELA Average Raw Score Stats Changes in raw scores do not necessarily mean state results will go down. * *Transition to Next-Gen MCAS in 2019

16. 2019 vs. 2018 Math Average Raw Score Stats Changes in raw scores do not necessarily mean state results will go down. * *Transition to Next-Gen MCAS in 2019

17. 2019 vs. 2018 STE Average Raw Score Stats * * *Transition to Next-Gen MCAS in 2019

18. CBT Released ELA Essays in PDF for All Grades ELA released essay responses for ELA grades 3‒8 & 10 in PearsonAccessnext • Available in school’s Published Reports • One file per grade • Released items posted at mcas.pearsonsupport.com/released-items/ela

19. Superintendent Survey on Parent/Guardian Report Preference • DESE surveyed superintendents in late February 2019 regarding districts’ preference in receiving parent reports in paper and/or electronic form. • Sixty-eight responses were received to the following three questions.

20. Superintendent Survey on Parent/Guardian Report Preference

21. Superintendent Survey on Parent/Guardian Report Preference • Representative comments from the survey include: • Hard copy goes in student cumulative folder. Until electronic cumulative folders are widespread, two hard copies will have to be provided. Sending parents the pdf would be cumbersome as matching student report to parent email would require significant manual labor in school. • My concern with not receiving any paper copies is that I think parents can/should receive a personalized report and I don't think the printing burden should be shifted to districts. We could make do with one paper copy to be mailed home and just use electronic copies for internal purposes. • The electronic version would be very challenging in small rural areas where some families still do not utilize internet. Paper copies are much appreciated. • The biggest issue is distribution of reports to multiple households when a child's parents are divorced.

22. Grade 10 ELA Essay Item Not Scored • ELA Grade 10 essay item 27 was not scored. Results for the item will be reported on the Parent/Guardian Report as follows:

23. Grade 10 Next-Gen Parent/Guardian Report • Scaled score, SGP, and Competency Determination status are on p. 1 • Inside and back pages are similar to grades 3–8 reports.

24. Equipercentile linking model for establishing the interim CD standards for the Classes of 2021 & 2022 (ELA example) %

25. Estimated Equipercentiles for Grade 10 ELA and Math

26. Next-Gen 2019 MCAS Released Item Sets • The Department posted interactive released item sets in TestNav on the MCAS Resource Center at mcas.pearsonsupport.com/released-items. • works in a browser; provides selected response score • PBT Released item sets are posted on the Department’s MCAS Test Questions page at www.doe.mass.edu/mcas/release.html. • includes high school Science and Technology/ Engineering

27. Changes Between Previous Versions and the New Data File • Released typed response essays will be available in PAN • Medical absences, First-Year EL exemptions noted • Security breaches invalidated (cell phone use, inappropriate accommodations, cheating cases that have been resolved) • Duplicate cases resolved, small numbers of data processing errors corrected

28. Using Preliminary Results – Embargo Policies Do • Use preliminary results for educational planning purposes and research with appropriate staff. • You may share preliminary results with students and families, but take care not to imply certainty about preliminary results. Don’t • Discuss aggregate results in public. • Show preliminary aggregate reports at school committee meetings.

29. Discrepancy Reporting Window 03

30. MCAS Discrepancy Reporting • July 24−August 6, 5 p.m. • CSV data files are now in DropBoxes • Discrepancy reporting tool is at www.mcasservicecenter.com • Use IE 8−10; Firefox 17+; Safari 5−6 • Do not use Chrome • MCAS Service Center passwords were distributed in January 2019 • Spring 2019 MCAS Guide to Reporting a Discrepancy is posted both places • Call 800-737-5103 if you need assistance

31. How to Review Your Data for Potential Discrepancies • Review individual student data using data files (now in DropBox; July 31 in Edwin Analytics) • Confirm that each student has a SASID. • Confirm that each student enrolled during the testing windows is listed only once. • Verify that each student’s test participation status is correct (the Discrepancy Reporting Guide contains explanations of the codes).

32. Test Participation and Test Status Codes

33. Discrepancies Schools/Districts Can Change Directly • A missing SASID can be added. • Test participation status can be changed from “Medically documented” to “Absent” or vice-versa. • Incorrectly voided tests can be un-voided (all other changes to participation status must be reviewed by Department staff).

34. Discrepancies Requiring DESE Data Review • Student was not tested at your school (or an affiliated test site) but appears on your roster with test results • Incorrect or invalid SASID can be corrected; • Test results belong to a different student • Student was fully tested but does not appear in results • Student has two sets of results (listed as “DUP”) • Student is reported as “Absent” but was not enrolled at your school during testing window • Student is reported as “Absent” but was tested and received a response accommodation (typed or scribed response) • Participation status should be changed from “Absent” to “Tested” (student confirmed as present during all test days)

35. Discrepancies Requiring DESE Data Review (continued) • Participation status should be changed to “First-year EL” or “Transferred” (subject to Department validation using SIMS) • Special access accommodation status is incorrect • High school only: • Student is reported as “Absent” for ELA and/or Math tests but previously participated in a springgrade 10 test • summarize = 1 but student was not required to test • CD Status is not yet available • A grade 10 student is reported with a test status of “TR” (Tested–Retest)

36. What Not to Report as a Discrepancy • Students with correct SASID but incorrect name, birth date, gender, race/ethnicity, or any other SIMS data; update next SIMS submission • Students involved in a test breach whose results appear as “INV.” These results have been invalidated due to a security breach. • Correctly voided CBT test or answer booklets (listed as “VAB”) • Students reported as enrolled in March and June 2019 SIMS, did not participate in test(s), and therefore assigned a performance level of “Absent”: update enrollment in next SIMS submission • Due to the proven accuracy of paper test-booklet scanning, missing items and groups of missing selected response items should not be reported for paper-based test-takers

37. What Not to Report as a Discrepancy (continued) • Classroom reports—incorrect Student Course Schedule data • High school only: • Student who skipped grade 10 and did not participate in grade 10 spring ELA and Math tests and is reported as “Absent” (retests do not count)

38. Email and Phone Support MCAS Service Center DESE Office of Student Assessment Services • All questions on logistics and technology (e.g., PearsonAccess Next, SR/PNP, TestNav) • Web: mcas.pearsonsupport.com/ • Email: mcas@measuredprogress.org • Phone: 800-737-5103 • All policy questions (e.g., student participation, accommodations) • Web: www.doe.mass.edu/mcas • Email: mcas@doe.mass.edu • Phone: 781-338-3625

39. THANK YOU The Office of Student Assessment Services 781-338-3625 mcas@doe.mass.edu www.doe.mass.edu/mcas 75 Pleasant Street, Malden, MA 02148