Regents Scanning Pilot –CFN 201

1. Regents Scanning Pilot –CFN 201 • Amy Reynolds, Assessment Liaison • Damarine Reyes, Data/IT Liaison

2. Agenda Overview of new Regents data requirements & DOE’s plan to meet these requirements Scanning Process Key points for each step in the process Method Next Steps: Using Regents data to inform instructional decisions ATS reports overview Activity Discussion / Questions Support 2

3. Regents Exam Data • New York State Department of Education: • New Requirement: Beginning in June 2011, NYSED will require that ALL Districts submit item-level data for select Regents exams. This includes the following subjects: • English Comprehensive (not included for January pilot) • Integrated Algebra • Earth Science • Living Environment • US History & Government • Global Studies & Geography • Impact: Student responses need to be scanned to efficiently capture the required item-level data without impacting the turnaround time for results required by the schools. • Note: • Teachers must still score constructed response portions of exams • Other Regents exams and RCTs will still use the traditional scoring process

4. DOE Data Capture Solution • To leverage existing technology and processes currently in use in schools to capture item-level information & provide real-time results. This includes: • Answer Documents: Generated & printed at school-level • Scoring:Teachers to score Constructed Response items only • Scanning: Using schools’ attendance scanners, locally scan documents on-site • Reporting Results:Automated transfer from ATS to STARS • Benefits: • Increased automation of time-consuming manual processes • Less overall time required to complete scoring by teachers • No further data entry requirements of aggregate scores • Reduction in clerical scoring errors • eliminate manual scoring of MC items • adding totals from different sections • application of conversion tables • Districts/schools/teachers will be able to better utilize item-level results to help inform instruction, similar to Grade 3-8 exams

5. Scanning Process Overview STARS ATS Attendance Scanner 1. Schedule students for Regents Exams in STARS 2. Generate & Print answer documents for each scheduled student 3. Answer documents printed for each scheduled student Exam Day Scoring • Scoring committee marks Open-Ended items and bubbles in score on answer document. 4. Student uses pre-slugged answer document Attendance Scanner ATS STARS (and beyond) • ATS receives data, applies scoring key & conversion chart for scale score instantly. • ATS passes data nightly to STARS for Report Cards and Transcripts and for reporting to NYSED 6. Scan answer documents for upload to ATS Re-scanning (if needed)

6. STARS – (Review. Nothing has changed) • Schedule students for Regents exam in STARS • Who is responsible for STARS scheduling? • Program chair of each school usually handles STARS scheduling process • STARS scheduling process remains exactly the same as before • Major Consideration: Should I schedule one section or multiple sections for each Regents exam? • One section: • Shorter scheduling process • Use “Master Schedule” function • Multiple Sections: • Allows you to break students up into certain groups, assign each group to a certain room, print attendance rosters by exam room, and print exam invitation reports • Use “Master Schedule Builder” function For support with using STARS, contact your network’s Data/IT person

7. ATS • Print answer documents from ATS to attendance scanner • Who is responsible for ATS printing? • Pupil accounting secretary • Print answer documents up to a week before Regents week • There are three printing options: Regents Scanning Handbook / ATS Screenshots, page 20 • “Print scheduled students not previously printed”- these students have been scheduled in STARS • Print all answer sheets that have not been printed yet • “Print by student ID” • Print an answer sheet for any student one at a time. This option could be for another public school student taking the exam at your school • “Print by name” • Print an answer sheet for a non-public school student who is taking the exam at your school For support with using ATS, contact your network’s Data/IT person.

8. ATS Screenshots • Log in to ATS. On the screen that appears below type: REGT TAB to RGPR and press RIGHT/ CTRL/RED ENTER

9. ATS Screenshots

10. Exam Day • Use printed answer documents for test taking & scoring • Students use pre-slugged answer documents on the day of the exam to answer multiple choice questions • Students and Teachers need to use PENCILfor marking responses on answer documents • Responses are only to be marked in the new answer document • Proctors ensure that student answer documents are signed and bubbled in appropriately. • Scoring team bubbles in their scores for the open-ended questions on the teacher portion of the answer document • Teacher portion for all January Regents exams is page 2, except for the one-page Integrated Algebra form

11. Attendance scanner • Scan answer documents for upload to ATS • Do not scan until you’ve received notice from the DOE • Use your school’s attendance scanner • Scan the answer documents face up & bar code to the left • Scan both pages for 2-page answer documents • Batch page 1 documents separately from page 2 documents • Scanner will print a report of how many documents were scanned successfully; for those not scanned successfully the document will be reprinted with an error message on it • Rescan answer sheets if necessary • Audit trail captured of re-scans and changes to the item responses • Scanning must be complete by the end of the Regents window • For support with using the attendance scanner, contact your network’s Data/IT person or call the DIIT Helpdesk at (718) 935-5100

12. ATS • ATS processes scanned data and makes it available for use • ATS receives data, applies scoring key and conversion chart for scale score instantly • If both pages of a 2-page answer document are not scanned, scale score will not be displayed (Incomplete Record) • ATS passes data nightly to STARS for Reports Cards and Transcripts, and for reporting to NYSED, etc. 

13. Accessing Results • ATS Reports Available for Printing • Document Status Report • Ensure that answer documents have been properly scanned. • Item Report • Analyze the Regents results for individual students and inform decisions on how to move these particular students forward. • Review error report – multiples and omissions • Item Distribution Report • Analyze the Regents results for the student group as a whole in order to inform instructional decisions for these students as well as future students.

14. Document Status Report • Allows you to ensure that answer documents have been properly scanned. # of Answer Sheets Generated # of Answer Sheets Scanned Errors Warnings Raw and Scale Score Scan Date and Time Student Name and ID

15. Item Report • Allows you to analyze the Regents results for individual students and inform decisions on how to move these particular students forward. • Question number • Answer key • Student answer choices • Open-ended question scores Student ID # Student Name Raw and Scale score Dash = correct answer

16. Item Distribution Report • Allows you to analyze the Regents results for the student group as a whole in order to inform instructional decisions for these students as well as future students. For each answer choice, number and percentage of students that selected it Total number of responses for each question Question Number

17. Using data to inform instructional decisions ITEM REPORT • Questions this report could help you to answer • Which students scored the highest? • Which students scored the lowest? • Which students will have to retake this Regents exam? • Which questions did individual students have trouble with? • Are there any patterns in terms of what skills these questions focus on? • Next Steps • Plan instructional interventions for students who will need to retake the Regents • Support students who passed the Regents with further mastery of the areas that they had trouble with **Keep in mind that you will need to have a copy of the corresponding Regents exam to see the actual questions which correspond to the data.

18. Quality Assurance – Data Validation • 5% manual verification • Schools need to manually verify the scores of 5% of the answer documents for each exam have been scored properly. • Identify two people at school level perform the data verification • CFN Assessment Liaison will collect signed document from each pilot school • Document will indicate that the school has performed the 5% manual verification

19. Review - Scanning Process • Pre-Exam - Regents Scanning Implementation Checklist • STARS – Schedule students for Exams • Supplies: Paper, toner • Printing answer documents • Exam Day - Regents Scanning Implementation Checklist • Personnel available to printout extra answer sheets if necessary • Students use PENCILon the answer documents • Post-Exam - Regents Scanning Implementation Checklist • Complete score / open-ended questions. • Wait for official notice from DOE to begin Scanning • Accessing reports • Quality Assurance – Data Validation Documentation

20. Ongoing Support for Assessment Liaisons • Technical Support • Scheduling, ATS and STARS • Contact your Data/IT Liaison • Scanner and printer • Contact the DIIT helpdesk at (718) 935-5100 • Overall Process Support • Contact your Borough Assessment Implementation Director (AID) • Queens – Barbara Marcisak • Training Documents • Access documents from today’s training in ARIS Connect’sSummative Assessment Implementation Community: http://tinyurl.com/regents-scanning