Regents Scanning CFN Training Sessions

1. Regents ScanningCFN Training Sessions Updated April 28, 2011

2. Objectives The purpose of today’s session is to EDUCATE you on the new Regents scanning procedures, and EMPOWER you to turn-key the process at the school level. After the training, you will understand: • Why the DOE is implementing the new Regents scanning process and the benefits of the changes; • The overall scanning process to be performed by schools, and step-by-step procedures for each stage; • How schools should plan for Regents exam week, and special situations to prepare for; • What documents and resources are available for training and support, including how to use the demo IT environment to train schools; and • Best practices for turn-keying the scanning process at schools. 2

3. Agenda New SED Regents requirement & DOE scanning solution Regents scanning A–Z Process overview and timeline Step-by-step review Live demonstration Turn-keying to schools Lessons learned from the pilot program Demo testing environment Group activity: case study and discussion NY SED Special Study – June 2011 Regents Ongoing support and resources Q & A 3

4. SED Regents Requirement & DOE Solution Comprehensive English Earth Science Global Studies & Geography Integrated Algebra Living Environment U.S. History & Government SED Requirement: Beginning in June 2011, the SED will require all districts to collect and submit item-level data for the following Regents exams:* DOE Solution: Leverage existing technology and systems currently used by schools to capture and report item-level data. New scanning process highlights include: • Answer documents: student-specific documents will now be generated and printed at school level • Scoring: teachers mark Constructive Response items only • Data capture: scan answer docs with attendance scanners • Reporting results: real-time results available in ATS immediately after scanning; automated data transfer from ATS to STARS *Other Regents exams & RCTs will use the traditional administration and scoring process. 4

5. SED Regents Requirement & DOE Solution Time savings through automation of manual processes Less overall time required for scoring No further data entry requirements for aggregate scores Increased accuracy in reporting test results Elimination of manual scoring of MC questions Section scores totaled automatically Automatic application of conversion tables Improved instruction Item-level data available for use in informing instruction at student, teacher, school, and district levels Audit trail Answer document image-capture and line-item results will aid in special inquiries Benefits to schools under the new process: 5

6. Regents Scanning A-Z: Process Overview Overnight Data Transfer Pre-Planning STARS ATS & Scanner Schools receive training and begin preparations for Regents Scanning process. 2. Schedule students for Regents and produce exam invitations in STARS. 3. STARS data flows to ATS. Schools test scanner and generate and print pre-slugged answer docs in advance. Exam Week & Scoring Logistics Planning • Schools administer exams. Answer documents for walk-ins are printed on-demand. Scoring committee marks Constructive Response items and bubbles in scores on answer documents. 4. Schools conduct Regents week planning activities for logistics, including assignment of roles, and performing checklist items in the week before exams are administered. Overnight Data Transfer Attendance Scanner ATS STARS (& Beyond) • Schools scan answer docs for upload to ATS. • ATS instantly applies scoring key & conversion chart for scale score. Results available immediately. • ATS passes data nightly to STARS for report cards and transcripts, and for reporting to NYSED. 6A. Re-scan docs. 7A. Score verification Re-Scanning (As Needed) Manual Score Verification

7. Timeline of Key Steps 7

8. Step 1: STARS Main changes from the traditional process: No changes – the process is exactly the same. Main tasks: Register students for Regents exam, and generate exam invitations. Task owner: typically school Program Chair Best practices: Complete registration process for majority of student by May 27. Middle Schools: use proper exam codes in STARS for 8th graders. If appropriate, notify the high school where students will take exam. Only register students expected to take exams (though when in doubt, err in favor of registration). Use “Master Schedule” function for creating a single section, and “Master Schedule Builder” for multiple sections. Supporting materials:Regents Scanning Handbook, School Progress Reports available to Assessment Liaisons Support: CFN Data & IT Liaison 8

9. Steps 2–3: Printing Answer Docs in ATS Main changes from the traditional process: Answer documents are no longer included in the Regents exam booklet to be torn out. Pre-slugged student-specific answer documents are printed in advance, or generated on-demand at school level. No generic answer documents may be used. Answer docs are 1 page for Algebra & English; 2 for all others. Main tasks: Test scanner to ensure readiness, adequate supplies of toner and paper. Print answer docs (for pre-registered students). Task owner: typically Pupil Accounting Secretary Supporting materials:Regents Scanning Handbook, School Progress Reports Support: DIIT Help Desk, 718-935-5100 9

10. Traditional & New Answer Documents Regents Scanning Traditional Process Individualized bar code Student MC answers CR teachers’ grades Alt. language & accommodations Pre-populated student info Scorers’ info 10

11. Steps 2–3: Printing Answer Docs in ATS There are five options for printing answer docs: “Not previously printed” (Batch printing for registered students) Prints batch of answer documents that have not been printed “Print by student ID” (For DOE walk-in student) Prints an answer sheet for any student one at a time. This option may be used for another DOE student taking the exam at your school. (Use ATS SBIO screen to look up student ID.) “Print by name (non-pub)” (For non-DOE walk-in student) Prints an answer sheet for a non-DOE student who is taking the exam at your school. “Charter school option” (For schools not using STARS) Batch printing option for schools who are unable to register students in STARS. “Reprint answer documents” Reprint previously printed docs by student ID or answer document page number. 11

12. Steps 2–3: Printing Answer Docs in ATS • Use RGPR screen for printing options 1, 2, and 3. Sorting Options 12

13. Steps 2–3: Printing Answer Docs in ATS Best practices for printing answer documents Answer documents must be printed on white paper only. Use the same stock paper as is normally used for attendance forms. Using colored paper or heavy stock may result in scanning errors. Print answer docs for pre-registered students by June 10. After printing batches of answer documents, a “test page” will print. Scan the test page to confirm ink quality for batch. When printing, you may sort print jobs by teacher, section, or room number. Ensure that adequate supplies of paper and toner are on-hand. 13

14. Step 4: Logistics Planning for Regents Week Main changes from the traditional process: Educate relevant staff members on new procedures and responsibilities for Regents scanning, and organize staff roles. Main tasks: Distribute test rosters and page one of answer docs to appropriate staff in advance. Train exam proctors on new procedures and policies. Ensure adequate supply of toner, paper, pens/pencils. Establish process and task owners for dealing with walk-ins. (See slide “Instructions for Walk-Ins”) Review Implementation Checklist Best practices: Roles and responsibilities are clear and communicated to all relevant staff before Regents Week. Perform all checklist items. Supporting materials:Regents Scanning – Implementation Checklist Support: CFN Assessment Liaison 14

15. Step 5: Exam Week & Scoring Main changes from the traditional process: Answer docs for “walk-ins” printed on-demand on test day. Students use pre-slugged answer documents; pencil for MC, pen for essays and DBQ questions. Teachers mark Constructive Response items only using pencil. Main tasks: Generate answer documents for “walk-in” test-takers. (See next slide.) Proctors administer exams according to best practices and emphasize new procedures (e.g. completely fill in bubbles, no stray marks, sign student declaration when finished). Best practices: Good preparation will lead to smooth exam administration Support:CFN Assessment Liaison 15

16. Step 5: Instructions for “Walk-Ins” Schools should make preparations and assign roles to handle walk-in students during Regents week. • Inform appropriate staff how to direct walk-ins upon arrival, and appoint staff to print answer docs using ATS. • Print answer documents by following the below instructions: • Detailed instructions on handling walk-ins is included in the Handbook and FAQ. *If Student ID is unknown, use ATS SBIO menu to look it up. 16

17. Step 6 & 6A: Scanning Answer Documents Main changes from the traditional process: Scanning will capture item-level data and images of answer documents (and any ancillary re-scans) for audit trail. Immediate results available in ATS; conversion chart automatically applied for instant score. Main tasks: After notification by DOE, scan docs using attendance scanner. Docs with errors in reading bar codes will be reprinted with an error message. Docs with bubble read errors will be indicated on DFS confirmation sheet. Re-scan affected documents. Review ATS reports for “Warnings” observed in Teacher section. Task owner: staff members who process attendance scanning Best practices: Scanning tips available in Regents Scanning Handbook Ensure that scanner has been cleaned in advance. Support: DIIT Help Desk, 718-935-5100 17

18. Step 6 & 6A: Scanning Tips Ensure the scanner is clean; a dirty scanner may result in inaccurate results. See Tips for Scanning appendix for instructions on cleaning. When scanning 2-page documents, it is recommended to batch all page 1 documents and scan separately from page 2 documents. Schools may scan the first page of two-page documents immediately, before scoring is complete for page two. If only one page of a 2-page document is scanned, it will be shown in ATS as an “Incomplete Scan” until the second page is processed. Use the ATS Status Report to review warnings. After scanning, a confirmation page will print indicating pages with scanning errors. The scanner will reprint pages for which the bar code could not be read. File the scanned batches together with the DFS confirmation page. This will assist in retrieving papers as needed in the future. Daily cutoff for overnight data transfer to STARS is 9pm on weekdays and 4pm on weekends. 18

19. Step 7: ATS Reports & Results ATS reports available immediately after scanning: Use Document Status Report to ensure that all answer docs have been properly scanned, and to flag warnings. Use Item Report to see students’ scores and inform promotion status of individual students. Review report for “warnings/multiples/omits” by using “Include only W/M/O” option. Use Item Distribution Report to inform decisions on present and future instruction. Detailed instructions on how to access the ATS reports are available in the Regents Scanning Handbook. Given the length and format of the ATS reports, they are best viewed when printed. You may not see all the data in the ATS screen. Schools may download the ATS reports into Excel for further analysis. 19

20. Step 7: ATS Reports (REDS - Status Report) Report uses: Displays score, accommodations, warnings, for each student. Shows summary of documents scanned, outstanding, and warnings. Use report to track outstanding unscanned and incompletely scanned document. Scores Accom. Scan-specific info Document statuses 20

21. Step 7: ATS Reports (REDS – Item Report) Report uses: Displays line-item data for each student. Shows specific questions to review (warnings). Use to flag Teacher section questions to investigate (e.g. omits, multiples). STUDENT SECTION TEACH SEC Correct answers CR raw scores Multiple answers Incorrect answers 21

22. Step 7: ATS Reports (RADI – Item Dist. Report) Report uses: Shows school-level aggregated data for each question. Use data to flag specific subject matter, question types, etc. for instruction. ‘*’ denotes correct answer 22

23. Step 7A: Re-Scoring & Score Verification Main tasks: AS OF MAY 24, 2011 – THERE IS NO RESCORING OF ANY REGENTS EXAM INCLUDING ALL MATH AND SCIENCE EXAMS with scaled scores of 60-64. Verification of scanned results for approximately 5% of answer docs. Complete Manual Scoring Data Validation Form. Consult the FAQ as Regents week approaches for updates on scoring procedures. Best practices: Two people at school level perform data verification Supporting materials: Regents Scanning Handbook Support:CFN Assessment Liaison 23

24. Step 8: Accessing Results in STARS Main changes from the traditional process: Data flows to STARS automatically; no manual data entry required. Results will be automatically applied to transcripts and report cards from STARS. Main tasks: Provided scanning is completed by 9pm – on day after answer documents are scanned, schools may generate a STARS reports for all registered students who sat for Regents, regardless of test location. Best practices: Any future score changes (through re-scanning, DOE answer key alteration, etc.) will flow through to STARS automatically. Support:CFN Data & IT Liaison 24

25. Live Demonstration

26. Turn-Keying: Demo Environment The demo environment is available from May 2-31. After May 31, the demo period will end and all demo data will be removed. Detailed instructions on how to access and use the demo environment are available through a Demo Script. You will use your existing ATS username and password. Demo scope: We highly encourage schools to take advantage of the demo environment to gain familiarity with the new processes and to help ensure smooth Regents administration. A demo environment is available for training purposes. Schools are encouraged to use this to prepare for Regents Week and gain an understanding of the required processes. STARS (Scheduling) ATS (Printing) ATS (Scanning & Reporting) STARS (Report Card &Transcripts) DEMO 26

27. Turn-Keying: Lessons Learned from Pilot Schools with advanced preparation performed well during January Regents Preparations include: communication of new processes and policies, defined staff roles, on-time scheduling/printing of answer documents, prep for “walk-in” students, etc. Many schools needed additional support and training around the technical systems. Many schools did not utilize ATS reports (Document Status Report, Item Report, Item Distribution Report) for scanning validation and instructional benefits. The demo environment is intended to assist with this for the June implementation. Scanning technology was validated through the pilot. DOE conducted a pilot program in January to administer the new process at select schools. The most valuable lessons learned from the pilot are: 27

28. Turn-Keying: Case Study & Discussion Review of Regents Scanning Implementation Checklist. Case Study exercise. Discussion: in groups of 3-5, discuss the following questions: How do you plan to effectively turn-key this new initiative to schools? Who will you include in the school-level training? What challenges do you anticipate? What items need to be emphasized or modified for different size schools within your network? How will you support your schools in using the new item-level data that is produced from this process? What questions do you have about the process itself or its implementation? (Record for Q&A) 28

29. Turn-Keying: Roles & Responsibilities 29

30. Turn-Keying: Network & School Checklist *Refer to Implementation Checklist for a complete list of tasks to be performed by schools. 30

31. SED Special Study – June 2011 Regents Affected Regents exams: Comprehensive English Algebra 2/Trigonometry Affected schools: See NY SED Special Study for June 2011 Regents – Affected Schools handout Process highlights: The special study process will supersede any NYC DOE administrative process – including Regents Scanning for the Comprehensive English exam. CFNs will receive instructions on how affected schools must administer these exams. The NYS Education Department is conducting a state-wide special study which will affect select schools for two June 2011 Regents. 31

32. On-Going Support & Resources Documentation and updates: ARIS: https://www.arisnyc.org/connect/node/395478/community Technical support (STARS & ATS): CFN Data & IT Liaison Scanner and printer: DIIT Help Desk, 718-935-5100 Overall process support: Borough Assessment Implementation Directors (BAID) Regents week emergency inquiries: Patrick Purcell (ppurcell@schools.nyc.gov; 212-374-5553) Marc Bush (mbush3@schools.nyc.gov; 201-755-2519) Kirby Fields (kfields6@schools.nyc.gov; 646-379-8422) The following resources are available to support implementation and execution of Regents scanning. *Please note that guidance from DOE on unresolved items will be issued on a rolling basis. 32