Presentation Transcript

1. Regents Distributed Scoring SchoolTraining Preparing the January 2013 Regents for Distribution to Scoring Sites Office of Assessment Operations December 2012
2. Agenda Background School Preparation Preparation for Day and Per Session non-Electronic Scoring Preparation for Electronic Scoring Scoring Scanning
3. Background In October 2011, the Board of Regents voted to prohibit teachers from scoring their own students’ state assessments, beginning in SY 2012-13. NYC has moved to strengthen this requirement for Regents exams by shifting to a distributed scoringsystem(DSS), where teachers will not score the exams of any student that attend their school. Requirement: Teachers will not score the exams of students attending their school. Preference: Each student’s exam is marked by staff from at least two different schools. To prepare for this shift, NYC implemented distributed scoring in select schools and for select exams during 2012 administrations.
4. January 2013: Framework
5. January 2013: Scoring Format & Models Day-Scoring: Only schools administering these exams in January will be required to contribute staff to meet the scoring requirements. Per-Session: Staff from schools citywide will be eligible to apply for per session scoring positions.
6. January 2013: Exams & Scoring Timeline Scoring staff are expected to be at the scoring site by 8:30 a.m. each day and will be expected to stay for the full length of the teacher work day (i.e., 6 hours and 20 minutes plus extended time). Staff serving in site management roles should be available before and after the scoring day. The exact days and times for per session scoring of each exam have yet to be determined. Scoring periods for exams scored in per session will be communicated later this fall.
7. Key Information for January 2013 All exams (including translated versions) will be scored via the DSS model. All answer documents will be two (2) pages except Physics*. Schools print ATS information labels and affix them to answer booklets for all exams except Earth Science and Chemistry or handwrite Name and Student ID on the booklet. Schools administering the Earth Science and/or Chemistry Regents will participate in the electronic scoring process, which have distinct packaging directions. Test booklets for Earth Science and Chemistry must be labeled with pre-printed CTB labels. Schools will print pre-populated rating sheets for the English, Global and US History Regents. Schools will RETAIN and scan all page 1 answer documents except Physics* by 12 noon the day following administration.. Only page 2 will be scored and scanned at central scoring sites for all exams except Earth Science and Chemistry**. Each school’s scorers and exams are sent to different scoring sites. Schools will provide staff to score the Comprehensive English, Integrated Algebra and Living Environment Regents exams during the day. All other exams will be scored as a per session activity. Schools that host a YABC should package those exams with their own. Read DAPS Regents Memo #4 at https://portal.nycenet.edu/Accountability/Assessment/YearlyTesting/MemorandaProcedures/&AuthResend1908BC2350124b5095AB75012FA405BA . *See section on Physics. **See section on electronic scoring

8. School Preparation

9. School Preparation Required Prior to Exam
10. Important Administration Procedures Program students in STARS by room; each room is a section. Distribute exam invitations produced in STARS. NEW! Print Student Information labels and affix to the appropriate area of Regents test booklets and essay booklets of all math, English, history, Living Environment and Physics Regents. (More information to follow.) Earth Science and Chemistry test booklets must be labeled with bar-coded labels provided by CTB. Track attendance. Ensure students use appropriate writing utensil. Ensure students sign declaration. Place student answer documents inside student test booklets. Collect test booklets and answer documents.
11. Pen/Pencil Policy *Students are permitted to sign the declaration in pen.
12. Track Attendance Proctors should track student attendance using the section attendance roster. See DAPS Regents Memo #4 attachment for sample. Schools may use an equivalent school generated document. Indicate the following on the section attendance roster: The testing school’s DBN and the name of the exam on the top of the roster, including each students’ name, NYC Student ID number. Attendance (absent or present). If the student wrote the exam in a language other than English . If the exam was invalidated i.e. missed labs, cheating, etc.
13. Section Attendance Rosters

14. Preparations for Day and Per Session (Non-Electronic) Scoring

    English, US and Global History, Integrated Algebra, Geometry, Algebra2/Trigonometry,Living Environment and Physics
15. Label Test Booklets Schools will run student labels and affix them to Regents test and essay booklets in the upper left hand corner of all exams EXCEPT Earth Science and Chemistry (see directions in packing for electronic scoring). Do not put labels on the answer documents. Use Avery 5160 (30 labels per sheet) or equivalent. Labels for all Regents except Chemistry and Earth Science Only English, US and Global History have an essay booklet. Sample Sample
16. Collecting Student Answer Documents Place student answer documents and, if appropriate, essay booklets inside the Regents test booklet. To ensure that answer documents and test booklets remain appropriately matched, proctors should ask students to place their answer document (and essay booklet) inside their Regents test booklet upon completion of the exam. Essay booklets are used in the English, US and Global History Regents.
17. Collecting Student Answer Documents When collecting a Regents exam every proctor must: Check that the student has signed the declaration. Ensure that an answer document, and if appropriate, an essay booklet, are received from every student and inserted into the Regents test booklet. Alphabetize booklets and check against student roster.   Declaration must be signed Essay booklets are for English, Global and US History only. This step N/A for all other Regents exams.
18. Packaging Test Materials for Scoring Page 1 answer documents are removed from the test booklet. Page 2 answer documents with the ELL and Special Education accommodations completed, as appropriate, are inserted into the correct students’ test booklets and packaged for shipment to the scoring sites. Page 2 documents for Earth Science and Chemistry are retained and will be scanned at the school. The performance score for Earth Science must be bubbled in on page 2 before scanning. Essay booklets for the English, US and Global History Regents are also included inside the test booklet. Retain and scan ALLpage 1 answer documents by 12 noon the day following administration. Retain test booklets and scan the page 2 answer documents of absent students and students with invalidated tests.
19. Packing Sections When packing Regents exams for Day (non-electronic scoring): Complete any IEP/504 accommodations. Bubble in alternative language if administered to a student. Ensure that the page 2 answer document and, if appropriate, an essay booklet, are inserted into the Regents test booklet*. Bubble IEP/504 and/or ELL accommodations, and alternate language versions administered to students. Essay booklets are for English, Global and US History only. *See special directions for Physics.
20. Packaging Sections A section is defined as group of students who write the test in the same room. Each section will be packaged in its own polybag, which should include the following in the order listed: A completed Section Header (see DAPS Memo #4 Attachment). A Section Attendance Roster (see DAPS Memo #4 Attachment). (English, US and Global History only) school produced preslugged Essay Rating sheet that corresponds to the section list. Test booklets – except Physics Page 2 answer document, and (English, US and Global History only) Essay booklet are placed inside the test booklet, for all tested students in that section. Do not send Page 1 answer documents to the scoring site. Do not send test booklets for students who were absent or had invalidated tests. The exams of graduating seniors are not separated for January.
21. Section List Be sure that: All information on the Section List form is completed by the testing school including day and cell phone numbers. Each student test booklet includes the corresponding page 2 answer document. The count of answer documents and test booklets on the Section List match the number of answer documents and test booklets in the polybag. All answer documents and test booklets for absent students and students with invalidated tests are retained at the testing school for scanning.
22. Section List
23. Submission of Scoring Materials: Non-electronicPacking the Polybag STEP 3: English, US and Global History will also have a school produced preslugged Essay Rating Sheet included in each polybag. Skip this for math and science exams STEP 4: The Section Attendance Roster is placed on top of the class pack. Annotate with notes about absentees, translations, etc. STEP 1: Page 2 answer document is placed inside the same student’s Regents booklet STEP 2: Alphabetize each class. English, US and Global History: Insert Page 2 answer document and essay booklet Math and Living Environment exams: Insert Page 2 answer document only Physics: Answer document and answer book Do not send Page 1 answer documents or booklets or documents of absent students or invalidated tests. STEP 6: Each section is placed in a large polybag. Do not mix sections in the same polybag. STEP 5: The Section List is placed on top of the Roster
24. Physics Packaging The Physics Regents is a secure exam that requires special handling. Each student answer booklet is numbered and included in the same envelope as the exam booklet. Students are to write their answers to all of the exam questions, multiple-choice and open-ended, directly in the answer booklet. No answers are recorded by students on the one page school generated answer document. At the conclusion of the exam students place their exam booklets, reference tables and scrap paper in an SED supplied secure return envelope and seal according to SED directions. After bubbling any accommodations the school generated answer document is placed inside the numbered SED Physics answer booklet. Do not scan the school generated answer document. Pack answer books for scoring using directions for all other non-electronic exams. Bundle together the envelopes containing ALL used and unused exam booklets and pack with the answer documents for shipment to the scoring site. All Physics testing materials will be returned to SED directly from the scoring site.
25. Physics Answer Document The Physics school generated scannable answer document is one page and is NOT distributed to students.
26. Packaging Boxes Each exam subject is packaged in its own box. If there is more than one box for an exam subject, indicate the box number out of the total shipment (e.g. Integrated Algebra – Box 1 of 2, 2 of 2, etc.) on the outside of the box. Every box should include the following in the order listed: Packing List. Polybags with exams (equal to the number of testing sections). Be sure that: All information on the Packing List to be completed by the testing school is completed. The number of sections listed matches the number of polybags included in the box.
27. Packing List Please complete all sections of the form
28. Packing the Box: All exams except Earth Science and Chemistry Each section is packed in a large polybag. Do not mix sections in the same polybag. One Packing List is included in every box. ALL sections of an exam are put into one or more boxes DO NOT MIX subjects in boxes! Complete information on each side of the box
29. Exam Pick-Up Schedule Deluxe Courier will pick-up non-electronic scoring test materials for delivery to the scoring site Assessment Distribution Services (ADS) will pick up Chemistry and Earth Science test materials for electronic scoring. A school staff member must be at the school and available during the delivery window to provide the test materials to the courier. * Electronic scoring - Assessment Distribution Services (ADS) will pick up Chemistry and Earth Science test materials.
30. Exams Not Ready Exams not included in the initial pick up will have to be hand delivered to the BAO with an explanation as to why they were not included in the scheduled pick up i.e. student had extended time, student was in a conflict room, etc.

31. Electronic Scoring

    Chemistry and Earth Science
32. Preparing for Electronic Scoring Earth Science and Chemistry will be scored electronically. Electronic scoring does not affect the administration of the exams. Affix all pre-slugged CTB bar-coded labels to all test booklets. Page 1 and 2 answer documents for Earth Science and Chemistry will be scanned at the school. Complete all Accommodations and Alternate Language Information on Page 2. Ensure that Earth Science performance scores are bubbled on the page 2 answer document. Do not score any constructed response questions at the school. Scan Earth Science and Chemistry answer document pages 1 and 2 at your school.
33. Packaging Exams for Electronic Scoring Detailed directions for packaging exams for electronic scoring can also be found in DAPS Regents Administration Memo #4 at http://schools.nyc.gov/Accountability/resources/testing/MemorandaProcedures.htm Ensure that a bar-coded label with the student information is attached to every test booklet. Remove any answer documents from inside the test booklet and scan them at your school.
34. Label Earth Science and Chemistry Booklets Chemistry and Earth Science Test Booklets- affix a bar-coded label provided by CTB is placed in the upper LEFT hand corner of the test booklet. Generic labels will provided for students who do not have a pre-slugged label i.e. walk-ins Pre-slugged CTB bar-coded Chemistry and Earth Science labels Sample
35. Prepare Earth Science and Chemistry for Packaging Prior to packaging collected exams for delivery to CTB schools should take the following steps: Remove answer documents for scanning at the school Ensure the following items are not in the test booklets: scratch paper (retain in school for 1 year) tape of any kind Post-it™ Notes Staples Pins paper clips
36. Room Header Form- Documenting how many completed test/essay booklets are returned to CTB. Complete a Room Header Form for each assessment subject and booklet administered within his or her class. It is essential that the Room Header Forms be returned filled in completely and accurately. It is recommended that the school make a copy of each Room Header Form. Make sure to complete the following steps: Verify that the information on each Room Header Form is correct by checking the sheet against the actual number of student test booklets returned by the Room Proctor. Make any necessary adjustments required to the total number of test booklets listed on the Room Header Form.
37. Stack Cover Card- Documenting Exam Stacks per Section Complete one Stack Cover Card for each section and place it on top of the stack of test/essay booklets and Room Header Form(s). Check the upper right corner of the stack cover card for complete DBN, section, room, and exam subject (content) information. Validate the number of assessments returned for scanning matches the information on the Room Header Form. Place the first stack cover card on top of the Room Header Form. Secure the test books with paper bands as illustrated on the stack cover card. A room may have one room header sheet but result in several stacks. Fill out an additional Stack Cover Cards for each stack of X exams. Record the total number of stacks for that room in the “Stack __ of ___” fields.
38. Band the Test Booklets into Stacks After removing all materials from test booklets books must be bundled into stacks. Use the paper bands sent by CTB to bundle the used test booklet together after completing the Stack Cover Card and the Room Header Form. A section may have one or more stacks of exams. Paper Band Used test booklets
39. Completing the Box Packing List The Test Coordinator completes the Box Packing List by listing the total number of stacks being returned to CTB/McGraw-Hill from each room. IMPORTANT: Document the box label number from the return shipping label in the space provided. List the test coordinator name and phone number. Enter the total number of rooms returned in the box. List the room, subject, and total number of stacks for each room. Make a copy of the Box Packing List for the school’s records.
40. Packing Sections for Electronic Scoring Each section should be packaged separately, and include the following in the order listed: A completed Stack Cover Card. A completed Room Header Form. Test booklets, secured with a paper band, for all tested students in that section. Be sure that: all information on the Stack Cover Card and the Room Header Form is complete each student test booklet has the pre-coded barcode label placed in the top right corner of the test booklet the count of test booklets on the Room Header Form matches the number of test booklets in the polybags all page 1 answer documents are scanned and retained in the school all page 2 answer documents with accommodations performance scores and alternative languages bubbled are scanned and retained in the school all answer documents and booklets for absent students and students with invalidated tests are retained at the school for scanning
41. Packing Boxes for Electronic Scoring Each exam subject (Earth Science or Chemistry) is packaged in its own box. If there is more than one box for an exam subject, indicate the box number out of the total shipment (e.g., Chemistry– Box 1 of 2, 2 of 2, etc.) on the outside of the box. Every box should include the following in the order listed: a subject-specific Box Packing List placed directly on top of the stacks of test booklets. Be sure that: all information on the Box Packing List to be completed by the testing school is completed the number of sections listed matches the number of stacks included in the box
42. Packing Process Stack cover card for each section goes on top of Room Header form – One per Section. Please note: this form should go under the paper band Box packing list goes in the box on top of the stacks– one per box Room Header Form goes on the top stack of each section – one per section Bundled stacks of each test section’s booklets go into box for CTB (repeat for each section) test booklets Section 1 Section 1 To CTB Shipping label provided by CTB goes on top cover of the box test booklets Section 1

43. Scoring

44. Scoring Schedule Day Scoring Scoring period for English and Integrated Algebra will be during the school day and for three days or until scoring of all included exams is complete. Living Environment will be scored during the day for two days or until scoring of all included exams is complete. Scoring staff are expected to be at the scoring site by 8:30 a.m. each day and will be expected to stay for the full length of the teacher work day (i.e., 6 hours and 20 minutes plus extended time). Staff serving in site management roles should be available before and after the scoring day. Per Session Scoring Regents to be scored per session are US and Global History, Chemistry, Earth Science, Physics, Geometry and Algebra 2/Trigonometry Scoring period for per session is scheduled to begin on Friday, January 25 and end on Sunday, January 27. Exams may begin and end scoring at different days and times. Exact days and hours will be sent to all selected scorers.
45. Scorer Training Scorers will be trained on the first day of scoring by the subject area Scoring Content Leaders. Scorers are required to attend the entire training session. Day scorers must bring a copy of the Rating Guide released on the day of the exam for the subject area to be scored. TCs can help by ensuring there is a plan for generating scoring guides. Use of electronic copies of the Rating guides will not be allowed at the scoring sites to protect the confidentiality of student responses.
46. Scoring Scoring committees will be established by the Scoring Content Leaders, in collaboration with the Site Supervisor, and structured according to SED policy. No teacher will be allowed to grade an exam written by a student at his/her school. Scorers are expected to: Score accurately. Complete a sufficient number of test books to finish scoring within the allotted time frame. Maintain student confidentiality and objectivity while scoring Maintain security of test materials. Scorers will sign their school’s Examination Scoring Certificate at the scoring site. The scoring certificate will be returned with each school’s test materials for return to SED in the school’s Regents boxes.

47. Scanning

48. Scanning - Schools Schools will scan: All page 1 answer documents by 12 noon the day following administration. All page 2 answer documents for Earth Science, Chemistry and exams of students who are absent or have invalid tests. All other page 2 answer documents will be scanned at the scoring sites. Schools will be responsible for data capture errors (RGTU) on all documents scanned at their school. Schools will not have access to RGTU for page 2 answer documents scanned at scoring sites except to change accommodations or language bubbles. Scoring sites will not have access to RGTU for answer documents scanned at home schools.
49. Scanning – Scoring Site All page 2 answer documents will be scanned at the scoring site. In some cases, scanners may not accurately capture the information that a teacher provided on page 2 of an answer document.  In those cases, corrections will be made at the scoring site using the RGTU function in ATS to manually correct the errant data. Please note that scoring sites scan in batches and all of a school’s answer documents may not be scanned at the same time. If schools do not see all of their data initially it is because scanning has not been completed. Scores should be available by the end of the scoring window.
50. Access to Data Schools will be able to access a number of reports in ATS, including REDS: Status Report Item Report Item Distribution Report. Scores for non-electronic exams will be available in ATS after both the page 1 and page 2 answer documents have been scanned. Scores for electronically scored exams will be available after page1 and page 2 answer documents have been scanned at the school and the scorers grade each exam electronically.  More details on scanning will be available shortly.
51. The Finality of Examination Scores Per SED guidelines, when the teacher scoring committee completes the scoring process, test scores must be considered final and will be entered onto students’ permanent records via the scanning process.   Scoring committees, once they have rated an exam, are not permitted to re-score them, regardless of the final score. Specifically, exams receiving a final score of 60-64 or 50-54 may not be re-scored. Principals and other administrative staff in a school or district do not have the authority to set aside the scores arrived at by the teacher scoring committee and rescore student examination papers or to change any scores assigned through the procedures outlined in the scoring materials provided by SED.
52. More Information For questions contact your network assessment liaison. See the NYC Distributed Scoring of New York State January 2013 Regents Exams memo for detailed instructions regarding preparations for exam administration.