2012-13 Data Verification Part I Spring 2013

Goal The goal of this session is discuss the New York City accountability reports, data used for reports, and data verification procedures as they relate to Accountability Reports.

Agenda Data Verification Overview Network Responsibilities/ Support Escalation Timelines, Tools and Resources EMS Accountability and Reports

What is Data Verification? Data Verification is a process by which data is reviewed for accuracy and inconsistencies after data migration.It helps determine whether the data was accurately translated when data is transported from one source to another, is complete, and supports processes in the new system. The verification of data is large part of data quality and reliability. The purpose of the DOE Accountability Data Verification process is to establish the reliability of the data utilized by the NYCDOE and NYSED accountability systems. Data accuracy will ensure accountability determinations and reporting are accurate to best of the school’s knowledge.

Why is data verification important? NYCDOE schools are accountable for the data entered into DOE source systemsand responsible for their accuracy. School staff are asked to work collaboratively in this process when reviewing and correcting their school’s data. Schools are given the opportunity to review and very their data before it is used for reporting purposes. The objectives of the NYCDOE Accountability Data Verification is assist schools in the process of reviewing and correcting data before it is used for reporting. DOE provides users with: friendly data files that can be easily manipulated to review data sufficient time for schools to enter, review, clean and correct accountability data resources and training that can be used to cross reference, check, and change data

Data and Accountability Reports? The data within verification reports are used to inform two accountability tools that evaluate New York City Public Schools. Reports are issued annually based on prior year data. The NYC Progress Report: A locally designed accountability report used to evaluate schools on their student progress, performance and overall school environment; and The State Accountability Report: A New York State published report used to evaluate schools on their student participation, performance, and progress on state exams and graduation. The report contains data used to determine Performance Indices (PIs) and Adequate Yearly Progress (AYP) determinations which, in turn, are used to assist in identifying a school’s State designation status.

Network Responsibilities To support schools, networks can prepare themselves supporting their schools during the verification process.

Accountability Escalation Support Path NYSED Accountability For nySTART accounts and questions, schools and CFN staff may email nystartsupport@schools.nyc.gov For ESEA questions Clusters and Central staff should contact SFESupport@schools.nyc.gov

Accountability Escalation Support Path Progress Report Accountability For PR feedback and appeals questions, schools and CFN staff may email PR_Support@schools.nyc.gov

Accountability Escalation Support Path Assessment Click here for a list of 2012-13 Borough AIDS

Timeline/ Important Dates The timeliness of collected, entered and reviewed can improve data accuracy with source systems and verification reports. The chart below lists milestone dates for report release, data entry and collection. Dates written are estimated and subject to change. A complete calendar will be published in the Data Verification Guide and on the Data Verification wiki. Additional Information: https://wiki.nycenet.edu/display/DataVerification/Verification+Calendar * Waiting approval from ATS

Data Applications and Tools The DOE student management applications below to are used to enter and review student data accountability reports.

Data Refresh CycleDOE applications migrate data between the source systems in real time for some reports/functionality and for others on a specific schedule. Data populated in the DOE DV and SED L2RPT reports are refreshed on a weekly cycle. ATS 2012-13 EOY Calendar

ResourcesSchool Documents and Guides Guides are posted on the SFE intranet page: https://portal.nycenet.edu/Accountability/SchoolPerformance/nclb/data/ Data Verification Guide Data Cleaning Guide REMS/RHSV

Resources

Elementary and Middle School Accountability Elementary/ Middle School Accountability – Progress Report Progress Report Workbook Elementary/ Middle School Accountability – NCLB/ESEA EMS Data Verification Report – REMS L2RPT EMS Accountability Verification Report – EMS AVR EMS Assessments Test Takers Report – RETT L2RPT Tested Not Tested ELA Exempt ELL Parent Choice Option - ELPC Entering Months Enrolled - UYRE Exam Errors - XAMM

What is the Progress Report? Additional Information: http://schools.nyc.gov/Accountability/tools/report/default.htm The Progress Report is a large part of NYCDOE’s efforts to set expectations for schools and to promote school empowerment and accountability. Reports are issued annually shortly after the start of the school year based on prior year data. Schools review data used for the report via a workbook distributed to principals at the start of the school year Schools are assigned letter grades A through F based on a years worth of performance in five main areas of measurement. Scores are based on comparing results from one school to a peer group of up to 40 schools with the most similar population and to all schools citywide, student’s year to year progress and additional credit based on exemplary progress with high-need student groups. .

NCLB/ ESEA Accountability College and Career Readiness under ESEA – changes made to NYSED Accountability: Under ESEA, NYSED implemented the following changes: : Additional Information: http://schools.nyc.gov/Accountability/tools/accountability/default.htm

What is EMS Accountability? Progress Report Reviewing the data: Biographical data is current as of June 30th of the reporting year. Corrections made to these data should be prior to the end of the SY. Course outcome data is collected late June and in some instances again in August. Additional Information: https://wiki.nycenet.edu/display/DataVerification/Progress+Report Reporting: Student progress, Student Performance and Closing the Achievement Gap where schools achieve exemplary outcomes in moving special student populations. In addition to regular biographical and demographic data schools should pay close attention to the following: Enrollment:To be included in the school’s Progress Report a student must be on a school’s DOE audited register for the current reporting year. For core course passing rate and Grade 8 students earning high school credit, a student must be continuously enrolled from October 31 until June 30 of the reporting year. Schools should review and correct when necessary admission and discharge codes /dates in ATS. Assessment: Students should have taken the appropriate state ELA or Math assessments within the current year (to measure student progress a valid assessment is needed within the current and previous SY.) Schools should review the “expected assessments” for their active students using the RETT report for Grades 3-8. Grade 8 students earning high school credit must have a valid Regents score when applicable. For regents assessments, students should be assigned to correct exam code in STARS. Course Data: PR reviews course data for core subjects: ELA, Math, Science, Social Studies. Students in grade levels 6, 7, and 8 must have course data in STARS. Schools should ensure students are scheduled with the correct course code and that a final grade is recorded on the transcript. Attendance: The attendance rate is calculated by adding together the total number of days attended by all students and dividing to by the total number of days on register for all students. The attendance rate used for the PR can be reviewed using the RGAR report in ATS.

How to Review PR Accountability Data? Progress Report Preliminary Workbook When is the report available?: Late August About the Report: Principal’s are sent a preliminary PDF file of their school’s unofficial Progress Report and a excel workbook with the data that was used to calculate their Progress Report. Data Displayed: The workbook allows schools to verify to their best of their knowledge the data used in the Progress Report is accurate. Schools will have the opportunity of correcting particular data elements (e.g. academic records) where needed. Each workbook contains a tab for each PR metric in addition to the Summary of Metric Values, a list of all Worksheets, and a column key.

NCLB/ ESEA Accountability Additional Information: http://schools.nyc.gov/Accountability/tools/accountability/default.htm In 2005–06 New York State established an accountability system in accordance with the federal No Child Left Behind (NCLB) Act. This system requires the State to collect assessment and graduation-rate data to determine progress of districts and schools in these areas towards predefined goals. In May 2012 the United States Department of Education (USDE) approved New York’s Elementary and Secondary Education Act (ESEA) waiver. This waiver provides New York with increased flexibility with the implemented NCLB accountability system to help close the achievement gap and assist students in becoming college and career ready. As part of NCLB/ESEA, New York State is required to hold districts and schools accountable for their students, in part, through a process for evaluating participation and performance of ten accountability students sub-groups in specified measures and determining if Adequate Yearly Progress (AYP) is made in those measures. AYP indicates satisfactory progress by a district or school toward the goal of proficiency for all students. Districts and schools that meet predefined participation and performance criteria on New York State’s accountability measures are considered to be making AYP.

What is EMS Accountability? – NCLB/ESEA Additional Information: https://wiki.nycenet.edu/display/DataVerification/SED+Elementary+Middle+School+Verification Reporting: Schools report data on measures of student Participation and Performance in 1) Grades 3 – 8 English Language Arts, 2) Grades 3 – 8 mathematics and 3) Grades 4 and 8 science. Participation Definition: EMS Participation rate students are those who were enrolled for the entire test administration period, even if they were not continuously enrolled in the school from BEDS (first Wed of October) day until the test administration period.* Students who enter or leave a school during the test administration period are not considered as participation rate students unless the school provides valid scores for the students. To meet the participation requirement in ELA and Mathematics 95% or more of the students on register must be tested and have valid scores within each accountable subgroup. To meet the participation requirement in science 80% or more of the students on register must be tested and have valid scores within each accountable subgroup. The subgroup must have 40 or more students to be held accountable for this criterion. If the group does not make the required participation rate NYSED automatically computes a weighted average using the previous year’s participation data. NOTES: NYSESLAT can be used in lieu of an 3-8 ELA assessment for students considered as ELA Exempt (NYSESLAT Eligible) Students who are medically excused are excluded from the numerator and the denominator of the participation and performance calculations.

What is EMS Accountability? – NCLB/ESEA Performance Definition: EMS Performance rate students are those who were continuously enrolled from SED BEDS Day (first Wed of October) AND during the entire test administration period with a valid assessment appropriate to their grade (or age if ungraded) and the program services reported for the student. To meet performance criterion, accountability groups must show evidence of acceptable performance on standardized assessments in the current school year or evidence of improvement in performance from the previous school year. ELA, Math and Science: In ELA, math and Science, the Performance Index (PI) of an accountability group with 30 or more continuously enrolled students must be equal to or greater than the group's Effected Annual Measurable Objective (EAMO) or the group must make Safe Harbor (ELA/Math) or its Progress Target (Science) for the group to make AYP. Performance Index: The Performance Index (PI) is a value from 0-200 that is assigned to an accountability group, indicating how that group performed on a required State test (or approved alternative) in English Language Arts, math, or science. Student scores on the tests are converted to performance levels. As per the ESEA waiver, performance levels for ELA/Math have changes. NOTES: Students who use the NYSESLAT in lieu of an 3-8 ELA assessment for Participation will be “Not Tested” for performance. Students who are medically excused are excluded from the numerator and the denominator of the participation and performance calculations. SED BEDS Day for the 2012-13 SY is October 3,2012.

What is EMS Accountability? – NCLB/ESEA Performance Index/ Performance Levels: In elementary/ middle school 3-8 ELA and math, the methodology used to calculate PI has been modified to include students showing adequate growth towards proficiency as proficient students. The Student Growth Percentile (SGP) methodology is used for students in grades 4 – 8 to measure student growth and to determine whether Level 1 and Level 2 students are on track to proficiency in three years or by Grade 8, whichever comes first. New performance levels are: Level 1 On Track = Basic and On Track to Proficient Level 1 Not On Track = Basic and NOT On Track to Proficient Level 2 On Track = Basic Proficient and On Track to Proficient Level 2 Not On Track = Basic Proficient and NOT On Track to Proficient Level 3 = Proficient Level 4 = Advanced In elementary/middle-level science, the performance levels remains the same as in prior years and has not changes as a result of the ESEA waiver. Performance Levels are: Level 1 = Basic Level 2 = Basic Proficient Level 3 = Proficient Level 4 = Advanced The PI for ELA / Math is calculated using the following equation: PI = [(number of continuously enrolled tested students scoring at Level 1 On Track + Level 1 On Track + Level 2 On Track + Level 2 On Track + Level 2 NOT On Track + Level 3 + Level 3 + Level 4 + Level 4) ÷ number of continuously enrolled tested students] 100 The PI is calculated using the following equation: PI = [(number of continuously enrolled tested students scoring at Level 2 + Level 3 + Level 4 + Level 3 + Level 4) ÷ number of continuously enrolled tested students]  100

What is EMS Accountability? – NCLB/ESEA Accountability Inclusion/Exclusion for Participation/Performance at the EMS Level Students enrolled/tested during the period of continuous enrollment shown in the table below will be included in the document/calculations indicated. Key: Day 1 = BEDS Day (October 3, 2012) Day 2 = First day of test administration period Day 3 = Last day of make-up period Specific Day 2 and Day 3 Dates by Assessment

How to Review Accountability Data? Elementary/ Middle School Verification Report – REMS When is the report Available?: April 1st until June 28th. The State and Federal Evaluation Team (SFE), in collaboration with DIIT and Data Managementcreated excel workbooks that contain the student level data used to for accountability determinations. Schools use these files to begin reviewing and familiarizing themselves with SED accountability data sets prior to the release of the SED L2RPT Elementary/ Middle School Accountability Verification Report (EMS AVR). About the Report: Student rosters with biographical and sub-group data for each SED EMS accountability measure: (1) ELA Participation and Performance, (2) Math Participation and Performance, and (3)Science Participation and Performance Data Displayed: Schools are encouraged to review the report for incorrect student detail data. For rules as to how the data set is determined review the EMS NCLB/ESEA Accountability Overview slides.

Exploring the REMS Participation and Performance data is separated into three tabs for each accountability measure. For each tab is a header containing a link to the REMS ATS Wiki page where users can find descriptions of the detail data in addition to the data refresh date; date of download; and Data/Application Support person. Schools are asked to review student demographics, accountable assessment / tested status, and assessment outcome data (when available). All data are continuously captured until of June 30th of the reporting year or the date of the student’s last enrollment record if prior to June 30th. Additional Information: https://wiki.nycenet.edu/display/ATSWiki/EMS+Data+Verification+Report+-+REMS Downloading the Report The REMS Report is an FTP download from ATS. Directions on how to download an FTP file from ATS can be found on the ATS wiki. Contact information displays on both the Report Screen and FTP report header.

SED L2RPT Elementary/Middle Accountability Verification Report – EMS AVR When is the report available?: 5/2013 – 8/2013 About the Report: The NYSED Elementary/Middle School Accountability Verification Report (EMS AVR), available on the SEDs Level 2 Reports (L2RPT) portal, allows schools to verify the accuracy of the student records as they appear in the Student Information Repository System (SIRS) used to determine school and district accountability based on 2012-13 data Data Displayed: The report provides verification that the appropriate students, with assessment results and outcomes, are being used to determine whether districts and schools have made adequate yearly progress (AYP) in elementary/ middle level English language arts (ELA), mathematics, and science. How to Review Accountability Data? * The L2RPT portal is only available to school Principals and certain CFN functional roles. L2RPT portal displays up to four years of reports in both aggregate and student detail form.

Exploring the EMS AVR Student Detail Report View Additional Information: https://wiki.nycenet.edu/display/DataVerification/EMS+Accountability+Data+Verification+Report The EMS AVR includes summary reports for ELA, mathematics, and science. Summary reports include data aggregated by school and district and by student subgroup. To view the data for each measure, select the appropriate subject from the “Select Subject” dropdown. In the “Select Grade” dropdown, data may be viewed by grade or for a combination of applicable 3–8 (ELA and mathematics) or 4 & 8 (science) grades if “All Grades” is selected. Data for “All Grades” is what is used to make accountability determinations for the school or district. The EMS AVR also includes detail reports, showing student demographic, assessment, and outcome information for students in the aggregate summary reports.

Elementary/Middle School Assessments All general-education students and students with disabilities in grades 3–8, andungraded students of equivalent age, must take: the New York State Testing Program (NYSTP) in English language arts (ELA) and mathematics, and the elementary- and middle-level science assessments, in the appropriate years; or if eligible, the New York State Alternate Assessment (NYSAA) in ELA, mathematics, science, and social studies, in the appropriate years. All students in these grades or equivalent ages who are also limited English proficient must take the New York State English as a Second Language Achievement Test (NYSESLAT) until they achieve proficiency on both the Listening and Speaking and the Reading and Writing portions of the NYSESLAT.

Elementary/Middle Accountable Expected Test Takers Report – RETT When is the report available?: April 1 until the last day of testing. About the Report: The DOE ATS Expected Test Taker Report (RETT) allows schools to review a list of currently active students expected to sit for the New York State Testing Program (NYSTP) 3-8 ELA and Math exams. If any inaccuracies exist in the RETT report, schools should make the corrections in ATS and/or SESIS when possible. Data Displayed: Review student roster, ensure that the “ELA Exempt” and “NYSAA Eligible” students are correcting identified as well as grade level and grade code. EMS Assessment Reports Additional Information: https://wiki.nycenet.edu/display/ATSWiki/NYSTP+ELA+MATH+Test+Takers+Report+-+RETT

SED L2RPT NYSTP Tested/Not Tested Confirmation Report – T/NT Elementary/Middle School Assessments When is the report available?: 6/2013 – 7/2013 About the Report: This is the Level 2 (L2RPT) report that will contain all of the students expected to be tested in a school. Once assessments are uploaded the report will list for students the appropriate NYSTP assessment, grade/age and, if not tested, the reason why. Data Displayed: This allows schools to verify that all students who met the criteria were correctly reported with the appropriate assessment and the correct relevant biographical record in SIRS. 35 Additional Information: https://wiki.nycenet.edu/display/DataVerification/Tested+Not+Tested+Confirmation+Report

Exploring Tested/Not Tested Under the “Tested” heading, “Not Scored” indicates records were:a)Successfully transmitted to Level 2 and are waiting to go to the testing contractor; or b) successfully transmitted to the testing contractor and are waiting for a score to be returned. Under “Not Tested”, “Incomplete” indicates students’ records were NOT submitted to the testing contractor for scoring, “No assessment*” indicates the student has an enrollment record; however, no assessment was reported to level 2. Student Detail Report View 36 *Schools who offer the science regents exam in place of the NYSTP for grade 8 students the tested not tested report for grade 8 science will list these students as “Not Tested” even if they took the NYSTP assessment in grade 7.

ELA Exempt NCLB requires that the reading/language arts proficiency of LEP students be measured as part of the school accountability program. USED has approved a one-time use of the NYSESLAT in lieu of the Grades 3–8 NYSTP for some LEP students. LEP-eligible students (including those from Puerto Rico) who on April 1, 2013, will have been attending school in the United States for less than one year may use the NYSESLAT in lieu of the 3–8 NYSTP in ELA to meet the NCLB participation requirement for AYP in elementary/middle-level ELA. For this purpose, the United States is defined as schools in the 50 States and the District of Columbia and does not include Puerto Rico, the outlying areas, or the freely associated States. Students may be exempt from only one administration of the NYSTP in ELA. The one-year exemption window does not have to be 12 consecutive months. In addition, students enrolled anytime during a month, including July and August, are considered enrolled for that month. As such, eligible students may be exempt from taking the NYSTP in ELA for the first year in which they are enrolled during the NYSTP ELA test administration period. Such students may not be exempt in subsequent years, even if they have been enrolled in a United States school for less than 12 months. Months in which students are enrolled as PK–8 or ungraded elementary are counted toward this 12-month exemption window. Notes: ELA Exempt is a name used by NYC only, NYSED refers to this population as NYSESLAT Eligibility and on NYSED answer documents “1st Year ELL” Calculation: Enrollment in US Schools must be less than 12 months as of April 1st of the testing year. Schools should use the Months Enrolled Update (UYRE) function in ATS to identify students who qualify for ELA exemption.

ELA Exempt To qualify for ELA exemption the student must (1) have LEP flag = Y, (2) have less than 12 months of enrollment before April 1st of the current testing year and (3) did not receive an exception in previous years. Schools should complete/review the ATS UYRE and ELPC functions to ensure all criteria are met in a students ELA Exempt status is in question. LEP Status: Students with a ATS LEP Flag = ‘E’ or ‘Pending’ ELLs awaiting Lab-R scanning . Students may still have a LEP/ELL indictor of “E” because either: The LAB-R was not administered The LAB-R scores were not uploaded in ATS There is an Exam Error for the assessment on XAMM. Students with a LEP flag of “E” will be highlighted. Students records on UYRE that are: Blue/Green – Students who are “E” with an indication on the ELPC function that the student is an ELL based on local (school level) LAB-R scoring. These students will be flagged as “ELA exempt” within the UYRE function. Red – Students who are “E” with either no entry made in ELPC function that the student is NOT an ELL or the based on local (school level) LAB-R scoring. These students will not be flagged as have an “ELA Exempt”.

ELL Parent Choice Update –ELPC The purpose of ELPC is for schools to enter/update the ELL Parent Choice Data. This screen is only open for students whose LEP Flag is either "Y" or "E“. The first two fields must be entered. If the answer to the first question is "Y", then the remaining three fields must also be entered. Additional Information: https://wiki.nycenet.edu/display/ATSWiki/ELL+Parent+Choice+Update+-+ELPC

Months Enrolled Update Screen - UYRE This function is designed to collect months enrolled in another USA school as noted on the Home Language Identification Survey (HLIS) for ELLs with less than six years of DOE enrollment. Information gathered will assist in identifying students who qualify for ELA exemption in informing promotion criteria. LEP Flag = ‘E’ Pending Blue/Green– Students who are “E” with an indication on the ELPC function that the student is an ELL based on local (school level) LAB-R scoring. These students will be flagged as “ELA exempt” within the UYRE function. Red – Students who are “E” with either no entry made in ELPC function that the student is NOT an ELL or the based on local (school level) LAB-R scoring. These students will not be flagged as have an “ELA Exempt”. Additional Information: https://wiki.nycenet.edu/display/ATSWiki/Updating+Months+Enrolled+in+US+Schools+-+UYRE

Exam Errors - XAMM Pre-slugging the documents reduces the possibility of human error and ensures that exam results will be posted to a student’s exam record. However, errors do occur. Schools should periodically review and correct any exam errors. The school STATscreen lists the number of exam errors pending. Schools should use the XAMMfunction to identify these errors (or REXE) and investigate the discrepancy to resolve and remove from the function. Exams listed on this function will not post on the students exam record until verified. Once corrected the exam is immediately posted to the students exam record, for LAB-R corrections, this may change LEP pending status. Additional Information: https://wiki.nycenet.edu/display/ATSWiki/Accepting+Exam+Records+-+XAMM

Appendix Age Ranges for Students with Disabilities Participating in the 2012-13 (NYSAA) Order of Precedence – Elementary/ Middle Schools

Age Ranges for Students with Disabilities Participating in the 2012-13 (NYSAA) New York State Alternate Assessment NYSAA is a datafolio assessment that measures the progress of students with severe disabilities in achieving the learning standards through alternate grade level indicators. All students in the following age ranges who are eligible to take the NYSAA must be tested in 2012-13. Students with disabilities participating in the NYSAA are assessed according to chronological ages aligned with grade levels as indicated below. Students should be tested only once at each grade and in all the content areas indicated for each grade. (http://www.p12.nysed.gov/assessment/nysaa/2012-13/age-range-13.pdf) *Note: NYSAA-eligible students who do not meet the age criteria above and will be leaving school before they reach their eighteenth birthday must take the secondary-level NYSAA before they leave school (i.e., when they are 17 years old). NYSAA-eligible students with a birth date prior to September 1, 1994 who have not been assessed at the secondary-level must be assessed in 2012-13 before they leave school.

Order of Precedence – Elementary/Middle School Grades 3–8 English Language Arts and Mathematics Order of Precedence Rules: If a student has more than one applicable ELA score, the order of precedence for selecting a performance level for use in the PI calculation is: 1) NYSTP and 2) NYSAA. The order of precedence for selecting participationis 1) NYSTP, 2) NYSAA and 3)NYSESLAT Grade 8 Science Order of Precedence Rules: If an eighth-grader has more than one applicable science score, the order of precedence for selecting a performance level for use in the PI calculation is: 1) New York State Grade 8 Middle-Level Science Test for the current year, 2) NYSAA Grade 8 Equivalent in Science, 3) highest Regents science examination, and 4) New York State Grade 8 Middle-Level Science Test taken by the student in 7th grade in the previous year. NYSESLAT: At the elementary/middle level, if a district chooses to give the NYSTP ELA assessment to a LEP student who is eligible to take the NYSESLAT in lieu of the NYSTP, NYSED will count the student’s NYSTP ELA scores when computing the school’s and district’s accountability PI.

2012-13 Data Verification Part I Spring 2013