Update on Virginia’s growth measure

1. Update on Virginia’s growth measure Deborah L. Jonas, Ph.D. Executive Director for Research and Strategic Planning Virginia Department of Education April 2011

2. Federal Requirement: Student-level growth data(b)(2) As part of conditions of accepting money from the State Fiscal Stabilization Fund (SFSF), Virginia is required to: • Develop a growth measure; and • Provide growth data, at minimum, to teachers of reading/language arts and mathematics in tested grades. • Data must be provided to two groups of teachers: • Teachers who taught students previously: • Example: growth data from tests administered in spring 2011 must be provided to teachers who taught students in the 2010-2011 school year. • Teachers who are currently teaching students • Growth data from tests administered in spring 2011 must be provided to teachers working with these students in the fall of 2011. • The state must be capable of providing student-level data to teachers no later than September 30, 2011.

3. Federal Requirement: Teacher impact data, indicator (b)(3) As part of SFSF requirements Virginia is required to: • Provide teachers of reading/language arts and mathematics … with reports of individual teacher impact on student achievement on state assessments. • Virginia’s approach • Use the student-level growth data to provide teachers with context for understanding their impact on student achievement as measured by statewide assessments. • As multiple years of data become available, VDOE will create reports that enable teachers to consider trends in student growth data. • When tests measuring the same SOL are available over multiple years, trend data can be used in aggregate.

4. Student Growth Percentiles (SGPs) • SGPs provide a measure that quantifies the progress each student made on Standards of Learning (SOL) tests compared to other Virginia students statewide who had similar prior achievement histories on previous SOL tests. • SGPs are calculated using a statistical methodology that accounts for student prior achievement to determine each student’s progress. • SGPs provide an understanding of how much progress students made based on where they started—regardless of whether they started as low, moderate, or high achieving students.

5. How are SGPs quantified, and how can they be interpreted? • SGPs range from 1 to 99, and represent the percent of students who had similar prior academic achievement (based on SOL tests) and who earned lower scores on the SOL test. • Example: a student who earns an SGP of 65 scored better than 65 percent of students who had the same prior achievement as measured by SOL tests. • Example: a student who earns an SGP of 15 scored better than 15 percent of students who had the same prior achievement as measured by SOL tests.

6. Three students with the same scaled scores, grade 6 reading

7. Same three students—in grade 7 SGP = 18 SGP=48 SGP=61 Example only. Note that SGPs account for as much historical data as are available.

8. Growth can be measured for high and low achieving students • Low achieving students can demonstrate high growth. • High achieving students can demonstrate low growth.

9. Growth Measure Availability • VDOE finalized statistical models for student growth percentiles (SGPs) working with Dr. Damian Betebenner from the Center for Assessment. • SGPs will be available for most students who participate in the following SOL (not alternative) assessments: • Mathematics, grades 4-8 and EOC Algebra I through ninth grade; • Reading, grades 4-8. • VDOE will provide student-level data by September 30, 2011; aggregate data will be provided in the future.

10. Growth levels • To support interpretation of growth data, VDOE will define categories/levels of student growth. Growth levels defined as of April 2011.

11. Some students will not have student growth percentiles calculated • Some students who have taken the grades 4-8 reading tests and/or the mathematics tests for grades 4-8 or Algebra I will not have growth percentiles. • There are three primary groups who will not have student growth percentiles available from the state: • Students with only one year of assessment data available. • Students who participated in Virginia’s alternative assessment programs (VGLA, VSEP, VAAP) in the year of, or year prior to, the reporting year. • Students who participated in the traditional SOL assessment for two consecutive years but for whom no valid growth measure is available. • This group consists primarily of students who achieve at the highest levels of Virginia’s assessments; growth for these students cannot be adequately captured with current statewide assessments. • As Virginia’s assessments change, VDOE anticipates that more students will have student growth percentiles available in reading and mathematics.

12. Note about students earning high scores on Virginia’s tests • Student growth percentiles, at their most basic level, describe the likelihood of a student with a particular score history earning a particular score on the next state assessment. • If a student scores at or near 600 in one year, the SGP for next year is dependent on the likelihood that students who earn perfect scores earn them multiple years in a row. • In Virginia, the likelihood of a student earning two or more of the highest scores in a row varies by content area and grade level.

13. Results of VDOE’s SGP Analysis of SOL Scores and SGPs • SPGs meet statistical and subject matter expert requirements for accurately measuring relative growth on statewide assessments for most students. • Student growth cannot be accurately determined for some students who earn advanced proficient scores two or more years in a row. • For some of these students, growth as calculated by the SGP is clearly high, but how high is not known. These students will be reported as having “high growth” but will not be assigned an SGP. • For other students, the growth category cannot be determined accurately with current assessments. These students will have neither an SGP or a growth category reported.

14. VDOE’s approach to reporting SGPs for high achieving students • VDOE is establishing business rules for determining which students will have growth data reported. • Business rules may differ in mathematics and reading. • VDOE is striving for similar structure in these rules between content areas.

15. Example of business rules for SGP data

16. Providing Growth Data to School Divisions • VDOE may provide student-level growth data before September 2011 via SSWS. Data would be from test years 2009-2010 and earlier. • If these data can be provided prior to September 2011, they will not be connected to teachers and will not be aggregated. • Data would be made available to central office personnel, and VDOE would provide a webinar to explain the data and answer questions as they arise.

17. How will teachers receive data? • VDOE will provide data to the school divisions’ central offices, via the SSWS system (2011-2012 only; changes may take place in future years). • To comply with SFSF, school divisions are responsible for providing student-level data to teachers consistent with requirements of indicator (b)(2): • Teachers who previously taught students • Teachers who currently teach students • Teacher impact data (b)(3) will be comprised of student level data and, when available and appropriate, aggregate and trend data.

18. Training on Growth Percentile Data and its Uses • VDOE is establishing an agreement with a team comprised of statistical experts and professional development providers to offer school division teams training on growth percentile data use and interpretation. • Plans are being developed to provide introductory training opportunities in-person and via webinars. • More information about training opportunities will be provided as it becomes available.

19. Potential Uses of Student Growth Measures in Virginia • School improvement • Program evaluation • Communications with parents, teachers, staff • One component of comprehensive performance evaluation • Consistent with the Code of Virginia requirement to incorporate measures of student academic progress in evaluations (§ 22.1-295). • Information on appropriate use is included in draft guidance documents to be considered by the Virginia Board of Education, March 2011. • Federal accountability • To be explored after Virginia meets current federal requirements for linking students to teachers and developing growth measure; • Cannot be implemented without two years of data from new tests.

20. Implications for Teacher Performance Evaluation

21. Background: Code of Virginia • § 22.1-253.13:5 Standard 5. Quality of classroom instruction and educational leadership. …B. Consistent with the finding that leadership is essential for the advancement of public education in the Commonwealth, teacher, administrator, and superintendent evaluations shall be consistent with the performance objectives included in the Guidelines for Uniform Performance Standards and Evaluation Criteria for Teachers, Administrators, and Superintendents….

22. Background: Code of Virginia • § 22.1-295. Employment of teachers. …C. School boards shall develop a procedure for use by division superintendents and principals in evaluating instructional personnel that is appropriate to the tasks performed and addresses, among other things, student academic progress and the skills and knowledge of instructional personnel, including, but not limited to, instructional methodology, classroom management, and subject matter knowledge….

23. Teacher Evaluation – Measuring Progress in Student Achievement • In March 2011, the Virginia Board of Education accepted for first review revised Uniform Performance Standards and Evaluation Criteria for Teachers.* • These documents recommend that student academic progress account for a significant component of the evaluation (recommendation is 40 percent; six other performance standards comprise the remaining 60 percent). • Recommends that determinations of student academic progress be assessed with multiple measures. *see: http://www.doe.virginia.gov/boe/meetings/2011/03_mar/agenda_items/item_l.pdf

24. SGPs as One Measure of Progress in Teacher Performance Evaluation • Growth percentiles may form the basis of one of multiple measures of student progress when: • Data from at least 40 students are available; and • Students with growth data are representative of students taught; and • Growth data for at least two years are available; three years should be reviewed whenever possible. • Aggregate growth percentile data is not recommended when these conditions are not met.

25. Use of SGPs in other states

26. How Can Teachers Use This Data? • Reflect on student needs quickly and efficiently • Is the student growing appropriately to meeting state standards ?* • Is the student growing as much in reading as math? • Did the student grow as much this year as last year? • Spot trends in a student’s learning so they can encourage more and/or faster growth or act to stop a downward drift. • Identify students scoring low but who have grown dramatically in the past – requiring different strategies than students with low, flat scores. *Predictive targets will not be available in Virginia until there are two years of data available from the new mathematics and reading assessments. Source: Presentation by Rebecca Gau, November 2010, to Virginia stakeholders. Full presentation is available at: http://www.cit.org/programs/cit-connect/vdoe-gm.

28. How Can Leaders Use this data? • Teacher professional development • Plan resources for intervention and remediation Source: Presentation by Rebecca Gau, November 2010, to Virginia stakeholders.

29. Do students at certain grade levels grow faster than others? Source: Presentation by Rebecca Gau, November 2010, to Virginia stakeholders, example from Arizona data.

30. Are the low scoring students growing as much as the high scoring students? Source: Presentation by Rebecca Gau, November 2010, to Virginia stakeholders, example from Arizona data.

31. Growth, achievement, and economically disadvantaged status in Colorado Median growth percentile is not correlated with economic status (R=-.124) School achievement is correlated with economic status (R=-.759) Mean Scale Score Medan Growth Percentile Percent economically disadvantaged Percent economically disadvantaged Adapted from: Schmitz, D. (2010). Utility of Growth Percentiles. Presented November 2010, Virginia stakeholders. Full presentation is available at: http://www.cit.org/programs/cit-connect/vdoe-gm.

32. Assessment and questions about growth calculations: student_assessment@doe.virginia.gov Teacher performance evaluations: licensure@doe.virginia.gov Data/master schedule collection: resultshelp@doe.virginia.gov Contact information