Lou Jacobson Presentation at the IES Research Conference June 7, 2007

1. Use of Growth Models to Measure Adequate Yearly Progress (AYP) Under the No Child Left Behind (NCLB) Act Lou Jacobson Presentation at the IES Research Conference June 7, 2007

2. Topics to be Covered Research for REL Appalachia: • Year-1 study examining the likelihood Virginia schools would have all students proficient in reading & math by 2014. • Today’s focus is on reading results for all students together. • The study also covers math and each of 6 subgroups. • The report will be on the REL website in about a month. • Year-2 study examines a second key NCLB goal—identifying under-performing schools • Use growth models developed in year-1. • Examine the accountability systems in Virginia, Kentucky, and Tennessee using school level data.

3. Virginia’s NCLB Status Standard • 100% of the students in a school must be proficient in math and reading by 2014 • Adequate Yearly Progress (AYP) requires meeting interim standards annually.

4. Starting Reading Proficiency Scores

5. As proficiency levels rise, how will growth change? Strategy for answering this question: • Estimate the relationship between the 2002 proficiency level and the change in proficiency 2002-05. • Forecast growth assuming that as proficiency rises schools starting at low levels will follow the same path as schools starting out at high levels.

6. Growth is Inversely Related to Starting Point

7. Our Growth Model ΔP= α+ β P0 + ε where: ΔP = (P3 – P0)/3 (annualized proficiency change) Ps,t is the percent of test-takers scoring proficient in school “s” in year “t” α = the intercept coefficient β = the slope coefficient ε = error term Averaging the proficiency change over 3 years reduces the effect of random variations, which accounts for as much as 75 percent of year-to-year change.

8. Model Estimates Standard errors in parentheses

9. What Does the Model Say about High Schools? • Annual proficiency growth fell by 2.2 percentage points for each 10-point gain in proficiency. • It becomes progressively more difficult to boost proficiency as higher and higher proficiency levels are reached. • Schools are unlikely to raise proficiency by a constant number of students as the pool of non-proficient students shrinks.

10. How High Will Proficiency Rise? • Is there a proficiency ceiling? • VA schools have had major increases in proficiency, BUT • Proficiency levels will plateau in about 3 years • How do we know? Solve the growth equation for zero growth. C = α / β where: C = the proficiency ceiling α = the intercept coefficient β = the slope coefficient • Where are the plateaus?

11. Steady State Proficiency Levels 13 points 20 points 27 points Upper Limit Lower limit = 1 std dev below average

12. Implications of These Findings • Few, if any, schools will sustain 100% proficiency levels, if current patterns continue. • Schools under-performing the status standard can avoid being labeled “in need of improvement” because there is an alternative NCLB standard. • Safe Harbor (Growth) Standard requires that schools not meeting the status standard must reduce the percentage of non-proficient students by 10 percent from one year to the next. • A school at 80 percent proficiency has to increase proficiency by 2 percentage points [100-80 = 20; 10% of 20 = 2] • A school at 90 percent proficiency has to increase proficiency by 1 percentage point. • In contrast, the Virginia Status Standard rises by 4 points per year.

13. Can a growth standard identify under-performers? • Yes, if we can determine how proficiency changes among schools facing similar challenges. • To do this it is necessary to develop a peer-group standard—the standard experts agree provides the best means to identify schools needing improvement.

14. Developing a Peer-Group Standard • Use our growth model to measure the difference between actual and predicted growth over 3-years. • Identify under-performing schools based on performance being below average by a statistically significant amount. • Group schools based on exogenous characteristics such as students’ entry level proficiency.

15. Distribution of High Schools Sorted by Difference Between Actual and Predicted Performance Under performing Over performing

16. What the figure tells us • The difference between actual and predicted growth is close to normally distributed. • Most schools’ actual performance is within one standard deviation of predicted levels. • A few schools under-perform by large and statistically significant amounts. • A few schools over-perform by large and statistically significant amounts.

17. Advantages of Using the Growth Model • The growth model provides: • A “rating” for each school that tells us how far its performance is from average for schools with similar baseline proficiency. • Measures of the statistical significance of the ratings. • In contrast, Virginia’s NCLB AYP standards provide: • Indiscriminant pass/fail measures. • Ambiguous indicators of whether performance is truly below a standard.

18. Benefits of the Peer-Group Concept • Improvement efforts can be focused on schools needing improvement that should be able to make substantial progress. • Incentives to improve performance will be strengthened by not asking educators to do the impossible. • External pressure to correct problems will be increased because confidence that problems are real and can be effectively addressed will grow.

19. Benefits of Identifying Over-Performing Schools • Comparisons between over-performing schools and under-performing schools can identify factors that are: • not under the control of school officials that should be taken into account in creating peer-groups. • under the control of school officials that make a substantial difference among schools facing similar challenges.

20. Can Safe Harbor Standards Identify Under-Performers? Yes, if: • Growth is averaged over several years. • Tests of the statistical significance are applied. However, regression-based standards better measure average performance of schools at different proficiency levels.

21. Some Schools are Misclassified by Current Standards Under performing Over performing Some of these schools miss AYP Some of these schools make AYP

22. Safe Harbor Pass/Fail Based on Annual Average Growth by Performance Quintile

23. Safe Harbor Pass/Fail Based on 3-Year Average Growth by Performance Quintile

24. Summary: Proficiency Forecasting • The simple linear model using base-period proficiency fits the data well and produces reasonably small confidence intervals. • Additional factors are unlikely to have much of an effect. • The future may not precisely track the past, but is likely to come reasonably close. • It would be useful to see how well our model works in projecting proficiency in other states.

25. Summary: Peer-Group Modeling • Our peer-group model comes reasonably close to: • Revealing how far above or below average is actual performance. • Identifying cases where deficits are statistically significant. • Averaging growth over several years greatly improves the statistical reliability of the measures. • Our model addresses widely recognized problems that reduce support for the positive aspects of NCLB.