SES Student Achievement Methods/Results: Multiple Years and States

1. SES Student Achievement Methods/Results:Multiple Years and States Steven M. Ross Allison Potter The University of Memphis Center for Research in Educational Policy

2. Supplemental Educational Services (SES) • Required under No Child Left Behind (NLCB) for Title I Schools that have not made Adequate Yearly Progress (AYP) for three consecutive years. • Low-income students from identified Title I schools are eligible to receive free tutoring services. • Students are prioritized by greatest academic need if district funds are limited.

3. Service Providers • Potential service providers apply to serve students and may be approved by the State Department of Education. • Providers contract with Local Educational Agencies (LEAs) to provide tutoring services to students. • Providers are paid for their services - an amount not to exceed the Title I per pupil allotment.

4. Overall Provider Assessment Figure 1. Components of a Comprehensive SES/Evaluation Modeling Plan Service Delivery and Compliance District Coordinator Survey Customer Satisfaction Principal/Liaison Survey Provider Survey Teacher Survey Parent Survey Effectiveness (Student Achievement) State Tests Additional Tests

5. Evaluation Designs:Student Achievement A. Benchmark Comparison Rating = ++ (Low to Moderate rigor) Percentage of SES students by provider attaining “proficiency” on state assessment

6. Evaluation Designs:Student Achievement A. Benchmark Comparison Upgrades • Percentage of SES in all performance categories (“Below Basic”, “Basic”, etc.) • Comparison of performance relative to prior year and to state norms • Comparison to a “control” sample

7. Evaluation Designs:Student Achievement Benchmark Comparison • Advantages • Inexpensive and less demanding • Easily understood by practitioners and public • Linked directly to NCLB accountability • Disadvantages • Doesn’t control for student characteristics • Doesn’t control for schools • Uses broad achievement indices

8. Evaluation Designs:Student Achievement B. Multiple Linear Regression Design Rating = +++ (Moderate rigor) Compares actual gains to predicted gains for students enrolled in SES, using district data to control for student variables (e.g., income, ethnicity, gender, ELL, special education status, etc.).

9. Evaluation Designs:Student Achievement Multiple Linear Regression Design • Advantages • More costly than Benchmark, but relatively economical • Student characteristics are statistically controlled • Disadvantages • Doesn’t control for school effects • Less understandable to practitioners and public • Effect sizes may be less stable than for Model C.

10. Evaluation Designs:Student Achievement C. Matched Samples Design Rating = ++++ (High Moderate to Strong rigor) Match and compare SES students to similar students attending same school (or, if not feasible, similar school) Use multiple matches if possible

11. Evaluation Designs:Student Achievement C. Matched Samples Design • Advantages • Some control over school effects • Easily understood by practitioners and public • Highest potential rigor of all designs • Disadvantages • More costly and time consuming • Within-school matches may be difficult to achieve

12. Tennessee Results2003-2004 Multiple Regressions Analysis Model: Predictors: Prior achievement, district, poverty, gender, special ed status State-wide Aggregate Results: Out of 17 Reading analyses, 0 were significant Out of 14 Math analyses, 0 were significant

13. Tennessee Results2004-2005 Multiple Regressions A. Analysis Model: Predictors: Prior achievement, district, poverty, gender, special ed status State-wide Aggregate Results: Out of 25 Reading analyses, 0 were significant Out of 23 Math analyses, 0 were significant

14. Tennessee Results2004-2005 Multiple Regression using SAS EVAAS B. Analysis Model: Control Variable: Prior achievement and teacher State-wide Aggregate Results: Out of 9 Reading analyses, 0 were significant Out of 12 Math analyses, 0 were significant

15. Tennessee Results2004-2005 Matched Student Pairs using SAS EVAAS C. Analysis Model: Control Variable: Predicted achievement and teacher State-wide Aggregate Results: Out of 9 Reading analyses, 0 were significant Out of 12 Math analyses, 0 were significant

16. Tennessee Results2005-2006 Multiple Regression using SAS EVAAS A. Analysis Model: Control Variable: Prior achievement and teacher State-wide Aggregate Results: Out of 10 Reading analyses, 1 was significant & negative Out of 10 Math analyses, 0 were significant

17. Tennessee Results2005-2006 Matched student pairs using SAS EVAAS B. Analysis Model: Control Variable: Predicted achievement and teacher State-wide Aggregate Results: Out of 10 Reading analyses, 1 was significant & negative Out of 10 Math analyses, 1 was significant & negative

18. Louisiana Results2004-2005 Matched Samples Analysis Model: Controls (covariates): Prior achievement, district, ethnicity, and gender State-wide Aggregate Results: Out of 7 Reading analyses, 0 were significant Out of 10 Math analyses, 1 was significant & positive

19. Virginia Results2005-2006 Benchmark Analysis Model: Controls (covariates): Poverty and school State-wide Aggregate Results: Out of 13 Reading analyses, 5 were significant & negative Out of 12 Math analyses, 1 was significant & positive 4 were significant & negative

20. SUMMARY OF ACHIEVEMENT EFFECTSBY STATE AND SUBJECTS