Nicole Traxel & Cindy Walker University of Wisconsin - Milwaukee April 14, 2009

1. The Impact of Including Predictors and Using Various Hierarchical Linear Models on Evaluating School Effectiveness in Mathematics Nicole Traxel & Cindy Walker University of Wisconsin - Milwaukee April 14, 2009 The Milwaukee Mathematics Partnership (MMP) is supported by the National Science Foundation under Grant No. 0314898.

2. Introduction • Value added models • Fair and accurate way to assess the effectiveness of schools • Determine how much value a school adds to student learning by examining student progress over time • Operational definition: Effectiveness=growth • Hierarchical linear models can be used to implement a value added accountability system • Hierarchical because students nested within schools • Can determine how much of growth can be attributed to the student and to the school

3. Types of Hierarchical Linear Models • Several different hierarchical linear models can be used to assess school effectiveness, so which is best? • 2-level hierarchical model predicts final achievement from initial achievement. Can include student level and school level predictors of achievement, not growth • 2-level growth model predicts change in test scores from one year to the next. Can include student level and school level predictors of growth, not achievement • 3-level individual growth model predicts achievement and change over time. Can include student level and school level predictors of growth and achievement

4. Research Questions • Do effectiveness rankings differ depending on which type of model is used? • Does predictor significance remain constant across model types? • Does including predictors change effectiveness rankings of school?

5. Sample & Measures • 7,232 students from 128 school from a large urban school district in the Midwest • 3rd to 4th grade • 87% minority, 79% receive free/reduced lunch • Mathematics scores on a state mandated standardized test • Math Focus score for each school – “There is a strong focus on increasing student achievement in mathematics at my school.” • Gain in Math Focus calculated by subtracting Math Focus score from 1st year from Math Focus score from 2nd year

6. The Models That Were Fit Note: Initial Score was included as a student level covariate in all 2-level and 3-level models, but not in the 2-level gain models.

7. Comparisons • Predictor significance across models • Effectiveness rankings across predictors being included within each model type • Effectiveness rankings across model types for models including only initial score or no predictors • Effectiveness rankings across model types for models including only initial score or no predictors validated using gain in Math Focus score

8. Predictor Significance • Student-level SES and Student-level Race were significant predictors of the average achievement of students within schools but not of the average growth of students within schools • School-level Race and School-level SES were significant predictors of the average achievement among schools but not of the average growth among schools • But not when both were included, due to collinearity

9. Predictors Do Not Change Effectiveness Spearman Correlations between the null model and models including predictors for each model type.

10. Correlations Among Model Types (Null Models) • 2-level and 2-level gain models • r = .090 • 2-level and 3-level models • r = .101 • 2-level gain and 3-level models • r = .993 • Therefore, rankings from 2-level gain and 3-level models are very similar to one another

11. Validating Effectiveness Rankings • Pearson correlations between gain in math focus and effectiveness estimates from null models of each model type. • 2-level: r = .034, p = .747 • 2-level gain: r = .200, p = .055 • 3-level: r = .177, p = .089

12. Conclusions, Part One • Including predictors, even if they are significant, does not change effectiveness estimates • Effectiveness estimates from null 2-level model were different from effectiveness estimates from null 2-level gain and 3-level models • Effectiveness estimates from 2-level gain and 3-level models were very similar

13. Conclusions, Part Two • Correlation between 2-level gain and 3-level models and gain in math focus had higher magnitudes than correlation between 2-level model and gain in math focus • Effectiveness estimates from 2-level gain and 3-level models are more valid than those from 2-level model

14. So which model type should I use? • 3-level model has several advantages over 2-level gain model • Includes ALL available data-all participants with at least one observation are included • Can include many years of data • Provides more information (growth and achievement estimates)