Longitudinal Analysis of MAP Achievement Growth: Preliminary Estimates of School Effects

1. Longitudinal Analysis of MAP Achievement Growth: Preliminary Estimates of School Effects Sept. 15-16, 2010 Mark Ehlert Cory Koedel Michael Podgursky Department of Economics, MU CALDER, NCPI Kansas City Area Education Research Consortium Prepared for Missouri Technical Advisory Committee meeting. Kansas City, MO. September 15-16, 2010

2. Overview • Examination of emerging MOSIS data system • Patterns of Scale Score growth in MAP • A Simple VAM for School Effects • Model and results • Covariates or not? • Estimation of Teacher Program Effectiveness • Future directions

3. Missouri is developing a sophisticated P-20 data system • IES State Longitudinal Data Grant • Ranks very favorably compared to other states • Data quality is high

4. Data • Matched Spring 2006-2009 student MAP scores using MOSIS ID • Exclusions • Bad/duplicate values of MOSIS • Students retained in grade • Special districts • Match rate for 4 years roughly 85% - 87%; match rate for 1 year regularly at 95%

5. Math MAP Testing Regime

6. MAP Math: Average Performance By Cohort

7. MAP Com. Arts : Average Performance By Cohort

8. MAP Math: 2008-2009 Average Gain Score By Grade and Decile of 2008 Performance

9. MAP Com Arts: 2008-2009 Average Gain Score By Grade and Decile of 2008 Performance

10. Value-added models • Why “value-added?” • Traditional economic definition • Business or firm value-added • Value of output – value of inputs • VAT • Education analogy • Control for initial (pre-treatment) performance • Estimate the effect of contemporaneous inputs on education outcomes

11. We want to identify causal effect of inputs • “what works” • Treatment and control / comparison groups • Example: teacher training programs and teacher effectiveness • Class size • Teacher credentials

12. A Simple VAM Student Characteristics Lagged or baseline performance random error A i jt = f (A it–k , S i , SCH j ) + εij t Educational outcome (e.g., test score, graduation, college attendance) School / classroom inputs or treatment i – th student j –th school or classroom t – th year or grade

13. Gain Score Lagged Test Scores in both subjects Ai g - Ai g-1 = f (Ai g-1 (m, ca), student char, grade, year) + school effects + εi t Average Effect by school (state mean = 0) Model estimated over all Missouri students, grades 3-8 Schools included if n > 20 student gain scores 3 gain scores x multiple grades per school

22. Effect of Covariates (math results) Model 1 = student covariates Model 2 = no student covariates

32. Work Under Way • Teacher training program effects • New teachers • Retirement system effects • Effectiveness of teachers x retirement behavior

33. Ai g - Ai g-1 = f (Ai g-1 (m, ca), student char, grade, year) + school effects + teacher effects + εi t Within school Model estimated over all Missouri students, grades 3-8

34. Ai g - Ai g-1 = f (Ai g-1 (m, ca), student char, grade, year) + school effects + teacher char + εi t Within school Model estimated over all Missouri students, grades 3-8

35. Comparative Effectiveness of Teacher Preparation Programs

36. 37 Teacher Training Programs

37. Schools with at least one new teacher graduate: Fall, 2005 – Fall, 2009

44. 37 x 37 cross placement of program grads x school, gr. 4-8

45. Other research Teacher Pension Effects How do pension rules affect workforce quality?