Understanding Student Achievement: The Value of Administrative Data

1. Understanding Student Achievement: The Value of Administrative Data Eric Hanushek Stanford University

2. Big Issues in School Policy Debates • Relating analysis to policy interests • Confidence in causation • Generalizability

3. Analytical designs • Random assignment experiments • Natural experiments • “Data solutions” • Trade-offs • Credibility • Expense • Questions that can be addressed

4. UTD Texas Schools Project • Multiple cohorts followed 1993-2002 • Annual achievement in grades 3-8 (TAAS math and reading) • Each cohort > 200,000 students in over 3,000 schools • Augmented with district data

5. Examples of Topics • Teacher quality variations • Charter schools • Not discussed • School choice and mobility • Special education • Teacher mobility • Racial composition • Peer achievement

6. Existing Evidence on Teacher Quality • Substantial variation in teacher quality • Observable characteristics of teachers explain little of the variation • Salary and other factors affect teacher transition probabilities • No evidence on transitions and teacher quality

7. Questions Addressed • What is variation in teacher quality? • Measurable characteristics? • Do urban schools lose their best teachers? • Quality by transitions • Do districts hire the best teachers?

8. Basic model

9. Measurement Error and Calculation of Variance of Teacher Quality • Observe teachers in two years: • Correlation across years:

10. Estimated Variance in Teacher QualityLonestar District

12. Conclusions on Teacher Quality • Very large differences among teachers • Differences within schools much larger than between schools • Conventional measures not good index of quality (master’s degree, certification test) • Observable characteristics • First year of experience • Teacher-student race match • Common assumptions about market for teachers not correct • Best do not leave • Districts with advantages do not use them

13. Popularity of charter schools • 3,000 charter schools • 40 states plus DC since 1991 • 1 percent of total students • 10 percent of size of private school market • 7+ percent rate of closure

14. Evaluation issues • Most analysis of entry and participation • No reliable information on performance • Difficulty of selection issue • Very political

15. Evaluation approaches • Model selection process [Heckman (1979)] • Instrument for attendance [Neal(1997)] • Intake randomization [Howell and Peterson (2002)]

16. Difficulties with traditional approaches • Difficult to find factors affecting attendance but not achievement • Cannot handle treatment heterogeneity

17. Empirical framework • Mean differences in individual value-added • Identify charter school from individual entry-exit • Consider time varying effects associated with charter school movements • Heterogeneity across schools • Consumer responsiveness to quality

18. Charter enrollment

19. Participation rates by race/ethnicity

20. Charters by vintage (analytical)

21. Charters by vintage (analytical)

22. Charters by vintage (analytical)

23. Charter school effect

24. Demographically Adjusted School Quality

25. Do parents make good decisions? • Parents cannot see value added • Considerable mobility/exiting • Models: • Exit=f(quality, age, year, race, grade)

26. Parental Choice(linear probability of exit)

27. Parental Choice(linear probability of exit)

28. Conclusions on Charter Schools • Difficult start-up period • Mean performance regular ≈ charter after two years • Heterogeneity in both markets • Parents react to quality in charter market • Low income reaction one half upper income

29. Administrative data • Pros • Broader generalizability • Understanding heterogeneity • Perhaps less costly • Cons • Requires structure (e.g., linearity, time pattern of achievement) • Regulatory problems (confidentiality) • Data quality issues

30. Papers on Teacher Quality and Charter Schools • www.hanushek.net or www.nber.org Hanushek, Eric A., John F. Kain, Daniel M. O'Brien, and Steve G. Rivkin. 2005. "The market for teacher quality." National Bureau of Economic Research, Working Paper No. 11154, (February). Hanushek, Eric A., John F. Kain, Steve G. Rivkin, and Gregory F. Branch. 2005. "Charter school quality and parental decision making with school choice." National Bureau of Economic Research, (March).