1 / 45

THE 34TH ANNUAL NATIONAL CONFERENCE ON LARGE-SCALE ASSESSMENT June 20 – 23 2004 Boston, MA

Using Hierarchical Growth Models to Monitor School Performance: The effects of the model, metric and time on the validity of inferences. Pete Goldschmidt, Kilchan Choi, Felipe Martinez.

jerome
Download Presentation

THE 34TH ANNUAL NATIONAL CONFERENCE ON LARGE-SCALE ASSESSMENT June 20 – 23 2004 Boston, MA

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using Hierarchical Growth Models to Monitor School Performance: The effects of the model, metric and time on the validity of inferences Pete Goldschmidt, Kilchan Choi, Felipe Martinez UCLA Graduate School of Education & Information StudiesCenter for the Study of EvaluationNational Center for Research on Evaluation, Standards, and Student Testing THE 34TH ANNUAL NATIONAL CONFERENCE ON LARGE-SCALE ASSESSMENT June 20 – 23 2004 Boston, MA

  2. Purpose: We use several formulations of multilevel models to investigate four related issues: one, whether (and when) the metric matters using longitudinal models to make valid inferences about school quality; two, whether different longitudinal models yield consistent inferences regarding school quality; three, the tradeoff between additional time points and missing data; and four, the stability of school quality inferences across longitudinal models using differing number of occasions We examine three types of models: Longitudinal Growth Panel Models Longitudinal School Productivity Models Longitudinal Program Evaluation Models:

  3. Longitudinal Growth Panel Models (LGPM) Research Questions: Inferences affected by test metric (Scale Scores Vs. NCEs)? Estimates for Growth Estimates of school effects Estimates of Program effects

  4. LGPM • Longitudinal Panel Design • Keep track of students’ achievement form one grade to the next • E.g., collect achievement scores at Grades 2, 3, 4, and 5 for students in a school • Focus on students’ developmental processes • What do students’ growth trajectories look like?

  5. Choice of Metric: Scale Scores: Normal Curve Equivalents: • Change represents a relative position from year to year not absolute growth in achievement • Relative standing compared to a norming population • IRT-based scale scores • Vertically equated scores across grades and years • Theoretically represent growth on a continuum that can measure academic progress over time • Change from year to year is an absolute measure of academic progress

  6. 2 Student Characteristics

  7. 3 Sampling Conditions for Monte Carlo Study

  8. 4 Summary Parameter Estimates Compared

  9. 5 Summary of Estimates Compared Using Rank Order Correlations

  10. 6 Summary of Results Describing SAT-9 Reading Achievement

  11. 7 Percent Reduction in Between School Variation in Growth

  12. 11 Correlations between Estimated Coefficients – Model 4

  13. 12 Correlation Pattern between Sampling Condition and Model – Reading SAT-9 Growth

  14. 13 Comparison of relative Bias to the Effect Size of Growth

  15. 14 Relationship between Relative Bias in NCEs for Initial Status

  16. 17

  17. 15 Relationship between Relative Bias in NCEs for Growth

  18. Longitudinal School Productivity Model (LSPM) Research Questions: Inferences affected by test metric (Scale Scores Vs. NCEs)? Estimates for growth Estimates of school effects Estimates of “Type A” and “Type B” effects

  19. LSPM • Multiple-cohorts design (Willms & Raudenbush, 1989; Bryk et. al., 1998) • Monitor student performance at a school for a particular grade over years • E.g., collect achievement scores for 3rd grade students attending a school in 1999, 2000, and 2001 • Focus on schools’ improvement over subsequent years

  20. Research Question • To what extent does the choice of the metric matter when the focus is school improvement over time (NCE vs. scale score)? • A Multiple-cohort school productivity model is used as the basis for inferences about school performance

  21. 3-level Hierarchical Model for measuring school improvement Model I : Unconditional School Improvement Model Level-1 (within-cohort) model: Yijt = βjt0 + rijt * βjt0 : estimates of performance for school j (j = 1,.., J) at cohort t (t = 0,1,2,3,4) Level-2 (between-cohort, within-school) model: βjt0 = j0 + j1Timetj + ujt * j0 : status at the first year (i.e., Timetj = 0) or initial status for school j * j1 : yearly improvement / growth rate during the span of time for school j Level-3 (between-school) model: • j0 = 00 + Vj0 * 00 : grand mean initial status • j1 = 10 + Vj1 * 10 : grand mean growth rate

  22. Model II: Student characteristics Level-1 (within-cohort) model: Yijt = βjt0 + βjt1SPEDijt + βjt2LowSESijt + βjt3LEPijt + βjt4Girlijt +βjt5Minorityijt + rijt Level-2 (between-cohort, within-school) model: βjt0 = j0 + j1Timetj + ujt Level-3 (between-school) model: • j0 = 00 + Vj0 • j1 = 10 + Vj1

  23. Model III: Student characteristics & School intervention indicator Level-1 (within-cohort) model: Yijt = βjt0 + βjt1SPEDijt + βjt2LowSESijt + βjt3LEPijt + βjt1Girlij4 +βjt5 Minorityijt + rijt Level-2 (between-cohort, within-school) model: βjt0 = j0 + j1Timetj + ujt Level-3 (between-school) model: • j0 = 00 + 01 LAAMPj + Vj0 • j1 = 10 + 11 LAAMPj + Vj1

  24. Model IV: Full Model Level-1 (within-cohort) model: Yijt = βjt0 + βjt1SPEDijt + βjt2LowSESijt + βjt3LEPijt + βjt1Girlij4 +βjt5Minorityijt + rijt Level-2 (between-cohort, within-school) model: βjt0 = j0 + j1Timetj + ujt Level-3 (between-school) model: • j0 = 00 + 01(%Minorityj) + 02(%LowSESj) + 03(%LEPj) + • 04 LAAMPj + Vj0 • j1 = 10 + 11(%Minorityj) + 12(%LowSESj) + 13(%LEPj) + • 14 LAAMPj + Vj1

  25. Comparison of Key Parameters:NCE vs. Scale Score • Type A effect: includes effects of school policies and practice, educational context, and wider social influences • Type B effect: Includes the effects of tractable policies and practices, but excludes factors that lie outside the control of the school

  26. NCE Results vs. Scale Score Results • School Ranking (rank-order corr.) • School improvement / growth rate parameter (corr. between estimates) • Effect size (par. est. / s.d. of outcome) • Statistical significance of the effect of the school intervention indicator variable

  27. Conclusion • NCE vs. scale score – for the purpose of measuring school improvement under the multiple-cohort design: • Differences are minimal in terms of: • school ranking • school improvement / growth rate • effect size • statistical significance of the effect of the school intervention indicator

  28. Conclusion (cont’d) • Results are consistent across sampling conditions, models, and content area

  29. Demographic

  30. Samples

  31. Reduction in Standard Error (SE) for Average Growth b/t Schools

  32. 100% Sample SE Reduction AVG GROWTH

  33. 85%-70%-55% Samples SE Reduction AVG GROWTH

  34. Reduction in Standard Error (SE) for Average Status b/t Schools

  35. 100% Sample SE Reduction AVG STATUS

  36. 85%-70%-55% Samples SE Reduction AVG STATUS

  37. Reduction in Tau for Average Growth b/t Schools

  38. 100% Sample Tau Reduction AVG GROWTH

  39. 85%-70%-55% Samples Tau Reduction AVG GROWTH

  40. Reduction in Tau for Average STATUS b/t Schools

  41. 100% Sample Tau Reduction AVG STATUS

  42. 85%-70%-55% Samples Tau Reduction AVG STATUS

  43. Program Effect on AVG GROWTH b/t Schools by Sample and Number of Occasions

More Related