1 / 52

Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council

Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council. Dr. Robert H. Meyer Research Professor and Director Value-Added Research Center University of Wisconsin-Madison February 25, 2011. Attainment and Gain.

arich
Download Presentation

Value-Added Systems Presentation to the ISBE Performance Evaluation Advisory Council

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Value-Added SystemsPresentation to the ISBE Performance Evaluation Advisory Council Dr. Robert H. Meyer Research Professor and Director Value-Added Research Center University of Wisconsin-Madison February 25, 2011

  2. Attainment and Gain • Attainment – a “point in time” measure of student proficiency • compares the measured proficiency rate with a predefined proficiency goal. • Gain –measures average gain in student scores from one year to the next

  3. Attainment versus Gain Gain Gain Gain Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

  4. Growth: Starting Point Matters Reading results of a cohort of students at two schools Grade 4 Proficient Cutoff 438 Grade 5 Proficient Cutoff 463 *Scale Score Average is below Proficient Example assumes beginning of year testing

  5. Value-Added • A kind of growth model that measures the contribution of schooling to student performance on the standardized tests • Uses statistical techniques to separate the impact of schooling from other factors that may influence growth • Focuses on how much students improve on the tests from one year to the next as measured in scale score points

  6. Value-Added Model Definition • A value-added model (VAM) is a quasi-experimental statistical model that yields estimates of the contribution of schools, classrooms, teachers, or other educational units to student achievement, controlling for non-school sources of student achievement growth, including prior student achievement and student and family characteristics. • A VAM produces estimates of productivity under the counterfactual assumption that all schools serve the same group of students. This facilitates apples-to-apples school comparisons rather than apples-to-oranges comparisons. • The objective is to facilitate valid and fair comparisons of productivity with respect to student outcomes, given that schools may serve very different student populations.

  7. A More Transparent (and Useful) Definition of VA • Value-added productivity is the difference between actual student achievement and predicted student achievement. • Or, value-added productivity is the difference between actual student achievement and the average achievement of a comparable group of students (where comparability is defined by a set of characteristics such a prior achievement, poverty and ELL status).

  8. In English Post-on-Pre Link x Pretest Posttest = Student Characteristics School Effects Unobserved Factors + + + Value Added

  9. VARC Philosophy • Development and implementation of a value-added system should be structured as a continuous improvement process that allows for full participation of stakeholders • Model Co-Build; Complete customization • Analysis • Reporting • Value–added is one tool in a toolbox with multiple indicators

  10. VARC Value-Added Partners • Design of Wisconsin State Value-Added System (1989) • Minneapolis (1992) • Milwaukee (1996) • Madison (2008) • Wisconsin Value-Added System (2009) • Milwaukee Area Public and Private Schools (2009) • Racine (2009) • Chicago (2006) • Department of Education: Teacher Incentive Fund (TIF) (2006 and 2010) • New York City (2009) • Minnesota, North Dakota & South Dakota: Teacher Education Institutions and Districts (2009) • Illinois (2010) • Hillsborough County , FL (2010) • Broward County, FL (2010) • Atlanta (2010) • Los Angeles (2010) • Tulsa (2010)

  11. Districts and States working with VARC Minneapolis Milwaukee Madison Racine Chicago New York City Los Angeles Tulsa Atlanta Hillsborough County Broward County

  12. Measuring knowledge • Many factors influence what a student learns and how their knowledge is measured • A variety of measures, including (but not limited to) assessments, tell us what a student knows at a point in time. • What are some ways we measure knowledge?

  13. Measuring knowledge End-of-course Exam Diagnostic Test MAP WKCE Daily Journal Unit Project After-school Activities Hands-on Project

  14. The Simple Logic of Value-Added Analysis • School Value-Added Report • School specific data • Grade level value-added • Comparison Value-Added Reports • Compare a school to other schools in the district, CESA, or state • Also allows for grade level comparisons • Tabular Data available for School Report and Comparison Reports

  15. Attainment and Value-Added

  16. How complex should a value-added model be? • Rule: "Simpler is better, unless it is wrong.“ • Implies need for “quality of indicator/ quality of model” diagnostics.

  17. Model Features • Demographics • Posttest on pretest link • Measurement error • Student mobility: dose model • Classroom vs. teacher: unit vs. agent • Differential effects • Selection bias mitigation: longitudinal data • Test property analysis

  18. MAP vs. ISAT • MAP dates: September, January, May • MAP: uses Rasch equating • ISAT: 3PL • MAP: slightly higher reliability - ~0.96 in math, ~0.94 in reading • ISAT math ~0.93, reading ~0.9 • Cut scores on MAP are determined by equipercentile equating to ISAT

  19. Minimal correlation between initial status and value-added

  20. Grade-Level Statewide Results

  21. Grade-Level Statewide Results

  22. Grade-Level Statewide Results

  23. MPS and MMSD Value-Added compared to Wisconsin 6th to 7th Grade (Nov 2006 – Nov 2007) Mathematics – State VA Model School Effects MPS School Effects MMSD School Effects School/District VA Productivity Parameters in WKCE Scale Score Units (Relative to State)

  24. Visit the VARC Website http://varc.wceruw.org/ for more information about VARC and value-added

More Related