1 / 18

Weighing the Evidence:

Weighing the Evidence: How Do Accountability Programs Impact Student Achievement?. Virginia L. Hudson & Gary W. Ritter Office for Education Policy University of Arkansas Christopher J. Lucas, Ronna C. Turner, & Karen Minkel

sheng
Download Presentation

Weighing the Evidence:

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Weighing the Evidence: How Do Accountability Programs Impact Student Achievement? Virginia L. Hudson & Gary W. Ritter Office for Education Policy University of Arkansas Christopher J. Lucas, Ronna C. Turner, & Karen Minkel University of Arkansas Tracy L. Williams Hamline University American Education Finance Association (AEFA), Louisville, KY March 2005

  2. Background • Context: State accountability plans • No Child Left Behind • More studies completed in recent years • Research Question: What is the impact of accountability programs (focused on schools or students) on student performance? • Method: Conducted comprehensive, systematic review of literature

  3. Method: Systematic Review • A review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review. • Sources: • Campbell Collaboration: • http://www.campbellcollaboration.org • Evidence for Policy and Practice Information and Coordination Centre (EPPI-Centre) • http://eppi.ioe.ac.uk

  4. Search Strategy • Electronic databases • Ebsco, Psycinfo, ERIC, Digital Dissertations, ProQuest • Hand-searching journals • Education Policy Analysis Archives (EPAA) • Educational Evaluation and Policy Analysis (EEPA) • Education Next • Journal of Policy Analysis and Management (JPAM) • Review of Educational Research (RER)

  5. Search Strategy • Education Weekarchives • Seeking newsworthy but unpublished studies • Reviewing lists of references from relevant articles • Ongoing

  6. Inclusion Criteria • Conducted and published after 1980 • Focus on impact of statewide or district-wide K-12 accountability plans in U.S.: setting standards, designing related assessments, and imposing consequences • Quantifiable results (no case studies/qualitative inquiries, theoretical analyses, etc.) • At least one outcome measure from among: • reading/language, math, or science exams; graduation rates; promotion/retention rates • No studies focusing solely on population subgroup • Recently added MCTs

  7. Data Extraction & Synthesis • Outcome measures • Results (did accountability affect achievement?) • Methodological weaknesses & criticisms of study designs • Source (peer-reviewed journal or not) • Inclusion decision (met criteria or not) • Study citation • Type of accountability & level of government imposing accountability • Study sample and unit of analysis (state or school) • Time frame • Analytical techniques (simple pre-post? comparison group? multiple regression?)

  8. Search Results

  9. Down to 19 (or 14) • After a thorough review of the 25, reduced studies that were related to accountability, but … • Tested whether state test score changes were aligned with changes on norm-referenced exams • Studies of district accountability systems within states with accountability systems • Crude Quality Measure (5 more) • Treatment N = 1 • Pre-Post growth with no controls

  10. Accountability Types 1990s System-wide Accountability (6) • 3 focus on all states (NAEP), 2 on FL, 1 on Chicago 1990s HS Testing (3) • Amrein/Berliner identifed 18 HS states; 3 studies reviewed these same states First Wave MCT Reforms (5) • Used NELS, identified students in states with minimum competency exams in 1980s (pupil = unit of analysis)

  11. 90s Style Accountability

  12. HS Testing States

  13. MCT First Wave Studies

  14. Outcomes (some overlap) NAEP = 6 Other Exams = 4 Graduation or Dropout Rate = 5 Analytic Methods Pre-Post with CG = 4 Multivariate Models = 10 Preliminary Results = Simple Vote Counting Outcomes either +, neutral, or - Results of 14 HQ Studies

  15. Results: By Accountability Type

  16. Results: By Outcome Measure

  17. Conclusions • Overall Outcome • No evidence that the existing accountability programs are “harming” students in terms of overall student achievement • 10 of 14 studies have positive results • Characteristics • NAEP (and other test scores) tend to show positive results • Grad and Dropout Rate more likely to show negative result • Limitations … next steps • MCT and accountability systems vary • Small set of research studies (for now) • No meta analysis (appropriate?)

  18. For Paper … • Email: • garyr@uark.edu

More Related