1 / 12

A Close Examination of Policy-Relevant Education Evaluations: Criteria for Judging Quality

A Close Examination of Policy-Relevant Education Evaluations: Criteria for Judging Quality Matthew Linick & Diane Fuselier -Thompson. Implicit vs. Explicit Criteria for Judgments of Program Quality. Explicit criteria for judging program quality: Can be clearly discerned in the text.

susan
Download Presentation

A Close Examination of Policy-Relevant Education Evaluations: Criteria for Judging Quality

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. A Close Examination of Policy-Relevant Education Evaluations: Criteria for Judging Quality Matthew Linick & Diane Fuselier-Thompson

  2. Implicit vs. Explicit Criteria for Judgments of Program Quality • Explicit criteria for judging program quality: • Can be clearly discerned in the text. • “A successful program will display the following characteristics…” • Implicit criteria for judging program quality: • Can be inferred by research questions. • “We will measure various aspects of the program…”

  3. Methodologies of Research Reports • Types of Studies: • Impact and Outcome reports (14) • Implementation reports (12) • Methodologies: • Mixed Method: Interview and Survey (17) • Qualitative: Interview and/or focus group (6), Observation (5) • Quantitative: RCT (4), Quasi-experiment (6), Comparative Statistical Analysis (4)

  4. Explicit Statements of Criteria for judging program quality • Explicit criteria were primarily included in implementation and outcome evaluations • When reports included explicit criteria, program quality was judged along methodological standards • Statistical significance in quantitative studies • Logic Model often used as rubric in implementation studies • Most evaluation reports refrain from making actual judgments of program quality • Authors tend to be uncritical of the evaluated program • Evaluations tend to report findings in lieu of making judgments

  5. Examples of Explicit Criteria used during Program Evaluations • Implementation evaluation: ‘Ending Violence in Schools: A Study of Morton North High School’ • Logic model used as rubric • Evaluators constructed logic model based on relevant research and used this model to evaluate the implementation of violence prevention approaches used by the school • Impact evaluation: ‘Start Reading: Impact Study’ • Statistical significance used as explicit criteria for judging program • Statistically detectable differences between treatment and control schools in using a regression discontinuity • student reading achievement • classroom reading instructional practices • student time engaged with print

  6. Explicit Statements of Criteria for judging program quality • Few explicit statements of criteria: • 9 of 31 reports have explicit statements • Explicit criteria are stated more often when the program is deemed to be successful • 6 of 9 reports with explicit statements were found to be successful

  7. Implicit Statements of Criteria for judging program quality • Frequently Provided as a basis for judging program quality: • Statistical significance was often set as a goal of a research model attempting to estimate the positive impact of a program. • Research questions were used to establish the goal of the study, but the questions often did not contain criteria for making judgments. • Program goals were often referenced as the desired outcomes of the stakeholders or clients, but evaluators usually avoided such statements.

  8. Example of Implicit Criteria used during Program Evaluations • Outcomes evaluation: ‘Extended School Day Program’ • Evaluators framed evaluation questions as research questions • What are the outcomes for students, teachers, and schools in this program? • What were the effects on test scores, attendance, teacher attitudes, etc.?

  9. Implicit Statements of Criteria for judging program quality • Many of the reports imply that stakeholder expectations are a guiding principle for program ‘quality’. • Implicit Criteria for Program Quality in each of the 31 reviewed reports. • Not easily discernable (found in discussion of results in 20/31 reports) • Implied criteria tied to stakeholder expectations (25/31 reports)

  10. Implicit Statements of Criteria in Reports • Examining the Program/Quality Criteria/Methodology. • Implicit criteria reflects stakeholders’ desired outcomes. • Desired outcomes influence methodological choices. • Methodological choices influence the criteria used to judge program quality.

  11. How do Statements of Criteria relate to Methodology Used?

  12. Questions, Comments, or Praise? • Contact Information: • Matt Linick • mlinic1@illinois.edu • Diane R. Fuselier-Thompson • diat@illinois.edu

More Related