290 likes | 443 Views
A Systemic Review of Evaluations of Pre-College Programs and Services. Elizabeth Glennie May 13, 2009. Systemic Review: Purpose. To provide a systemic description of research being conducted on services designed to increase the chances that disadvantaged students attend college
E N D
A Systemic Review of Evaluations of Pre-College Programs and Services Elizabeth Glennie May 13, 2009
Systemic Review: Purpose To provide a systemic description of research being conducted on services designed to increase the chances that disadvantaged students attend college Study focus – descriptive map of literature Methodology – the in-depth review
Systemic Review: Studies • To find studies, we • Asked for evaluation reports • Conducted key word searches for specific programs, such as GEAR UP or Upward Bound • Conducted key word searchers for general terms such as “bridge program” • Searches yielded over 1,400 reports • Focused on most relevant ones
Systemic Review: Filtering • For the descriptive map of the literature, studies had to • Examine program or service specifically designed to prepare middle/high school students for college entrance and success • Examine program designed to serve disadvantaged students • Be an empirical study with outcomes • Applying these filters left 158 studies in the descriptive map of the literature • 61 (39%) of these focused on GEAR UP
Systemic Review: In-depth Review • For in-depth review, we focused on studies that • Had a comparison group • Included numerical estimates/evidence of student outcomes with intervention as an explanatory measure • Provide estimates of the statistical association between the practice and the student outcomes • Applying these criteria left 53 studies in the in-depth review • 14 of these (26%) focused on GEAR UP
Distribution of in-depth review documents by type of data source
In-depth review documents by subgroup analysis status and type of subgroup analysis
Descriptive Map - Outcomes • Note that in the descriptive map (158 studies) • 90% had student-level outcomes • 29% had parent-level outcomes • 14% had teacher-level outcomes • 30% had school-level outcomes
Approaches to Studies of Different Outcomes • Studies examined a range of student outcomes • Highlight a few of them today • Grades • Math scores • High School Completion • Postsecondary institution attendance • Counted each analysis separately – studies can have more than one analysis
Distribution of STUDENT GRADE outcomes analyzed in in-depth review by data source (N = 35 analyses)
Distribution of STUDENT GRADE outcomes analyzed in in-depth review by control group selection method (N = 35)
Distribution of STUDENT GRADE outcomes analyzed in in-depth review by analysis method (N = 35)
Distribution of STUDENT MATH SCORE outcomes analyzed in in-depth review by data source (N = 13 analyses)
Distribution of STUDENT MATH SCORE outcomes analyzed in in-depth review by control group selection method (N = 13)
Distribution of STUDENT MATH SCORE outcomes analyzed in in-depth review by analysis method (N = 13)
Distribution of STUDENT HIGH SCHOOL COMPLETION outcomes analyzed in in-depth review by data source (N = 32 analyses)
Distribution of STUDENT HIGH SCHOOL COMPLETION outcomes analyzed in in-depth review by analysis method ( N =32)
Distribution of STUDENT HIGH SCHOOL COMPLETION outcomes analyzed in in-depth review by control group selection method (N = 32)
Distribution of STUDENT COLLEGE ENROLLMENT outcomes analyzed in in-depth review by data source (N = 25 analyses)
Distribution of STUDENT COLLEGE ENROLLMENT outcomes analyzed in in-depth review by control group selection method (N = 25)
Distribution of STUDENT COLLEGE ENROLLMENT outcomes analyzed in in-depth review by analysis method ( N =25)
Conclusions Range of approaches to studies of these programs About 1/3 used comparison groups of convenience without matching on or controlling for factors such as past performance Twenty percent are cross sectional Creation of statewide student databases may make it easier to follow students from middle school to high school to college
Next Steps Report with results for the descriptive map of the literature and the in-depth review Calculating the weight of evidence for these studies