1 / 29

A Systemic Review of Evaluations of Pre-College Programs and Services

A Systemic Review of Evaluations of Pre-College Programs and Services. Elizabeth Glennie May 13, 2009. Systemic Review: Purpose. To provide a systemic description of research being conducted on services designed to increase the chances that disadvantaged students attend college

ginger
Download Presentation

A Systemic Review of Evaluations of Pre-College Programs and Services

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Systemic Review of Evaluations of Pre-College Programs and Services Elizabeth Glennie May 13, 2009

  2. Systemic Review: Purpose • To provide a systemic description of research being conducted on services designed to increase the chances that disadvantaged students attend college • Study focus – descriptive map of literature • Methodology – the in-depth review

  3. Systemic Review: Studies • To find studies, we • Asked for evaluation reports • Conducted key word searches for specific programs, such as GEAR UP or Upward Bound • Conducted key word searchers for general terms such as “bridge program” • Searches yielded over 1,400 reports • Focused on most relevant ones

  4. Systemic Review: Filtering • For the descriptive map of the literature, studies had to • Examine program or service specifically designed to prepare middle/high school students for college entrance and success • Examine program designed to serve disadvantaged students • Be an empirical study with outcomes • Applying these filters left 158 studies in the descriptive map of the literature • 61 (39%) of these focused on GEAR UP

  5. Systemic Review: In-depth Review • For in-depth review, we focused on studies that • Had a comparison group • Included numerical estimates/evidence of student outcomes with intervention as an explanatory measure • Provide estimates of the statistical association between the practice and the student outcomes • Applying these criteria left 53 studies in the in-depth review • 14 of these (26%) focused on GEAR UP

  6. Systemic Review: Filtering Process

  7. In-depth review documents by grade level served

  8. In-depth review documents by type of study design

  9. In-depth review documents by type of study

  10. Distribution of in-depth review documents by type of data source

  11. In-depth review documents by type of comparison method

  12. In-depth review documents by subgroup analysis status and type of subgroup analysis

  13. In-depth review documents by type of outcome examined

  14. Descriptive Map - Outcomes • Note that in the descriptive map (158 studies) • 90% had student-level outcomes • 29% had parent-level outcomes • 14% had teacher-level outcomes • 30% had school-level outcomes

  15. Approaches to Studies of Different Outcomes • Studies examined a range of student outcomes • Highlight a few of them today • Grades • Math scores • High School Completion • Postsecondary institution attendance • Counted each analysis separately – studies can have more than one analysis

  16. Distribution of STUDENT GRADE outcomes analyzed in in-depth review by data source (N = 35 analyses)

  17. Distribution of STUDENT GRADE outcomes analyzed in in-depth review by control group selection method (N = 35)

  18. Distribution of STUDENT GRADE outcomes analyzed in in-depth review by analysis method (N = 35)

  19. Distribution of STUDENT MATH SCORE outcomes analyzed in in-depth review by data source (N = 13 analyses)

  20. Distribution of STUDENT MATH SCORE outcomes analyzed in in-depth review by control group selection method (N = 13)

  21. Distribution of STUDENT MATH SCORE outcomes analyzed in in-depth review by analysis method (N = 13)

  22. Distribution of STUDENT HIGH SCHOOL COMPLETION outcomes analyzed in in-depth review by data source (N = 32 analyses)

  23. Distribution of STUDENT HIGH SCHOOL COMPLETION outcomes analyzed in in-depth review by analysis method ( N =32)

  24. Distribution of STUDENT HIGH SCHOOL COMPLETION outcomes analyzed in in-depth review by control group selection method (N = 32)

  25. Distribution of STUDENT COLLEGE ENROLLMENT outcomes analyzed in in-depth review by data source (N = 25 analyses)

  26. Distribution of STUDENT COLLEGE ENROLLMENT outcomes analyzed in in-depth review by control group selection method (N = 25)

  27. Distribution of STUDENT COLLEGE ENROLLMENT outcomes analyzed in in-depth review by analysis method ( N =25)

  28. Conclusions • Range of approaches to studies of these programs • About 1/3 used comparison groups of convenience without matching on or controlling for factors such as past performance • Twenty percent are cross sectional • Creation of statewide student databases may make it easier to follow students from middle school to high school to college

  29. Next Steps • Report with results for the descriptive map of the literature and the in-depth review • Calculating the weight of evidence for these studies

More Related