120 likes | 216 Views
Update on Planning the Next Generation of GEAR UP Evaluation Studies. 2009 NCCEP/GEAR UP Annual Conference July 19 2009. Three Topics --Update. Background and New Direction Update on RTI facilitated Design Work Discuss Planning Awards. GEAR UP First Evaluation.
E N D
Update on Planning the Next Generation of GEAR UP Evaluation Studies 2009 NCCEP/GEAR UP Annual Conference July 19 2009
Three Topics --Update • Background and New Direction • Update on RTI facilitated Design Work • Discuss Planning Awards
GEAR UP First Evaluation • Evaluations mandated in authorization in 1998 • Westat began 10 year evaluation soon after program authorized—following cohort of 2000-01 7th graders through expected high school graduation year and fall entry into postsecondary • Quasi-experimental—matched comparison • Study analyses and reporting be concluding next year—Middle School report in 2007—High School report in early 2010
Challenges and Issues From First Evaluation • Long period to obtain results/ study attrition • Serious Threats to validity (spill over; treatment/comparison non-equivalencies) • Black box study not telling us about specific practices • Not strong focus on program improvement
New Approach to be Taken • Design next generation studies that will contribute to program improvement • Evaluation and Statistical Standards Based • Responsive to Congress and Accountability Efforts such as PART • Responsive to the needs of practitioners and students served
What would we like the next generation of GEAR UP evaluations to look like? • Partnership (key stakeholders) • Standards Based (feasible, accurate, useful, ethical) • Practice (Grounded in understanding) • Reflection (analysis and synthesis of information from multiple sources-formative assessment) • Innovation/Improvement (start and end)
Standards • Useful—to interested parties (congress, policy makers, practitioners, and ultimately students that the program is intended to benefit • Accurate--Scientific Rigor—evidence based • Feasible—possible to implement study • Proper—meet ethical standards for evaluation work
Rigorous Evaluation Design Requirements • Counterfactual/comparison • Sample representative –external validity • Treatment and control or comparison group are equivalent on dimensions related to outcomes • Treatment and control/comparison group are treated equally except for intervention of interest • Treatment and control are mutually exclusive with regard to the intervention
TRIO HEOA Evaluation Amendments • Prohibit ED requiring projects to deliberately recruit more students than they would normally serve and then denying service for study purposes • Call for working with the applicable institutions’ IRB’s • Call for rigorous studies focused on program improvement and addressing who can most benefit
GEAR UP Design Work • How can we develop designs that are: useful, rigorous, feasible and proper? • What would such designs look like? • How should they best be structured for implementation? • How can we make the best use of the funds for evaluation provided for by Congress?
What Are We Doing ? • Met with GEAR UP Project Director’s in Feb of 2008; July of 2008, February 2009—and now July 2009 • RTI Background work (profiles, systemic review, focus groups, expert papers) • Initiating Planning Awards Solicitation for GEAR UP projects (a little today and more on Monday)