2012-13 Montana GEAR UP Evaluation Resutls

1. 2012-13 Montana GEAR UP Evaluation Resutls January 15, 2013

2. Research Question 1: What services do GEAR UP schools implement with program funds? 1. Montana GEAR UP offers a full range of services that address all three GEAR UP goals. The average hours for students receiving GEAR UP services ranged from two hours (financial aid counseling) to over 80 hours (rigorous academic curriculum). 2. There is a core group of five key services on which schools spend the majority of funds, spend the majority of time implementing, and reach the largest numbers of students. These services are: • College Visits • Tutoring and homework assistance • Academic enrichment activities or STEM activities • Summer programs • Counseling, advising and academic planning

3. And what is the level of implementation of those services? 1. There is wide range in the quality of implementation of GEAR UP Services, from fair to very good. Overall, based on student and parent survey ratings and evaluator site visits, data suggest that most services had an average level of implementation. 2. The service that appears to have the overall strongest implementation is college visits, which is best described as having an above average implementation.

4. Research Question 2. To what extent do GEAR UP schools implement college & career readiness best practices? 1. Based on the educator survey, GEAR UP schools appear to have a fairly high level of college and career readiness best practices in all three practice areas of academic preparation, college aspirations and expectations, and (for high schools) college knowledge and taking action. On a seven point scale, most items and subscales had average ratings in the 5.0 to 6.0 range…somewhat agree to agree. 2. But there are some areas, where there is room for growth in the implementation. For example, many schools struggle with finding role models and mentors to help GEAR UP students, particularly our American Indian students, navigate the road to college access and success. 3. Together, the data suggest that the real value of the educator survey results is for state program leaders and individual school leaders to look at the relative results, targeting those practice areas and individual practices for improvement that are rated lower than others.

5. Research Question 3. Do GEAR UP activities achieve the following program goals and the Montana state performance indicators under each goal? • Montana GEAR UP established three goals and a total of 21 indicators to measure the success of the program. Currently data are available 11 of these measures to assess progress from one year to the next. • The available results are mixed at this early stage in program implementation. For every annual benchmark that Montana GEAR UP met for an indicator, Montana GEAR UP did not meet the annual benchmark for a similar indicator. • On the positive side, Montana GEAR UP has already achieved the final benchmarks for two indicators regarding the percent of students who take the PLAN and EXPLORE.

6. School Indicator Flags: Examines selected state performance indicators and their relationship • MTGU & PRA developed a few School Indicator Flags to more accurately gauge student performance by identifying consistencies and discrepancies of student performance on different indicators. • Based on a series of School Indicator Flags, Montana GEAR UP students seem to perform better on indicators that measure local standards, such as course completion, than on indicators that measure state and national standards, such as performance on state CRT and national ACT testing. • For example, Montana GEAR UP students increasingly are taking courses that align with the Montana University System college preparatory curriculum, but they are also outperformed on state and national assessments, such as the ACT, that can point to college readiness and success.

7. Research Question 4. Are GEAR UP services related to achievement of program goals and outcomes for all students and student subgroups, particularly Native American students? • The evaluation examined the relationships between the amount of time that students and parents participated in GEAR UP services with predicted student performance on short-term outcomes. • The results found positive associations between the set of Goal 1 services—such as academic tutoring and homework assistance, summer credit recovery, and academic enrichment activities—with Goal 1 outcomes—student completion of pre-Algebra in grade 8 and of Algebra I in grade 9. • The evaluation also found other positive associations between increased hours of participation in Goal 2 services—counseling/advising/academic planning, college visits, and student workshops—and Goal 2 outcomes, such as being on track to meet MUS college preparation requirements.

8. Research Question 4 (Cont) • The evaluation found fewer relationships between Goal 3 associated services—such as financial aid advising, parent workshops on college preparation, and family events—with Goal 3 associated outcomes, such as expectations about student enrollment in college, knowledge about financial aid, and FAFSA completion. • The key finding for Goal 3 was the number of hours that parents spent in Goal 3 services was positively related to FAFSA completion. • For the most part, all of the positive results across the three goals held equally true regardless of student gender or ethnicity. However, the positive results were not consistent across all grade levels measured.