Beyond Triage: A Randomized Experiment in Sustained Pre-College Advising

1. PRELIMINARY. PLEASE DO NOT CITE OR TWEET WITHOUT AUTHORS’ PERMISSION Beyond Triage: A Randomized Experiment in Sustained Pre-College Advising Eric P. BettingerBrent J. Evans With Anthony Antonio, jesse foster, Eileen Horng, riekijima UC Davis – 15 March 2015

2. Motivation • Non-financial barriers related to information deter students from preparing for and enrolling in higher education, especially among low-income and minority students (Avery & Kane, 2004) • Gauntlets • Financial aid (ACSFA, 2005) • Admission process (Klasik, 2012) • High school counselors are overloaded and cannot meet the demand in their schools. • Huge investment by college access programs to provide information and guide students through the process • TRIO (> $800 million) • Countless small local programs • Models vary substantially

3. Literature • Information and guidance matter at very specific stages in the process • H&R Block Study (Bettinger, Long, Oreopoulos, & Sanbonmatsu, 2012) • Information and guidance related to financial aid improves aid receipt and college enrollment • 2-8 percentage point effect • Summer Melt Studies (Castleman, Page, & Schooley, 2014; Castleman, Arnold, & Wartman, 2012) • Outreach to students in the summer between high school completion and college enrollment improves likelihood of enrolling • 3-14 percentage point effect • Holistic advising • Mathematica’s Upward Bound Study (Myers et al., 2004) • Primarily academic supports to promote college enrollment, but also provide information • RCT found no effects on college enrollment although some evidence of shift from two-year to four-year • College Possible (Avery, 2013) • Targeted ACT prep, information, and admission support • High-touch intervention-360 hours • Large effects (15 pp) on shifting from two-year to four-year college but no overall enrollment effects • Dartmouth Outreach (Carrell and Sacerdote, 2013)

4. Need for evidence of different models • Different models of providing information and guidance • Targeted versus school wide • In school versus out of school • Near-peer • By investigating wide range of models, we can start to tease apart mechanisms • Need well identified, causal estimates of efficacy • Randomized controlled trial evidence should be privileged where possible

5. Advise TX • Part of the College Advising Corps model (active in 15 states 350-400 high schools) • Goal is to help low-income, first-generation college students attend and succeed in college • Recent college graduates serve as full-time near-peer advisers to entire student body at a high school • Full school college advising intervention-concentrate on seniors but some time spent developing college plans for underclassmen • Encourage attendance; provide information; assist with selection of colleges, college applications, and FAFSA completion • Program pilot in 2010-2011 in 15 schools with rapid expansion to over 120 schools in 2011-2012 school year

6. Experimental Design • School level randomization • Schools invited to apply • > 35% FRL • < 70% graduates attend college within one year (average is 45% in 2009) • < 55% students undertaking a “distinguished” college-prep curriculum • Schools were ranked on these 3 criteria and a qualitative “fit” component • 84 schools automatically selected among 237 who applied • Next 111 schools eligible for random assignment (experimental sample) • 36 randomly selected for treatment assignment, others assigned control status (no program) • Blocked on region: individual lotteries were held within each region (23)

7. Data • School level data from Texas Education Agency (TEA) • Student level data from Texas Higher Education Coordinating Board (THECB) • 2 years of enrollment outcome data • From 2011-2012 (2012) and 2012-2013 (2013) high school graduating classes • All in-state public postsecondary institutions • Our results are lower bounds of effect of program if the program had any effect on out of state or private enrollments • Outcomes: 2-year and 4-year enrollment and enrollment in Fall after high school graduation Preliminary results from student surveys conducted in treatment/control schools

8. Treatment-Control Balance

9. Descriptive Statistics & Balance • Relative to all TX high schools, experimental schools • Are much larger (1683 to 728) • Have a higher percentage of low-income status (64% to 54%) • Have a higher share of minority students (77.5% to 60%) • Have same graduation rate (80%) • Treatment schools relative to control schools • Have more Black and less Hispanic students (8 percentage points difference) • Have slightly more Asian students (1 percentage point difference) • Same graduation rates and college enrollment rates • We control for race in all models to account for this imbalance

10. Compliance • 5 of 36 treatment schools declined to participate or eventually left the program • 9 of 75 control schools eventually received treatment • Although we had a randomly determined waitlist, it was not consistently adhered to by program staff • E.g. Austin loved the program and offered to pay for all control and treatment schools to participate after the first year. • Treatment on the treated estimates are approximately 33% higher than Intent to treat estimates

11. Main Impacts

12. Intent to treat estimates • 1.2 – 1.5 pp impacts on enrolling in fall after high school completion (significant at 10% level) • Impacts are observed in first year of treatment • enrollment effects on two-years become more significant if just examine first year • If we pool the data, small and insignificant effects (~ 1 pp) on overall enrollment • Concentrated at 2-years • Point estimates at 4-years close to 0 • All second year impacts attenuated

13. Impacts on Subgroups

14. Subgroup Analysis • Impact on Low-income students • More likely to enroll overall, split between two-years and four-years (2-3 pp) • Impact on Hispanic students • More likely to enroll overall, mostly at two-years (2-3 pp) • Impact on Black students • Less likely to enroll in a two-year (3 pp) • Some evidence of substitution into four-year, but four-year enrollments not significant • Impact on Female students • Less likely than men to enroll in four-year and start at a four-year in fall after high school (1.6 pp)

15. Survey Sample • Surveys conducted in spring 2014 • All treatment schools (36 + additional randomized group from 2013) • 42 control schools • Treatment/Control Balance • Less representation among Hispanic students (as in the administrative data) • Surveys conducted in spring 2014 • All treatment schools (36 + additional randomized group from 2013) • 42 control schools

16. Preliminary Survey Results • Key Differences • Degree expectations improve • Plans to work full- or part-time after college or to enter the military decrease • Similarly likely to participate in “soft” college prep activities (college visits, website, test prep) • More applications submitted and greater likelihood of receiving help • More likely to attempt AP exams or to retake college entrance exams • More likely to complete FAFSA (self-reported) • More likely to have submitted a deposit • Less likely to have talked to a high school counselor or teacher about admissions

17. Discussion • Unsurprising that treatment effects are small at the school level • Advisers likely spend time with inframarginal students who are already going to college as opposed to targeting support at students who need it most • Degree of complementarity is potentially important part of the story • Cost-benefit analysis • Scalability • Future work • Will examine persistence outcomes to observe whether advisers have effect on “fit”