2014 Welfare Research and Evaluation Conference Washington, D.C.

1. A Statistical Portrait of Populations in Nine Career Pathways Programs: Preliminary Findings David J. Fein With David Judkins, Azim Shivji, & Peter Honnef Abt Associates, Inc 2014 Welfare Research and Evaluation Conference Washington, D.C.

2. What is ISIS? • Random assignment evaluation of nine career pathways programs • Programs vary, but all promote access to and completion of one or more post-secondary education steps, target economically disadvantaged youth & adults • Nine separate studies set within a common conceptual framework • ISIS is sponsored by the Office for Planning, Research and Evaluation at ACF • With support for programs from Open Society Foundations, Joyce Foundation, Kresge Foundation, Meadows Foundation, OFA/ACF (3 HPOG sites) • RA ends late 2014; early impact results in 2015-16 • Today preview preliminary findings from a forthcoming working paper on the ISIS sample

3. Programs ISIS Is Testing • Des Moines Area Community College (DMACC) “WTA Connect” • Integrated Basic Education and Skills Training (I-BEST) Program (Washington State: Bellingham, Everett, Whatcom colleges) • Instituto del Progreso Latino, Chicago, “Carreras en Salud” • Madison Area Technical College (MATC) Patient Care Pathway Program • Pima Community College “Pathways to Healthcare Program” (Tucson, AZ) • San Diego Workforce Partnership (SDWP) “Bridge to Employment in the Healthcare Industry” • Workforce Development Council of Seattle-King County (WDC) “Health Careers for All” • Valley Initiative for Development and Advancement (VIDA) Program, South Texas • Year Up (8 offices nationally)

4. Paper Addresses Three Descriptive Questions • Who volunteers for diverse ISIS programs? • Basic characteristics, measured through forms administered at study intake • To what degree does this population access post-secondary training on its own? • Analyze patterns of college enrollment and persistence in the control group • What characteristics predict college enrollment and persistence absent ISIS programs? • “Road test” ISIS baseline covariates (various implications for later impact analysis) • Identify potential mediators and subgroups that programs may want to target

5. Baseline Characteristics of ISIS Sample Enrolled Through March 2014

6. Baseline Characteristics of ISIS Sample Enrolled Through March 2014 (Cont.)

7. Other Characteristics • Working paper provides similar comparisons in other domains of career pathways theory of change: • Psycho-social factors • Career knowledge • Time constraints • Personal and life challenges • Explores how characteristics in these domains predict college enrollment and persistence

8. College Outcomes by Site • Most ISIS programs focus on college, a few promote college as well as other training options • College outcomes for control group  “where is the bar”? • Analyze college records from match to the National Student Clearinghouse • The only national source of individual-level data on college enrollments and completions • Covers >90% enrollments nationally • Captures enrollments and completions • Under-coverage of students: at for-profits, refusing to share records, subject to matching error • Little/no evaluation of data quality for: certificates, non-credit enrollments, FT/PT status

9. Percent of Control Group Members Ever Enrolled in College Since Random Assignment

10. Average Total Months Control Group Members Enrolled in College Since Random Assignment

11. What Characteristics Predict College Enrollment and Persistence Absent ISIS? • Focus on key substantive domains in career pathways theory of change • Sample: control group pooled X-site, RA’d by 12/13 • Two outcomes (measured through 3/14) • Any college enrollment after random assignment (logistic regression, n=3,018) • Number of months enrolled after random assignment (linear model, n=1,387 ever-enrolled) • Adding each domain as a set to base model (site dummies, months exposed, demographic characteristics—estimates not shown) • Bold = estimate statistically significant at p<.10

12. Estimates for Educational Background

13. Estimates for Psycho-Social Factors and Career Knowledge

14. Estimates for Financial and Time Constraints

15. Estimates for Life Challenges

16. Preliminary Highlights (I) • Factors predicting both enrollment and persistence • Good high school grades (+), 1+ yr college (+), self-confidence (+), stress (-), low income (-), expected work hours (-) • Factors more predictive of enrollment • <1 year college/AA (+), training commitment (+), career knowledge (+), financial hardship (-), current work hours (-), substance abuse (-) • Factors more predictive of persistence • Parents attended college (+), <1 year college/vocational certificate (-), academic discipline (+), social support (-?), depression (+?), gov’t assistance (-), child care, transportation, illness (-), substance abuse (+?)

17. Preliminary Highlights (II) • Most findings in expected direction, a few unexpected • “Kitchen sink” models (not shown) leave site differences mostly unexplained • Unmeasured influences (local economy, training opportunities…)? • Specification and measurement error?

18. Summary • ISIS programs target diverse economically disadvantaged populations • Control groups vary substantially in college enrollment and persistence after random assignment—fairly high bar in several sites • Implications for targeting and program focus • More in paper (psychometrics, characteristics*site, college outcome detail) • Suggestions welcome!

19. For Further Information • David Fein (Abt), david_fein@abtassoc.com • Brendan Kelly (ACF), brendan.kelly@acf.hhs.gov • Molly Irwin, (ACF), molly.irwin@acf.hhs.gov