APHSA Members Webinar September 26, 2013

1. The Innovative Strategies for Increasing Self-Sufficiency (ISIS) Evaluation APHSA Members Webinar September 26, 2013

2. Introduction • Purpose of webinar • Introduction of speakers • Review of agenda • ACF Perspective • Evaluation Summary • Evaluation Partner: Workforce Development Council of Seattle-King County • Q & A

3. What Is ISIS? • ISIS – the Innovative Strategies for Increasing Self-Sufficiency evaluation – is an evaluation of promising strategies for increasing education, training, employment and self-sufficiency among low-income families. • Impact Study using Random Assignment • Implementation Study • Cost-Benefit Analysis • Evaluation by the Office of Planning, Research and Evaluation (OPRE) in the Administration for Children and Families (ACF) • Led by Abt Associates, in partnership with MEF Associates, APHSA, NCSL, NGA, consultants, Public Strategies, Inc. and the University of Michigan. • Support for programs provided by the Open Society Foundations, The Joyce Foundation, The Kresge Foundation, The Meadows Foundation and ACF/OFA Health Professions Opportunity Grants. • 10-year project, currently in the random assignment phase (year 6)

4. Why is ACF and OPRE evaluating career pathways? • OPRE is the principal research and evaluation office of ACF. • OPRE is to serve as the “principal advisor” to the Asst Secretary on research and evaluation matters. • Our research covers the entirety of ACF’s program areas (though not evenly). • Our research and evaluation activities are guided by the principles of rigor, relevance, transparency, independence and ethics. • We have a long history of conducting experimental research, particularly in welfare to work and employment and training programs (see new book by Gueron and Rolston). • Our approach to research and evaluation has also taken on greater prominence and importance in the current Obama Administration and OMB’s emphasis on “evidence-based policy-making.”

5. Poverty and Pathways:The Motivation for ISIS • 3 Key “Facts” • Previous research has demonstrated effectiveness in moving families off cash assistance and into employment. (NEWWS, GAIN, New Hope, MFIP, SSP, state waiver experiments, etc.) • Economic mobility is increasingly “skill-biased.” • Individuals with lower skills/lower income have fewer financial and human capital resources from which to draw on and make productivity investments. • Money is a major constraint, but time is also an important constraint. • 2 Important Limitations of Existing Research: • Education and training has not been as promising or financially beneficial for poor families as it has for the population as a whole. • Families were not substantially better off even in the successful interventions (large numbers remain poor.) • 1 Major Challenge • How can we integrate education, workforce development and human services to create pathways out of poverty and increase self-sufficiency for a large and diverse set of American families?

6. Emerging Evidence: Career Pathways and Sector-Based Strategies Year Up (Economic Mobility Corp, 2010) • National org provides 6 months customized training + 6 month paid internship in IT and finance sectors w/local colleges, completers earn 14+ credits • Results from small initial experiment (n=164) Two key questions: Do future earnings recoup those foregone while in training? Do the benefits exceed the costs?

7. Emerging Evidence: Career Pathways and Sector-Based Strategies Sectoral Training Strategies (Maguire et al. 2010) • 3 experienced CBOs provide customized short-term training in varied high-demand fields • Disadvantaged adults with HS+ education • Careful screening • Total sample n=1,014

8. The ISIS Evaluation • ISIS aims to learn about promising strategies for increasing employment and self-sufficiency among low-income individuals and families • Core principles • Collaborate with stakeholders and partners to Identify most relevant interventions and strategies • Semi-structured discussions with over 250 individuals (key APHSA role) • Test a few things well • Experimental design with random assignment • Test one type of intervention in nine sites, rather than nine disparate interventions • Aim for positive influences on social policy and practice

9. Key Career Pathways Ideas • A series of connected education and training programs and support services • Addresses the wide range of skill and other needs of low-skilled individuals • Provide credentials valued in high demand occupations/sectors • Enables individuals to secure employment within a specific industry or occupational sector, and to advance to successively higher levels of education and employment within that sector • Each step is designed to prepare the participant for the next level of employment and education • Build effective partnerships • Education and training providers, social service providers, workforce

10. V. BA+ Programs Upper-Skilled Jobs IV. 1-2-Year Certificate to AA Programs Mid-Level Skilled Jobs III. Short-Term Certificate ProgramsEntry-Level Skilled Jobs II. Sectoral Bridge Programs Semi-Skilled Jobs I. Basic Bridge Programs The Basic Career Pathways Model Prospects for good-paying, stable employment Occupational, academic, and life skills

11. Key Career Pathways Services • Comprehensive assessment • Academic and non-academic skills • Basic and technical skills instruction • Modularization, contextualization, acceleration, flexible delivery, active learning • Supports • Proactive advising and guidance, supplemental instruction, social supports, supportive services, financial assistance • Employment connections • During and after training

12. Initial Targeting & Placement Decisions Theory of Change for Career Pathways Participant Characteristics • Demographic • Educational • Economic Take First/ Next Step In Career Pathway/Lattice Increase Performance & Persistence in Training Foundational Academic Skills • Certificate/Diploma • 2- year, 4- year Degree Occupational Skills Comprehensive Assessment TO NEXT STEP Psycho-social Factors Improve Performance & Advancement in Jobs Core Curriculum •  Earnings •  Benefits •  Job security Career Orientation and Knowledge Supports Employment Connections Resource Constraints Improve Other Outcomes Other Personal and Family Challenges • Income & assets • Child & adult well being • Local economic growth Contextual Factors: Institutional, Economic, Social Primary Outcomes Program Inputs Intermediate Outcomes

13. The ISIS Sites

14. Health Professions Opportunity Grants (HPOG) • In 2010, the HPOG program was created as part of the Affordable Care Act in order to provide TANF recipients and other low-income individuals with access to jobs that pay well and address workforce needs in health professions. • $67 million a year to 32 grantees across 23 states, including 5 tribal grantees. • Program is administered by the Office of Family Assistance (OFA). • Research and evaluation overseen by OPRE.

15. Health Professions Opportunity Grants (HPOG) • To evaluate HPOG ACF has implemented a multi-pronged research strategy that includes: • Implementation, systems and outcomes evaluation for HPOG TANF/low income grantees • Evaluation of Tribal HPOG • Performance tracking: HPOG PRS collects uniform data across all grantees • Impact studies in a subset of grantees • HPOG Impact Studies • Innovative Strategies for Increasing Self-Sufficiency (ISIS) • University-based field initiated research

16. The Sample of ISIS Sites • Career pathways is both a framework and a model for particular programs – many variations incorporating basic ideas possible • 9 ISIS programs exemplify central career pathways ideas, but vary greatly in: • Basic elements such as target populations (e.g., TANF), occupations and training steps, provider partnerships • Lead organization—CBOs (3), Community/Technical Colleges (4) and WIBs (2) • Key strategies included; their design, intensity, and duration

17. Who is participating in ISIS?

18. Who is participating in ISIS?

19. Who is participating in ISIS?

20. ISIS Program Diversity

21. ISIS Research Questions • What is the impact of the program on persistence in education and achievement of credentials and degrees? • What is the impact of the program on career-track employment and earnings? • What is the impact on well-being?

22. Study Components ISIS will include site-specific: • Impact study • Using baseline data, administrative data (UI, college records), surveys • Implementation study • Interviews, staff surveys and site-level administrative data • Cost-benefit study Study parameters: • Randomly assign a minimum of 1,000 eligible individuals to treatment and control groups • Control group can access other service in the community

23. Study Deliverables • Evaluation design • Program profiles (2013) • Cross-program implementation report • Program-specific 15 month reports • Implementation • Impact • Cost

24. ISIS Program Health Careers for All Workforce Development Council of Seattle-King County

25. What is HCA? • Health Careers for All is funded by a 5-year HHS HPOG grant • Healthcare training via cohorts or individual training support • Career and education “navigation” support • Efforts to better align systems that serve the target population (workforce, education, social service)

26. Outcome Targets for HPOG Over the 5 years of the grant period: • Enroll 920 TANF recipients and other low-income adults and youth • Achieve a training completion rate of 70% (benchmark = 55%) • Achieve a training-related placement rate of 60% (benchmark = 25%) • Achieve a return-to-training rate of 25% (benchmark = 11%)

27. Outcomes to Date Over life of the HPOG grant • Nearly 600 customers enrolled • Approximately 300 customers completed healthcare training • More than 150 customers employed • Approximately 60 customers continued/returned for next level training

28. TANF Participation • Project Goal to have TANF recipients comprise approximately 1/3 of enrollments over the HPOG project period • TANF participation has averaged 40-45% over the first 3 years of the project • Project designed in close partnership with TANF agency

29. TANF Partnership • Partners in developing project model and proposal • Members of project steering committee overseeing implementation of project model • Clearly articulated outreach and referral protocol • Regular check-in meetings in addition to steering committee

30. HCA in ISIS • Implemented ISIS in summer 2012 (end of Year 2 of HPOG grant) • ISIS Goal: Randomly assign 700 individuals • Outcomes to Date (end of Aug 2013): • 267 individuals RA’d(51% TANF) • 134 assigned to treatment group (51% TANF) • 110 enrolled in HCA • 94 entered training

31. TANF PartnershipPost ISIS • Re-doubled coordination efforts during implementation of ISIS study • Refined referral protocols • Generated materials to support TANF staff • Efforts to safeguard against over-enrollment in control group during random assignment (ensure 50-50 treatment/control split among TANF referrals in addition to overall RA pool)

32. Customer Story Kim is a single mom of three boys who came to HCA in 2011 after nearly two years of training and job hunting in medical administration produced no employment. She had been trying to enter the healthcare field since 2008, was nearing the end of her lifetime TANF benefits, and had nearly exhausted her allotted training time through DSHS. Kim had 9 years of steady employment as a journeyman Industrial Maintenance Mechanic but in 2008 was terminated from employment because of excessive time off related to domestic violence issues. Due to the downturn in the economy, she was unable to find alternate employment in her profession. After a period of homelessness, she found transitional housing and subsequently entered subsidized housing. In addition to employment and housing challenges, Kim was busy managing 3 young boys all with behavioral issues and special needs. After some counseling to figure out the best path for her to enter a health career as quickly as possible, and to best utilize skills gained along the way, she enrolled in the HCA Phlebotomy cohort, which she completed with the highest scores in the class.  Aside from excellent grades, Kim demonstrated patience, bilingual skills, a wry sense of humor and excellent ability work calmly with diverse patients. Following the phlebotomy training, Kim received job search assistance through WorkFirst and the HCA job developer.   In September 2012, she was offered a position at LabCorp at $14.54 per hour, 25 hours per week to start.  Three months later, Kim called to say that she had received praise for the work she was doing and was offered a full-time position. Kim is very thankful for the ongoing services received from the HCA program and now wants to start nursing prerequisites so she can eventually become an RN.

33. Lessons Learned to Date • Importance of understanding the TANF program and how it is implemented in your area • Importance of cultivating relationships at both the local and regional/state level • Benefits and challenges of grappling together with barriers (e.g., participation requirements and 12-month limit on training as a “countable” activity)

34. Questions?

35. Additional Information www.projectisis.org http://www.acf.hhs.gov/programs/opre