Measuring Skills-Builder Outcomes

1. Measuring Skills-Builder Outcomes Kathy Booth, WestEd kbooth@wested.org October 18, 2019 bit.ly/wa-skills-builders

2. Today’s talk will focus on three topics: • What and who are skills-builders and how can they be identified? • How have skills-builders been integrated into California’s accountability framework? • Why does measuring skills-builder outcomes matter?

3. What and who are skills-builders AND how can they be identified?

4. Skills-Builders Skills-builders are workers who take a limited number of career education courses to maintain and add to skillsets required for ongoing employment and career advancement. Based on these goals, success is better measured by examining employment and earnings outcomes than completion.

5. Identifying Skills-Builders Students A number of studies have found significant numbers of students who meet the skills-builder profile: • VanDerLinden (2002) – self-reported data • Adelman (2005) – single variable course-taking analysis • Hagedorn and Prather (2005) – multi-variable course-taking analysis • Ammon et al. (2008) – survey plus course-taking, goal, and demographic analysis • Bahr (2010) - large-scale algorithms used to create clusters of students based on course-taking patterns

6. Identifying Skills-Builders Students Peter Riley Bahr at the University of Michigan has done the most comprehensive analysis, including studies in California, Colorado, Michigan, and Ohio. These analyses span both recession and non-recession timeframes and examine educational and economic outcomes over a 5-year period after initial enrollment. • California • 2,236,558 students who enrolled between Fall 2002 to Summer 2008 • Colorado • 218,954 students who enrolled between Fall 2006 to Summer 2011 • Michigan • 86,442 students who enrolled between Fall 2002 to Summer 2006 • Ohio • 225,565 students who enrolled between Fall 2007 to Summer 2010 Source: Bahr, et. al., 2019 unpublished manuscript

7. Who Are Skills-Builders Students? About 10% of students meet the skills-builder definition. Skills-builders are more likely to be: • Older (33-38 years old) • Male (57% - 64%) • White (7-15 percentage points higher than the general population) They also have a downward trend in wages prior to enrollment, but a significant upward trend after attending college. Source: Bahr, et. al., 2019 unpublished manuscript

8. Who Are Skills-Builders Students? • Most skills-builders report an educational goal of pursuing a personal interest, upgrading skills for a job, or obtaining a certificate. • They generally take between 3-8 credit units. • Skills-builder students are less likely to receive financial aid than other student types. • In California and Colorado, about a quarter of skills-builders had some form of postsecondary credential before attending community college. • In both states, there are a subset of skills-builders who earn low-unit certificates. Source: Bahr, et. al., 2019 unpublished manuscript

9. Recommended Skills-Builder Definition Based on his analysis from multiple states, Bahr recommends identifying skills-builder students using the following methodology: • enrolled for no more than three semesters • attempted no more than 24 credit units • over 75% of credits are in career education fields • passed all credits Source: Bahr, et. al., 2019 unpublished manuscript

10. A Note About Noncredit • In many states, skills-builder students may be enrolled in noncredit coursework. • Most states do not track noncredit course-taking information in their statewide data systems. • Work is underway in California (which does track noncredit course-taking) to create a skills-builder definition for noncredit, based on completing 48 contact hours in career education courses.

11. HOW HAVE skills-builders BEEN INTEGRATED INTO CALIFORNIA’S ACCOUNTABILITY FRAMEWORK?

12. How Were Skills-Builders Identified and Recognized in California? In 2013 and 2014, three different researchers in California used a variety of approaches to identify skills-builders and their economic outcomes: • Stated college goal, subsequent course-taking, and earnings patterns (Fuller) • Prepare for a new career (acquire job skills) • Advance in current job/career (update job skills) • Maintain certificate or license (e.g. Nursing, Real Estate) • Surveying former students (Greaney) • Large-scale algorithms creating clusters of students based on course-taking patterns, matched to earnings data (Bahr)

13. How Were Skills-Builders Identified and Recognized in California? • All three studies returned similar results regarding which students were identified as skills-builders and their earnings gains after leaving college. • As a result, skills-builders were incorporated into the state’s accountability framework and dashboards showing student outcomes.

14. California’s Official Skills-Builder Definition In dashboards produced by the California Community Colleges Chancellor’s Office, skills-builder students are those who: • Took one or more non-introductory career education courses worth 0.5+ credit units (sources: enrollment and course data) • Passed all career education courses (source: enrollment data) • No longer enrolled in a community college (sources: enrollment and National Student Clearinghouse data) • Did not earn a community college award (source: award data)* • Did not transfer to a four-year institution (source: NSC data) * because California community colleges are not required to report very low-unit certificates, some students may be completing low unit awards

15. California’s Official Skills-Builder Outcomes • Because the majority of students are employed both before and after attending community college, employment outcomes were not included in the accountability framework. • Skills-builder outcomes are measured by examining earnings gains (source: Unemployment Insurance Wage file). • Data are displayed at the institutional level, as well as by program.

16. California’s Skills-Builder Pathways • Skills-builders are most prevalent in the fields of Construction Crafts, Business & Management, Public & Protective Services, Early Childhood Education, and Information Technology. • Returns to skills-building vary significantly by discipline, with strong returns in areas like Water & Wastewater Technology, Administration of Justice, and Electronics & Electric Technology. • There are weak returns to skills-building in Computer Information Systems, Human Services, and Culinary Arts. Source: Bahr & Booth, 2013

17. Measuring Skills-Builders: Phase One • In 2013, California created an accountability dashboard called the Student Success Scorecard. • After repeated requests from the field, backed by research, a metric on skills-builder earnings gains was added to the Scorecard in 2016. • That same year, skills-builder filters were added to new statewide dashboards showing program-level outcomes. • After this point, skills-builder outcomes were integrated into required goal-setting frameworks, began to appear in college strategic plans, and were referenced in conversations about student success.

20. Measuring Skills-Builders: Phase Two • In 2017, the California Community College system’s new Chancellor released the Vision for Success, which focused on associate degree and transfer outcomes. • In 2018, the legislature imposed a new performance-based funding formula with a focus on momentum points for completion and transfer pathways, completion, transfer, and living wage attainment. • During 2018-19, the Student Success Scorecard was replaced by the Student Success Metrics dashboard, which displays a range of momentum, completion, and employment outcomes based on student goals (Adult Ed/ESL, Short-Term Career Education, Degree/Transfer).

22. Measuring Skills-Builders: Phase Two • There was some protest from the field about the lack of metrics specific to skills-builders, with the Chancellor’s Office (erroneously) responding that a metric on earning 9+ career education units met that need. • Although the Student Success Metrics dashboard includes a view specific to the outcomes of short-term career education students, many interpreted the system priorities as shifting back to completion. • Career education practitioners are beginning to report that their colleges are cutting offerings that target skills-builders, which may cause the collapse of programs that were predicated on training both skills-builders and students seeking longer-term course-taking in the same courses.

23. Measuring Skills-Builders: Complications 1) Framing skills-builders as non-completers may have been a mistake, because skills-builders sometimes complete short-term awards. • Practitioners don’t recognize the short-term career education journey in the Student Success Metrics dashboard as being predominantly skills-builders, and thus believe this measure of success has disappeared, even though earnings gains are displayed.

24. Measuring Skills-Builders: Complications 2) Because of historic academic divides, practitioners struggle to distinguish short-term and long-term career education programs, further complicating the discussion. • Practitioners conflate the short-term career education journey in the Student Success Metrics dashboard with all career education and assume the degree/transfer journey focuses on non-career education pathways. • Because of this, they don’t recognize the significant number of career students in the degree/transfer journey and complain that there is no measure of longer-term career education programs, rather than discussing the variety of pathways in career education.

25. Measuring Skills-Builders: Complications 3) Because college leadership is not familiar with employment and earnings indicators, they don’t know how to interpret the data. • Despite the availability of employment and earnings data in statewide dashboards for both completers and skills-builders, paired with concerted training efforts to familiarize college leadership with this information, career education remains largely absent from discussions about student success. • Even though the funding formula includes a living wage metric, the legislature, system office, and college leaders appear to be unaware that skills-builders are most likely to attain that outcome.

26. Measuring Skills-Builders: Complications 4) Turnover in Chancellor’s Office leadership undermined the movement to recognize skills-builder outcomes. • After a near-total turn over of staff in the system office, few remained who understood the research or the impetus for focusing on skills-builder outcomes. • Without a sustained shift in underlying assumptions about how long/often students attend college, cultural norms are prevailing—the assumption that each student attends college once.

27. WHY DOES MEASURING skills-builder OUTCOMES MatTER?

28. Talking about skills-builders can be part of a broader conversation about why conventional metrics don’t capture career education outcomes. • Variable math/English skills: basic quantitative reasoning (cosmetology) or advanced math (radiology tech) • Variable credit momentum thresholds: 8 units (EMT) or 110 units (aviation) • Variable credentials: no credential (transmission course), short-term certificates (petroleum tech), long-term certificates (sign language), degrees (accounting), external credentials (real estate), and bachelor’s degrees (nursing) • Variable employment outcomes: new job (computer science) or keeping a job (public & protective services) • Variable earnings outcomes: low wage but critical (early childhood education) or high wage (electronics technology) Traditional Success Metrics Don’t Fit for Career Education

29. Employers don’t always value academic credentials In California, Bahr found that while taking career education courses had a positive impact on earnings, there were very few disciplines in which receiving a community college certificate or degree meant that students earned more than those who had simply completed the same coursework (the most notable exceptions were in health, where programs are aligned with state licensing requirements). Source: Bahr, 2014

30. SOME students will prioritize third-party Credentials over academic awards • CTE Outcomes Survey: 33% of respondents to the 2016 survey reported they got a third-party credential. • Census Bureau Survey of Income and Program Participation: third-party credentials boost the income of those with “some college” by 13% and those with an associate’s degree by 18%. Sources: LaunchBoard; US Census Bureau, 2012

31. Employment Requirements Are Changing • Most jobs require mastery of quickly-evolving technology, particularly in the context of automation, meaning students often need to participate in retraining. • Workers are moving from jobs that no longer exist to jobs that may not have existed when they started their studies. • While large companies are investing in training programs, many employers expect workers to come to the job fully trained.

32. Skills-building is relevant to guided pathways • As guided pathways seeks to make equity even more integral, attention is turning to earnings outcomes. • Program and meta-major maps can be extended to identify courses that students may need to come back and take to build skills over time, in addition to mapping programs to transfer pathways. • In addition to streamlining matriculation for first-time students, colleges can evaluate their practices related to returning students.

33. We need to figure out how to re-skill more types of employees • Skills-builder pathways disproportionately serve older, white men, but other populations need retraining just as much. • Fulfilling our career education, access, and equity missions may require that we change the way we deliver education.

34. By addressing skills-builder outcomes, We can provide a more accurate understanding of student success • Understand and document the diversity of pathways within our programs • Help our colleagues understand the changing workplace, what it means for education, and how we are responding • Quantify the success of various educational routes • Community college credentials • External credentials • Wage gain and job retention

35. Find out more • Practitioner guides on skills-builders: https://www.wested.org/project/quantifying-non-completion-pathways-to-success/ • Peter Bahr’s research: http://www.soe.umich.edu/people/profile/peter_riley_bahr/ • California dashboards: https://www.calpassplus.org/LaunchBoard/Home.aspx