Assessing Technical Skills in CTE: Challenges and Approaches

1. Assessing technical skills in CTE • Challenges, Opportunities and Approaches James R. Stone III Director

2. Agenda • The Problem • A context for the conversation • The labor market trends • Education trends • A framework for technical skill assessment • A few issues, questions

3. …prepare young people to be productive citizens . . . What is the purpose of public education? College and work ready graduates

4. Definitions • What is “college”? any postsecondary education or training experience, including study at two- and four-year institutions leading to a postsecondary credential (i.e. a certificate, license, Associates or Bachelor’s degree). (www.achieve.org) • What is College Ready “…ability to succeed in college courses, but also the capacity to tackle the culture, intellectual norms, and social environment of the college setting” “habits of the mind” (Tanner, 2009). • What is work ready

5. Work and College Ready: Skills The Occupational Expression of Academic Knowledge The Technical Expression of Academic Knowledge Applies to all workplaces (e.g., SCANS, 21st Century Skills) Unique to each Cluster, Pathway, Occupation

6. The Workplace, the School-place Context: Myth & reality

7. Have you heard this … • India & China are producing more engineers than U.S. • US produces 222,000 engineers • India produces 215,000 engineers • China produces 352,000 engineers* * Duke University Study, 2006;

8. More Rhetoric… • If trends in U.S. research and education continue, our nation will squander its economic leadership, and the result will be a lower standard of living for the American people…. By 2015 [the country needs to] double the number of bachelor’s degrees awarded annually to U.S. students in science, math, and engineering. (National Summit on Competitiveness 2005) • The United States faces an unprecedented challenge to its long-term global economic leadership. And a fall from leadership would threaten the security of the nation and the prosperity of its citizens.… High school students in the U.S. perform well below those in other industrialized nations in the fields of mathematics and science … [and thus we need to make] STEM education a national priority. (Council on Competitiveness 2004).

9. What the data show • S&E occupations make up only about one-twentieth of all workers, • and each year there are more than three times as many S&E four-year college graduates as S&E job openings Urban Institute, 2007. • 435,000U.S. citizens and permanent residents a year graduated with bachelor's, master's, and doctoral degrees in science and engineering. Over the same period, there were about 150,000 jobs added annually to the science and engineering workforce. . www.businessweek.com/print/smallbiz/content/oct2007/sb20071025_827398.htm • S&E wages have actually declined in real terms and unemployment rates have increased Rand, 2006

10. What the data show… • 94% of workers reported using math on the job, but, only1 • 22% reported math “higher” than basic • 19% reported using “Algebra 1” • 9% reported using “Algebra 2” • Among upper level white collar workers1 • 30% reported using math up to Algebra 1 • 14% reported using math up to Algebra 2 • Less than 5% of workers make extensive use of Algebra 2, Trigonometry, Calculus, or Geometry on the job2 • M. J. Handel survey of 2300 employees cited in “What Kind of Math Matters” Education Week, June 12 2007 • Carnevale & Desrochers cited in “What Kind of Math Matters” Education Week, June 12 2007

11. The Real Labor Opportunity Middle Skill Occupations

12. Labor Market Skill Distribution - 2016

13. Middle Skill Occupations (B.A./B.S. NOT Required) Occupation Air Traffic Controller Storage and distribution manager Transportation manager Non-retail sales manager Forest fire fighting/prevention supervisor Municipal fire fighting/prevention supervisor Real estate broker Elevator installers and repairer Dental hygienist Immigration and Customs inspector Commercial pilot Salary 102,300 66,600 66,600 59,300 58,920 58,902 58,720 58,710 58,350 53,990 53,870 Farr, M. & Shatkin, L. (2006) The 300 Best Jobs That Don't Require a Four-Year Degree. (US Department of Labor, Bureau of Labor Statistics)

14. HS Reform and Work: Facing Labor Market Realities • No support … that those not going to college need to be qualified to enter college credit courses in order to enter the workforce. • Becoming qualified for college-level classes or for entering a job directly out of high school is not the sole purpose of a high school education, e.g. preparing citizens to participate in a democracy. Barton, P (ETS, 2006)

15. Misreading Labor Market Data has led to: . . . American Diploma Project indicates that regardless if students go on to college or into the workforce after graduation, they still need the same knowledge and skills, particularly in English and mathematics. At a minimum, high school course requirements need to cover four years of rigorous English and four years of math, including Algebra I, Geometry, Algebra II, and data analysis and statistics. The 4x4 approach

16. Rather, what is really needed are 21st century skills; “modern skills” • What students can do with knowledge, rather than what units of knowledge they have (Education Sector, 2008) • The Conference Board (2006) held a survey of over 400 employers regarding new employee readiness into the labor force. The most important skills new entrants need according to this survey were: • professionalism/work ethic, • teamwork/collaboration, • oral communication, and • critical thinking/problem solving.

17. 21st Century Skills Framework • 21st Century Themes • - Global Awareness • - Financial, Economic, Business • & Entrepreneurship Literacy • - Civic Literacy • - Health Literacy Learning & Innovation Skills • Critical Thinking & Problem Solving • Creativity & Innovation • Communication & Collaboration Life & Career Skills • Flexibility & Adaptability • Initiative & Self-Direction • Social & Cross-Cultural Skills • Productivity & Accountability • Leadership & Responsibility Information, Media & Technology Skills • Information Literacy • Media Literacy • ICT (Information, Communications & Technology) Literacy

18. Work and College Ready: Skills The Occupational Expression of Academic Knowledge The Technical Expression of Academic Knowledge Applies to all workplaces (e.g., SCANS, 21st Century Skills) Unique to each Cluster, Pathway, Occupation

19. (Unfortunately) The Solution = MORE Rigor The result?

20. Achievement Flat or Declining in Reading, 17 year olds, NAEP 79% at or above modal score 70% at or above modal score 12.9 Academic Credits 19 Academic Credits Note: Long-Term Trends NAEP Source: NAEP 2004 Trends in Academic Progress.

21. NAEP Science Scores – High School 3.2 Credits 1.5 Credits

22. HS Achievement In Math 3.6 math credits 1.7 Math Credits Note: Long-Term Trends NAEP Source: NAEP 2004 Trends in Academic Progress and NAEP 1999 Trends in Academic Progress.

23. High Stakes Tests? Evidence from rigorous studies based on longitudinaldata suggest that state high state exit exams: • have effect on student achievement, nor • do they appear to have effectson labor market outcomes. • They do contribute to reductions in high school completion rates and increases in the rate of General Educational Development (GED) test-taking. Grodsky, E., Warren, J.R., Kalogrides, D., (2009). State High School Exit Examinations and NAEP Long-Term Trends in Reading and Mathematics, 1971-2004. Educational Policy (23)(4), 589-614.

24. It is getting worse Your child is less likely to graduate from high school than you were; the United States is now the only industrialized country where young people are less likely than their parents to earn a diploma, Houston Chronicle, Libby Quaid, 10/23/08 68% Source: One-Third of a Nation (ETS, 2005)

25. Remediation • Once many of these same students get into college, 40% of four-year students and 63% of two-year students require remediation. (a report from Education Commission of the States) Patrick M. Callan, Joni E. Finney, Michael W. Kirst, Michael D. Usdan and Andrea Venezia, The Governance Divide: A Report on a Four-State Study on Improving College Readiness and Success (San Jose: The National Center for Public Policy and Higher Education, 2005). Source: NCES (2003), Remedial Education at Degree Granting PS Institutions in fall 2000

26. 40 percent of (college grads) not enrolled in graduate education say they are employed in a job where a college degree “is not required.” Paul Barton, ETS, 2006

27. HS Reform & Labor Market Realities …to right these workplace problems, policy makers are looking in the wrong direction…paying attention to skills workers really need to succeed, not on an assumed set of skills that may not be so critical after all . . . Robert Lerman (2008)

28. Perkins – “Programs of Study” • Incorporate secondary education and postsecondary education elements; • coherent and rigorous content • aligned with challenging academic standards • and relevant career and technical content • in a coordinated, non-duplicative progression of courses that align secondary education with postsecondary education • May include dual enrollment

29. Programs of Study – Goal Lead to an industry-recognized credential or certificate at the postsecondary level, or an associate or baccalaureate degree.

30. A Framework for TSA That is the context

31. Work and College Ready: Skills

32. If you’re only teaching welding . . . You’re not preparing your students . . .

33. TSA: The Process

34. Technical Skill Attainment Perkins IV tightens the expectation for technical attainment by requiring: “Technical skill attainment, aligned to industry-recognized standards if available and appropriate.” This language is in place at both the secondary and postsecondary levels.

35. Technical Skill Attainment The US Department of Education has supported national conversations around Perkins performance indicators through its Next Steps Work Group. As with every other performance indicator, the definition and measurement approach for technical skill attainment must be valid and reliable.

36. Types of Technical Skill Attainment • Industry-Based Any external, third-party assessment that objectively measures student attainment of industry recognized skills, appropriate to the educational level of CTE concentrators and that meets minimum state validity and reliability guidelines. Examples: • National/international credentialing or certification exams • State credentialing or licensing exam • Industry developed exam for occupations/specialties

37. Types of Technical Skill Attainment • State-Approved A state-approved, teacher/instructor developed assessment that aligns with either state-established or industry-recognized standards (at the career cluster, pathway or program of study level) and that meets minimum state validity and reliability guidelines. Examples: • State-developed exam, not tied to industry standards • Locally-developed, state-approved exams • Teacher administered, LEA-approved exams

38. Types of Technical Skill Attainment • Proxy Measures Any non-approved or non-assessment related indicator of technical skill attainment but meet minimum state validity and reliability guidelines. . Examples: • GPA • Course completion • Program completion • Teacher-developed exams, not approved externally

39. Building an Effective Skill Assessment • Open and transparent-everyone knows what’s expected • Scored by instructor working with a rubric • Tasks are well defined • Written and performance assessment (Darling-Hammond & Pecheone, 2009 Take out your wallet or purse

40. The Technical Skill Assessment Inventory Template(An NRCcte Project) A Framework

41. What is the SCRAPPER Principle? • Skill • Cost • Rationale • Alignment • Penetration • Performance • Estimation • Reporting The following eight characteristics describe any technical skill attainment measure: Based on solid rationale, can a cost-effective, widely-reported, well-aligned, broadly penetrable, perfectly-estimated assessment instrument truly measure student technical skill performance levels?

42. The SCRAPPER Principle • Skill • Cost • Rationale • Alignment • Performance • Penetration • Estimation • Reporting • Definition, Specificity • Test development/Administration • Signal; improvement; accountabilty • Industry standards; Educ Require • Target; baseline • Student, Program Coverage • Timing, Reliability, Validity • End of course; Program; Bridge to PS

43. Challenge Defining a technical skill

44. Table Exercise: Work and College Ready Skills

45. Table Talk: What is a technical skill-Finance Cluster Source: http://careerclusters.org

46. Table Talk: What is a Technical Skill - Construction Pathway Source: http://careerclusters.org

47. Defining Technical Skills Developed capacities used to design, set-up, operate, and correct malfunctions involving application of machines or technological systems (O*Net)

48. Cost • Bubble tests • NOCTI, CTSOs • $.07 • $12-$30 or more

49. Rationale (From the National Center) Purpose • Local Program Improvement • Signal • Aggregated data Function • Diagnostic • Certification • Accountability

50. Alignment – Where to target?