1 / 26

Employability and Thinking Skills: A Validity Study

Employability and Thinking Skills: A Validity Study. Megan Rodgers AERA 2013. What’s the Value of College?. Many colleges and universities uphold some form of a liberal arts education (e.g., general education).

javier
Download Presentation

Employability and Thinking Skills: A Validity Study

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Employability and Thinking Skills: A Validity Study Megan Rodgers AERA 2013

  2. What’s the Value of College? • Many colleges and universities uphold some form of a liberal arts education (e.g., general education). • Most higher education institutions include a set of common core outcomes (AAC&U, 2005) • Unfortunately, there has been recent public debate about the value of college • Many graduates have record levels of debt and are unemployed

  3. There is Value in College • Specifically, the skills obtained college should increase graduate employability, a student’s potential to be employed (Rampell, 2011; Yorke, 2004). • What is needed within higher education is a scale that measures a student’s potential to be employed

  4. Measurement of Employability • Typically, employability is measured by alumni surveys (Volkein, 2010) • Not good indicators of potential to be employed • Job market, personal circumstance, etc. • “Employability” instruments exist • Focus on knowledge of the employment process (e.g., Rothwell, Herbert, & Rothwell, 2008). • Basic skills (e.g., Brigance, 1995) • The Employability Skills Inventory (ESI; Liptak, 2010) • Purports to measure skills needed to be employed based on the U.S. Department of Labor’s SCANS Report (U.S. Department of Labor, 1991)

  5. SCANS Report Research Question: What skills are needed for young people to succeed in the increasing globalized world of work?

  6. Employability Skills Inventory • 8 Subscales that directly relate to the SCANS Report • 80 items • 3 Point Response Option: Not True, Somewhat True, Very True • Commercial Scale- $4 per administration • No detail about item development; No validity evidence

  7. Focus of Study • Focus on gathering structural validity evidence for one subscale (Cronbach & Meehl, 1955; Bollen, 2010) • Cross-section of Liberal Arts Education and Employability

  8. Thinking Skills • As defined in ESI (Liptak, 2010): • These abilities also align well with the critical thinking literature (Fisher, 2001). • In ESI measured with 10 items- 6 positively worded (e.g., I am able to…) and 4 negatively worded (e.g., I cannot…) Thinking Skills: People who score high on this scale tend to be able to think creatively, make effective decisions, solve problems logically and efficiently, visualize how things work, apply sound reasoning skills, and use effective learning techniques to acquire new knowledge and skills.

  9. Construct Validity • Benson’s (1998) strong program of construct validity • Substantive Stage • Specifying and operationally defining construct • Backwards Translation • Structural Stage • Relating the structure of the items to the construct • CFA on Thinking Skills • External Stage • Relating the construct to related constructs • Not conducted due to earlier findings

  10. Substantive Validity Do the items on the ESI align with the defined domain for each of the eight subscales? • Backwards Translation • 12 graduate students, 1 professor

  11. Backwards Translation Results • Overall, alignment is not bad • Dual-loading common • Results useful for ESI revisions

  12. Thinking Skills 7 9 10 47 49 6 8 46 48 50 Negative Wording Method Effect 6* 48* 50* 7* 47* 8* 49* 9* 46* 10* Testing Structural Validity Unidimensional Model Bifactor Model Negatively worded items may create a method effect(Tomas & Oliver, 1999)

  13. Method Participants • ESI was electronically administered to 283 undergraduates through the participant pool in a proctored setting Procedure • Finney (2001) suggests that a structured data collection effort will evoke greater attention by the student and reduce negative wording effects 40 items 40 items After finishing the scales, students worked on their puzzle until others completed

  14. Data Analysis • Confirmatory Factor Analysis (CFA) • Data is categorical in nature (Finney & DiStefano, in press) • 3 Point Response Option: Not True, Somewhat True, Very True • Observed data are not continuous; however, underlying latent variable “Thinking Skills” is thought to be continuous • Estimate polychoric correlations • ML Estimation assumes data are continuous • Use Robust Diagonally Weighted Least Squares estimation (Finney & DiStefano, in press)

  15. Data Analysis (Continued) • Polychoric Correlations are LOW, limited variability • Can the proposed model reproduce the data?

  16. Data Analysis (continued) • Fit Indices • χ2 (p>.05), RMSEA (~.06), CFI (~.95 or greater) • Tend to be fairly accurate with categorical data • Polychoric correlation residuals (~.10 or greater) • Nine cases were omitted using listwise deletion; N=274 • All analyses were conducted using LISREL 8.80

  17. Results • Unidimensional model did not fit the data • rDWLS χ2 (35) = 81.95, p <.001, RMSEA=0.07, CFI=0.86 • 16 residuals greater than .10 • Local misfit permeated the model

  18. Thinking Skills Thinking Skills 7 7 9 9 10 10 47 47 49 49 6 6 8 8 46 46 48 48 50 50 Negative Wording Method Effect 6* 6* 48* 48* 50* 50* 7* 7* 47* 47* 8* 8* 49* 49* 9* 9* 46* 46* 10* 10* Results (Continued) • Bifactor model • Did not converge to an admissible solution • Ancillary analysis e e e e e e e e e e e e e e e e e e e e

  19. Results (Continued) • Ancillary Analysis did converge, as expected • Also did not fit the data rDWLS χ2 (29) = 67.12, p <.001, RMSEA=0.07, CFI=0.89. • Bifactor didn’t converge because I was “Overfactoring” the data (Rindskopf, 1984)

  20. Discussion • Limited substantive validity support • The proposed unidimensional model did not fit the Thinking Skills data • Local misfit permeated the model • No structural validity evidence for this subscale • Negative wording did not appear to affect data • Could be due to proctoring and data collection methodology (Finney, 2001) • Need for a measure of employability still exists • Unlikely that this need will be fulfilled with the ESI

  21. Questions?

  22. References Arum, R., & Roksa, J. (2011). Academically adrift: Limited learning on college campuses. Chicago, IL: University of Chicago Press. Association of American Colleges and Universities. (2005).Liberal education outcomes. Washington, DC: Association of American Colleges and Universities. Retrieved from http://www.aacu.org/advocacy/pdfs/leap_report_final.pdf. Brigance, A. H. (1995). Brigance(r)diagnostic employability skills inventory. North Billerica, MA: Curriculum Associates. Finney, S.J. (1991). A comparison of the psychometric properties of negatively and positively worded questionnaire items. (Doctoral dissertation). University of Nebraska, Lincoln, Nebraska. Finney, S. J., & DiStefano, C. (in press). Nonnormal and categorical data in structural equation modeling. In G. Hancock & R. Mueller (Eds.), Structural Equation Modeling: A Second Course, 2nd Edition. Information Age Publishing Inc. Fisher, A. (2001). Critical thinking: an introduction. Cambridge, United Kingdom: Press Syndicate of the University of Cambridge. Retrieved from http://pactiss.org/wp-content/uploads/2011/09/Critical-thinking-introduction.pdf Hacker, A., & Dreifus, C. (2010). Higher education?:how colleges are wasting our money and failing our kids--and what we can do about it. New York, NY: Times Books. Liptak, J. (2010) Employability Skills Inventory. Indianapolis, IN: JIST Publishing.

  23. References (continued) Nye, C.D., & Drasgow, F. (2011). Assessing goodness of fit: simple rules of thumb simply do not work. Organizational Research Methods, 14 (3), 548-570. Rampell, C. (2011, May 20). Once again: is college worth it?. The New York Times. Retrieved from http://economix.blogs.nytimes.com/2011/05/20/once-again-is-college-worth-it/ Rindskopf , D. (1984). Structural equation models empirical identification, heywood cases, and related problems.Sociological Methods and Research, 42, 104-120. Rothwell, A., Herbert, I., & Rothwell, F. (2008). Self-perceived employability: construction and initial validation of a scale for university students. Journal of Vocational Behavior, 73, 1-12. Tomas, J. M., & Oliver, A. (1999). Rosenberg's self-esteem scale: two factors or method effects. Structural Equation Modeling, 6(1), 84-98. U.S. Department of Labor, Secretary's Commission on Achieving Necessary Skills. (1991). What work requires of schools. Washington, DC. Volkwein, J. F. (2010), Assessing alumni outcomes. New Directions for Institutional Research, 2010: 125–139. doi: 10.1002/ir.335 Yorke, M. (2004). Employability in the undergraduate curriculum: some student perspectives. European Journal of Education, 39(4), 409-427.

  24. Thinking Skills items

  25. Graph from Finney and DiStefano (2011, p.72)

More Related