Neal Schmitt Michigan State University Presented at College Board, ETS, AERA Conference December 10, 2010

1. Combining Cognitive and Non-Cognitive Measures: Expanding the Domain of College Performance and its Prediction Neal Schmitt Michigan State University Presented at College Board, ETS, AERA Conference December 10, 2010

2. Acknowledgements • Jessica Fandre Tim Pleskac • Alyssa Friede Abigail Quinn • Michael Gillespie Lauren Ramsay • Anna Imus Smriti Shivpuri • Brian Kim Ruchi Sinha • Stephanie Merritt Tae-Young Yoo • Fred Oswald Mark Zorzie • Matt Reeder Juliya Golubovich • College Board (Wayne Camara and Krista Mattern)

3. Outline • History and Background • “Job analysis” or conceptualization • Instrumentation (description of noncognitive measures used and outcomes examined) • Validity data • Subgroup Differences and Implications • Faking issues • Acceptability • Research (profiles, fit, goal orientation, dif) • Limitations and future research

4. Developing Alternative Measures of Student Potential Motivation for our work • Broaden the scope of student outcomes and capabilities considered in college admissions. • Reduce adverse impact. • Test the feasibility of developing “noncognitive” measures that are • valid • practical in terms of time and effort required to assess • less susceptible to faking

5. Developing Alternative Measures of Student Potential (Oswald et al, 2004, JAP) • Identify a broader domain of college student performance: • Review university mission statements and department objectives • Interview with university staff responsible for student life at Michigan State University • Review of the education literature on student outcomes • Our systematic search resulted in 12 dimensions of student performance…

6. Knowledge and mastery of general principles Continuous learning, intellectual interest and curiosity Artistic and cultural appreciation Appreciation for diversity Leadership Interpersonal skills Dimensions of College Student Performance intellectual intrapersonal Social responsibility and citizenship Physical and psychological health Career orientation Adaptability and life skills Perseverance Ethics and integrity interpersonal

7. Two “Noncognitive” Measures • 1. Situational judgment inventory • A situation is presented along with several alternative courses of action. • The respondent is asked to indicate what she/he would be most likely and least likely to do. • 2. Biodata • Short, multiple choice reports of past experience/background and/or interests/preferences.

8. Situational Judgment Inventory (SJI) Developed situational judgment items for each of the 12 performance dimensions • Student generated critical incidents (CIs) for each dimension • Translated CIs to stems for each item • Other students generated solutions to these questions • Researchers edited the options • Re-sorting back into 12 dimensions • 3 Answer Keys: (see Motowidlo, Dunnette, & Carter, 1990) • Expert student (junior and senior students) scoring • Resident Advisor scoring • African American key

9. Sample SJI Item for Leadership You are assigned to a group to work on a particular project. When you sit down together as a group, no one says anything. • -1 Look at them until someone eventually says something • Start the conversation yourself by introducing yourself • +1 Get to know everyone first and see what they are thinking about the project to make sure the project’s goals are clear to everyone • Try to start working on the project by asking everyone’s opinion about the nature of the project • You would take the leadership role by assigning people to do things or ask questions to get things rolling

10. Sample SJI Item forInterpersonal Skills You and some other students in your dorm area feel that a small group of students are highly disruptive during times when you would like to study or sleep. What would you do? • Talk to the resident assistant about it, as that is one of the responsibilities of their job. • -1 That’s part of life in the dorms. Let it go. • Bring it up at the next floor meeting. • +1 Politely talk to the disruptive students and ask them to be more considerate. If the problem persists, talk to the resident assistant. • Wear earplugs or headphones when necessary. • Develop and implement appropriate rules to address the problem.

11. Sample Item for Knowledge • You decided early in the term to do a paper on a topic very interesting to you. However, you have found it difficult to find information on your topic, your job has taken more time than you wanted, and you have had more work in your other courses than you anticipated. Now it seems like you may have to engage in several "all-nighters" to complete your paper on time. What would you do? • a. Seek help from other students who may have had a similar experience. • b. (-1)Pick a topic that can be completed quicker. An “A” is an “A”. • c. (1)Set up a schedule on which you can complete all of the other work you need to do, spend as much time on the paper as possible, and meet with the instructor to discuss what you have so far and get suggestions. • d. Do whatever it takes to complete the paper, including “all-nighters”. • e. Talk to the instructor about the situation and ask for advice. • f. Make the paper a priority, but take into account how much the paper is worth in the class.

12. Biodata Measure Developed biodata items for each dimension • Reviewed biodata item pools and adapted items related to each major performance dimension. • Resorted items into dimensions. • Asked a pilot sample to respond to open-ended versions of quantitative response options to determine appropriate scale anchors. • We used a rational scoring approach to these items, but also developed empirical keys against 3 criteria

13. Sample Biodata Items for Leadership • The number of high school clubs and organized activities (such as band, sports, newspapers, etc.) in which I took a leadership role was: • 4 or more • 3 • 2 • 1 • I did not take a leadership role • How often do you talk your friends into doing what you want to do during the evening? • most of the time • sometimes (about half the time) • occasionally (about as often as others in my group • seldom or infrequently • never

14. Sample Biodata Items for Multicultural Appreciation • How often have you participated in social service or charity organizations? • Four or more times • Three times • Two times • Once • Never • If given a choice at a restaurant, would you order any food with which you are unfamiliar? • Never, I would always order foods that I know and enjoy • Sometimes I might try a new food if someone else ordered it • Occasionally I will order something new provided I can also order familiar food at the same time • If given a chance, I will always order a new food and try it

15. Sample Biodata Items for Social Responsibility • In the past year how many times have you considered the environment when purchasing a product (for example hairspray, or a car?) • Never • Once • Twice • Three or four times • Five times or more • How often do you work with not for profit groups? • Never • Not very often • Sometimes • Often • Always

16. Outcomes Examined • Self Ratings on behaviorally anchored rating scales built around the 12 dimensions • Self rated class attendance • University archives (grades) • Organizational citizenship behavior • Deviance • Continuation in school and graduation

18. Validity Data: College GPA 2001-02 2003-04 2004-8 2009-10 (N=614) (N=568) (N>1900) (N>550) • Knowledge .22 .22 .26 .22 • Learning .05 -.02 .13 .10 • Art Appr. .01 -.03 .18 .16 • Mltcult.Appr. .07 -.04 .11 .12 • Lead .14 -.01 .09 .07 • Int. Skl. .04 • Soc. Resp. .08 .07 .13 .08 • Health .23 .14 .11 .04 • Cr. Ornt. -.02 -.06 -.14 -.11 • Adapt .21 .13 .05 .01 • Perser .15 .07 .07 .10 • Ethics .14 .22 .17 .13 • SJI .16 .11 .22 .09 • HSGPA .39 .53 .29 • ACT/SAT .33 .34 .53 .44

19. Validity Data: Class Absences 2001-02 2005-06 2008 2009-10 (N>630) (N>900) (N >556) (N>600) • Knowledge -.18 -.15 -.14 -.03 • Learning .00 -.07 -.06 .04 • Art Appr. .07 -.03 -.08 -.10 • Mltcult.Appr. .02 -.04 -.10 -.06 • Lead -.03 -.06 -.02 -.04 • Int. Skl. -.09 • Soc. Resp. -.09 -.08 -.08 -.11 • Health -.23 -.17 -.12 -.05 • Cr. Ornt. -.06 -.05 -.08 -.05 • Adapt -.15 -.10 -.10 .07 • Perser -.20 -.18 -.16 -.01 • Ethics -.31 -.17 -.24 -.11 • SJI -.27 -.16 -.17 -.14 • HSGPA -.04 -.01 -.02 • ACT/SAT .11 .17 .14 .07

20. Validity Data: Self Report-BARS 2001-02 2003-04 2008 2009-10 (N=614) (N=568) (N>547) (N>600) • Knowledge .33 .46 .21 .32 • Learning .27 .37 .24 .39 • Art Appr. .51 .33 .20 .33 • Mltcult.Appr. .38 .34 .29 .30 • Lead .43 .34 .29 .38 • Int. Skl. .15 • Soc. Resp. .35 .32 .26 .30 • Health .28 .16 .26 .29 • Cr. Ornt. .22 .36 .22 .31 • Adapt .24 .26 .28 .30 • Perser .34 .46 .36 .37 • Ethics .11 .39 .23 .22 • SJI .53 .54 .23 .25 • HSGPA .08 .08 .02 • ACT/SAT -.01 -.01 .01 .08

21. Incremental Validity: College GPA 01-02 03-04 08 09-10 Step 1: Act/SAT, HSGPA ∆R2 .103 .179 .398 .200 Step 2: Biodata, SJI ∆R2 .089 .070 .029 .069 Adjusted R .438 .499 .419 .480 N 610 331 1155 296 Noncog. Vars. Significant Know Know Lrng. Cr. Ornt. Lrng. Lrng. Health Know Health Ethics Cr.Ornt. Adapt SJI SJI

22. Incremental Validity: Absenteeism and BARS 05-06 08 follow-up Absenteeism BARS Absenteeism BARS Step 1: Act/SAT, HSGPA ∆R2 .028 .019 .033 .008 Step 2: Biodata, SJI ∆R2 .060 .215 .116 .240 Adjusted R .297 .484 .386 .497 N 800 801 556 547 Noncog. Vars. Health Health Lead Multic Ethics Ethics Health Health SJI SJI Ethics Persv Adapt SJT Ethics Cr.Ornt. Persev.

23. Subgroup Differences in Standardized Units (04sample) Male-Female Cauc-Afr.Am. Cauc-Hisp.Am. SJT -.15 .13 .10 Know .00 .36 .23 Cont.Lrn. .10 .05 .00 Artistic -.17 .31 .01 Multi.Appr. -.16 -.08 -.46 Leader -.15 .14 .10 Respons. -.25 .16 .01 Health .52 .44 .30 Car.Ornt. -.17 -.55 -.08 Adapt. .00 .11 .16 Persev. -.23 -.19 .00 Ethics -.19 .22 .29 HSGPA -.04 -.61 .42 SAT/ACT .43 -1.46 1.01

24. Percent of subgroups admitted under various strategies Hispanic Asian African Caucasian Cog Cog+ Cog Cog+ Cog Cog+ Cog Cog+ Top 15% 4.3 6.4 17.8 14.9 .9 4.1 77.0 74.6 Top 50% 4.0 4.6 10.5 10.1 8.3 10.0 77.1 75.3 Top 85% 4.5 4.7 7.6 7.7 18.4 18.7 69.5 69.0 All 3.7 9.0 19.4 67.8 Cog=equally weighted composite of HSGPA and SAT/ACT Cog+=equally weighted composite of HSGPA, SAT/ACT, and Non-cognitive measures.

25. Average Cumulative GPA for Subgroup Members who graduated at various levels of selectivity Hispanic Asian African Caucasian N Cog N Cog+ N Cog N Cog+ N Cog N Cog+ N Cog N Cog+ Top15 4 3.67 6 3.57 23 3.88 18 3.84 0 1 3.66 208 3.88 197 3.87 Top50 17 3.51 17 3 .42 55 3.65 54 3.63 15 3.46 15 3.38 651 3.66 638 3.66 Top85 23 3.44 22 3.47 64 3.62 64 3.62 49 3.05 47 3.08 929 3.52 905 3.54

26. Proportion of Subgroups Graduating in Four Years under Different Levels of Selectivity Hispanic Asian African Caucasian Cog Cog+ Cog Cog+ Cog Cog+ Cog Cog+ Top15 93 90 89 91 100 92 72 70 Top50 79 78 72 75 87 87 59 60 Top85 77 76 68 68 79 78 51 52

27. Admits versus Applicants: Standardized mean differences (d) 2004 2006 • Knowledge .55 .54 • Continuous Learning .43 .49 • Artistic Appreciation .35 .30 • Multicultural Appreciation .48 .40 • Leadership .43 .35 • Social Responsibility .46 .46 • Health .26 .30 • Career Orientation .25 .21 • Adaptability .22 .25 • Perseverance .39 .32 • Ethics .62 .53 • Situational Judgment .50 .52

28. Conclusions on Faking Research (represents summary of a series of studies • Applicants score higher than current students (d=.2 to .6) • Coaching has a significant impact on the degree of score inflation • Elaboration can minimize score inflation, but…feasibility is an issue and its effects do not appear to generalize to nonelaborated items • Elaboration has no impact on validity • Warnings do not appear to have much effect, but…generalizability to an applicant situation has not been evaluated. • Are they any less fakable than essays or less inflated than letters of recommendation???

29. Sample Bogus Items: Carelessness • How often in the past year have you programmed in AJMR? • never • once • twice • three or four times • five times or more • How often, in the past three years, have you operated a rhetaguard? • never • once • twice • three or four times • five times or more

30. Profiling Subgroups of Students based on HSGPA, SAT/ACT and Noncognitive Variables

37. Outcomes and Implications • There were predicted outcome differences (GPA, absenteeism, satisfaction) across profile groups • Highly motivated, career-oriented group is most likely to respond to remedial efforts particularly if they relate to their career objectives • Marginal group without career objectives is a high risk group. Career counseling and intensive remediation may be necessary • Efforts to broaden the scope of interests of the high ability, culturally limited may be desirable in some universities • Students in the high ability well rounded group might be identified as potential student leaders and peer mentors/tutors.

38. Overall Conclusions • We can develop valid noncognitive measures that relate to GPA and other important student outcomes • Faking of the biodata and SJI remains a problem. • Reactions to their use are not significantly different than reactions to the ACT/SAT • Subgroup differences are minimal and certainly much less than those we find for cognitive ability measures • There may be useful other ways to employ these instruments; that is, to identify subgroups for whom interventions designed to retain them will be needed.

39. Thank you for your attention!