“Personal Attribute” Assessments for Higher Education Admissions

1. “Personal Attribute” Assessments for Higher Education Admissions Patrick Kyllonen, DirectorCenter For New ConstructsETS pkyllonen@ets.org Kyllonen, P.C. (May, 2009). “Personal Attribute” Assessments for Higher Education Admissions. Colloquium. Catholic University, Santiago, Chile.

2. Outline • Importance of personal attributes in higher education • Projects designed to assess personal attributes • European Self Assessment Approaches • Robert Sternberg’s efforts • Michigan State/College Board project • ETS® Personal Potential Index (PPI) • Future steps

3. Validity of Personal Attributes • Noftle & Robins meta analysis (2007) • 20 studies (N = 5000), r (Conscientiousness, college grades) = .29 • 4 new studies (N = 8500) • r (conscientiousness, college GPA) = .19 (median over 3 samples), controlling for Gender, high school GPA, SAT-V, SAT-M • Partially mediated by • Academic effort • Perceived academic ability • Key facets of Conscientiousness • Diligence • Perfectionism • Prudence • Competence • Achievement-striving • Self Discipline • Not • Organization • Order • Dutifulness • Deliberation

4. % employers rating skill “very important” 4-yrgrads 2-yrgrads HS grads Non-cog cog

5. European Self Assessment Approaches • Several German (Aachen, Frankfurt) and Austrian (Vienna) Universities are using self assessments to help students select majors (see EAPA symposium, 2007) • Example: Prof. Psychology, Lutz Hornke, Aachen University • Engineering Department • Problem: High drop out rate • Solution: Self Assessments for high school students (internet delivered) • Content • Personality assessments, interest inventories, cognitive tests • Approach • Compare student applicants with department averages to help student decide “fit” • Evaluation • Compare outcomes after implementation

6. Robert Sternberg’s Efforts • Based on decades of research on “triarchic theory” • 3 kinds of intelligence—practical, creative, analytic • Study 1: University of Michigan Business School • Practical intelligence measures • Situational Judgment & Case Scenario problems • Findings: Incremental validity over GMAT, particularly for course projects • Results: Difficulty in implementation Hedlund, J., Wilt, J.M., Nebel, K., Ashford, S.E., & Sternberg, R.J. (2006). Assessing Practical Intelligence in Business School Admissions: A Supplement to the Graduate Management Admissions Test. Learning and Individual Differences, 16, 101-127.

7. “Successful Intelligence Assessment” (“Personnel Shortage” (5 others)) The setting is a manufacturing plant facing a personnel shortage. Employees are working excessive amounts of overtime and morale is low. (5-15 pages of further description.) You are the Human Resources Manager. What do you do? Situational Judgment Problems (3 subproblems each) Immediately hire 200 new workers (rate 1-7) Meet with unit heads to discuss strategies (rate 1-7) Case scenario problems What is the main problem? What would you do to address the main problem? What information did you focus on? What outcome do you expect? What obstacles?

8. Robert Sternberg’s Efforts • Study 2: Rainbow Project (undergraduate admissions) • Consortium of several universities; sponsored by the College Board • Measures • Creative—interpreting cartoons; writing essays • Analytic—traditional ability measures • Practical—situational judgment tests (next page) • Results • Incremental validity of new measures • Reduced impact with minority groups • Outcomes • Sponsor decided to experiment with other measures

9. Example “Tacit Knowledge Intentory” Item You’ve been assigned to work on a project for a day with a fellow employee whom you really dislike. He is rude, lazy, and rarely does a proper job. What would be the best thing for you to do? ___Tell the worker that you think he is worthless. ___Warn the worker that, if he is not “on his toes” today, you will complain to the supervisor. ___Avoid all conversation and eye contact with the other worker. ___Be polite to the other worker and try to maintain as business-like a manner as possible so that hopefully he will follow your example for the day. ___Tell your supervisor that you refuse to work with this man. ___The project is going to be impossible to accomplish with this worker, so you may as well not even try--you can always blame your bad work partner. ___See if you can convince one of your friends to take your place and work with this employee. ___Demand a raise from your supervisor; you should not have to tolerate these conditions.

10. Robert Sternberg’s Efforts • Study 3: Kaleidoscope Project (Tufts University undergraduate admissions) • Optional essays, evaluated for creative (“what if”), practical, and wisdom skills using a rubric • Results • New measures did not overlap much with standardized tests • New measures did not predict grades • New measures were correlated with extracurricular activities • Consequence • Increase in minority group enrollments • Sends message that other skills are important for college

11. 12 dimension taxonomy College Board/Michigan State University Effort • Taxonomy of Valued Academic Skills (mission statements from 35 universities) VII. Social responsibility, citizenship and involvement VIII. Physical and psychological health IX. Career orientation X. Adaptability and life skills XI. Perseverance XII. Ethics and integrity I. Knowledge, learning, mastery of general principles II. Continuous learning, intellectual interest and curiosity III. Artistic cultural appreciation and curiosity IV. Multicultural tolerance and appreciation V. Leadership VI. Interpersonal skills Schmitt, N., Oswald, F. L., Kim, B. H., Imus, A., Drzakowski, S., Friede, A., & Shivpuri, S. (2007). The use of background and ability profiles to predict college student outcomes. Journal of Applied Psychology, 92, 165-180. Oswald, F. L., Schmitt, N., Kim, B. H., Ramsay, L. J., & Gillespie, M. A. (2004). Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology, 89, 187-207.

12. 12 dimension taxonomy College Board/Michigan State University Effort • Sample Situational Judgment Item (“Leadership”) An important class project you have been working on with a group of other students is not developing as it should because of petty differences and the need of some members to satisfy their own agenda. How would you proceed? (mark “most” and “least” likely to do) __Try to solve the group problems before starting on the work. __Work hard by yourself to make sure the project is finished, taking on others’ share of the work if necessary. __Talk to the professor and get suggestions about solving the problem. If that doesn’t work, try to switch groups or have an independent project. __Schedule a number of meetings, forcing the group to interact. __Take charge and delegate tasks to each person. Make them responsible for their part of the project. __Talk to the group and demand that they start working together. • Sample Biodata Item (“Leadership”) How many times in the past year have you tried to get someone to join an activity in which you were involved or leading? a.  Never; b. Once; c. Twice; d. 3 or 4 times; e. 5 times or more.

13. 12 dimension taxonomy College Board/Michigan State University Effort • Results (N = 1000--2000; k = 10 institutions) • Small incremental validity (over grades and tests) • With grade point average as criterion • Large incremental validity • Time to graduate; Absenteeism, Self-ratings, Peer-ratings • Smaller disparate impact (vs. minority groups) • Status • Larger study outcome data currently being analyzed (2009) • N = 8000; k = 15 • Concerns about faking (especially biodata) • Concerns about cost and politics of implementation

14. The Case for Letters of Recommendation • Top three criteria for evaluating graduate students (interview studies): • Grades • Standardized-test scores (GRE) • Letters of recommendation • Letters are useful • Give qualitative information about applicants • Provide information about students’ motivation, persistence, creativity, and personality that are otherwise difficult to obtain

15. But Letters have their Problems • May reflect the writing skills of the referee more than the qualities of the applicant • Ambiguous language • Problem of the omission • Hard to determine reliability, inter-rater agreement, and validity • Time intensive for letter writers & readers

16. What is the ETS® PPI? • Rating system for advisors to rate prospective graduate students • Supplement to GRE • Based on extensive research on the critical attributes for success • Pilot tested with ETS summer interns; Project 1000 • Reveals candidate’s critical personal attributes beyond grades, test scores, and recommendation letters • Gives a more complete picture of a candidate’s potential • Enriches the evaluation process—offers greater opportunity for students • Gives students the opportunity to provide evidence of a broad range of capabilities • Helps create a more diverse and successful graduate program

17. PPI scales and items Resilience Accepts feedback without getting defensive Works well under stress Can overcome challenges and setbacks Works extremely hard Planning and Organization Sets realistic goals Organizes work and time effectively Meets deadlines Makes plans and sticks to them Ethics and Integrity Is among the most honest persons I know Maintains high ethical standards Is worthy of trust from others Demonstrates sincerity Knowledge and Creativity Has a broad perspective on the field Is among the brightest persons I know Produces novel ideas Is intensely curious about the field Communication Skills Speaks in a clear, organized, and logical manner Writes with precision and style Speaks in a way that is interesting Organizes writing well Teamwork Supports the efforts of others Behaves in an open and friendly manner Works well in group settings Gives criticism/feedback to others in a helpful way

21. Findings and Next Steps • PPI has good psychometric properties • Reliable • Measures several factors reliably (cognitive and noncognitive) • Predicts GPA in high school over standardized tests • Faculty ratings match mentor ratings • Project 1000 analysis underway • Is well received • Will be part of GRE starting summer 2009 • Fall 2009 – initial PPI data collection from several universities • Link PPI to variety of graduate school outcomes • Long term consequential validity • Will signal the importance of personal attributes • Will help create more diverse and successful graduate programs

22. Summary • Several uses for personal attributes • Self assessments for career decision making • High stakes assessments • For high stakes, several promising approaches for “personal attribute” assessment • Biodata • Situational Judgment Tests • Ratings by Others (e.g., teachers), as in PPI • We will know a lot more in one or two years! • Long term consequential validity • Will signal the importance of personal attributes • Will help create more diverse and successful classes/graduate programs