1 / 50

Real and Imaginary Concerns in Assessing College Readiness Nathan R. Kuncel Department of Psychology

Real and Imaginary Concerns in Assessing College Readiness Nathan R. Kuncel Department of Psychology. Favored Career Choices: In Rank Order. Jedi Knight Star Fleet Captain Batman Playboy Millionaire Psychology Professor Evolutionary Biologist Economist

gitel
Download Presentation

Real and Imaginary Concerns in Assessing College Readiness Nathan R. Kuncel Department of Psychology

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Real and Imaginary Concerns in Assessing College ReadinessNathan R. KuncelDepartment of Psychology

  2. Favored Career Choices: In Rank Order • Jedi Knight • Star Fleet Captain • Batman • Playboy Millionaire • Psychology Professor • Evolutionary Biologist • Economist • Brilliant but Misunderstood Novelist

  3. Jedi Knight Innate heritable ability, unrelated to social class, focused through mentoring and intense training, high self control

  4. Star Fleet Captain • Born but seasoned leaders who succeed through managed team effectiveness, charismatic and bold

  5. Batman Single formative childhood experience that motivates massive skill investment, augmented by tremendous family wealth, intense and disagreeable

  6. Professor of Psychology Masochistic enthusiasm for studying nearly intractable research questions, fairly friendly

  7. Stories We Tell About Achieving Success • Predictable versus Unpredictable • Book Smarts versus Street Smarts • General versus Specific Aptitudes • Social Class Dependent versus Not • Talent versus Effort • Linear versus Asymptotic

  8. Stories about Tests • Tests don’t even predict grades well • Tests don’t add anything to prediction • Tests don’t predict anything other than grades • Cognitive tests only predict academic outcomes • Cognitive tests are only proxies for SES • Beyond a certain point, scores don’t matter Stories about Traditional Alternatives • Letters of Recommendation • Personal Statements • Interviews Frightening Cautionary Tales about the Future • Faking • Bias • Unreliability

  9. Assertion: Tests Don’t Even Predict Grades Well What does “well” mean and how do we quantify it?

  10. Zombie Plague • Plague sweeps the world • Plague transforms 60% of infected people into horrifying and mindless monsters.

  11. A Partial Treatment Is Developed! What is the correlation?

  12. Tests and Grades Berry and Sackett (2009) examined SAT-grade r’s in a sample of 165,000 students from 41 schools Corrected for restriction of range using school-specific applicant pools Obtained individual course grades for each course, and computed validity for each course (148,072 validity coefficients) This removes influence of student choice of courses Estimate r between SAT and common course portfolio

  13. Berry and Sackett (2009) results

  14. Story:Tests Don’t Add Anything to Prediction

  15. Percent Earning a 3.8 or Higher Graduate GPA Bridgeman, Burton, & Cline (2009)

  16. Story: Tests only predict 1st year grades Paper submitted to American Psychologist: “As is well known, the SAT predicts nothing but first year grades” [with no citation]

  17. 7 Admissions Tests GRE-T GRE-S MCAT LSAT GMAT MAT PCAT 8 Outcomes 1st Year GPA Graduate GPA Faculty Ratings Degree Attainment Citation Counts Research Productivity Quals./Comps. Exams Licensing Exams Evidence Across All Admissions Tests: Kuncel & Hezlett (2007) Science Results are based on over 600,000 students across over 3,000 independent samples

  18. Kuncel & Hezlett (2007). Science, 315, 1080-1081.

  19. If the critics were right….

  20. Book Smarts vs. Street Smartsor Academic vs. Practical IntelligenceorBook Learning vs. Common Sense

  21. Book Smarts vs. Street Smarts • Fundamental argument is that different types of intelligence are needed in academic versus real world contexts • It is argued that academic contexts require an “academic intelligence” or book smarts • It is my belief that these arguments are based (in part) on an overly narrow conceptualization of the tasks that comprise academic performance

  22. Academic Formulated by others Well-defined Presents complete information Single method for obtaining answer Not embedded in ordinary experience Little or no intrinsic interest Practical Requires problem recognition and formulation Ill-defined Requires information seeking Multiple acceptable solutions Multiple paths to solution Embedded in ordinary experience Requires motivation and personal involvement Academic versus Practical Tasks

  23. Things Students Do Before Taking a Test • Determine study objectives and methods • Manage goal conflicts • Coordinate work with other classmates • Seek additional information or materials • Handle school related finances • Negotiate with peers and faculty • Avoid counterproductive school behaviors • Structure effective communications

  24. Traditional Classroom Success Written and Oral Communication Personal Discipline Resolving Goal Conflicts Studying and Learning Proficiency Sustained Goal Directed Effort Interactive Learning and Team Performance Administration Interpersonal Proficiency Non-Classroom Performance Development of Life Goals and Values A Model of Undergraduate Student Performance Dimensions Kuncel (2002); Kuncel, Campbell, Hezlett, & Ones (2001)

  25. Cross-Situational Validity: A Direct Test • However, we still lack a direct test: • Examine the validity of a single test developed for academic settings but used in both academic and work settings • Ideally we would also establish the relationship between this test and other cognitive ability measures • Unfortunately, a single ability measure is rarely used for both personnel selection and educational admissions decisions • With one notable exception….

  26. MAT – Academic Performance Kuncel, Hezlett, & Ones (2004) JPSP

  27. MAT – Transitional Variables and Creativity Kuncel, Hezlett, & Ones (2004) JPSP

  28. MAT – Work Criteria Kuncel, Hezlett, & Ones (2004) JPSP

  29. Story: Predictive Power of Cognitive Ability Tests is an artifact of SES

  30. Its Storytime… • In the interest of truth in advertising, the SAT should simply be called a “wealth test” - Guiner (Undated) • “The SAT merely measures the size of student’s houses” - Kohn (2001) • “Only thing the SAT predicts well now is socioeconomic status” – Colvin (1997)

  31. Recent Claims in the American Psychologist • “…SAT I scores lose any ability to predict freshman year grades if the regression analyses control for socioeconomic status” (p. 100) – Crosby, Iyer, Clayton, and Downing (2003) • “…SAT scores used for college admissions do not predict freshman year grades when socioeconomic status is controlled” (p. 1023) – Biernat (2003) • Atkinson and Geiser (2009) claim that SAT coefficients are “decisively diminished” when SES and HSGPA are controlled.

  32. Consistent SES Findings Moderate observed relationship Does not go to zero when controlling for SES

  33. University of California System Data

  34. “Ok, but I know someone who had really high test scores and they didn’t do as well as another friend who had lower test scores. So scores only matters to a certain degree, right?”

  35. More is also better at school Arneson and Sackett (2009)

  36. Coward & Sackett (1990) • 174 studies on the relationship between intelligence and job performance • Studies used the 9 scale GATB (General Aptitude Test Battery) • Total sample size across all studies was a substantial 36,614 • Found overwhelming support for a linear relationship between ability and job performance • The commonly held notion appears to be incorrect

  37. Traditional Supplementary Predictors • Letters of Recommendation • Personal Statements • Interviews

  38. Letters GRE Result suggest no incremental validity for letters, save for degree completion. Note that all letters were structured and quantitative. Kuncel, Vanelli, & Ones (2009)

  39. Personal Statements Cooper-Murphy, Klieger, Borneman, & Kuncel (2007) College and University

  40. Admissions Interview: Results

  41. Yet There is Good Stuff Out There

  42. Study Habits, Skills, and Attitudes Crede & Kuncel (2007) Perspectives in Psychological Science

  43. Class Attendance: Being There Crede, Roch, & Kieszczynka (2010) Review of Educational Research

  44. Personality Predictors of Graduate Student Performance Rigdon & Kuncel, 2010

  45. Cautionary Tales • Faking and Coaching • Most alternatives that have been proposed would be highly susceptible to deliberate faking or test preparation coaching • Personality Assessments • Other ratings • Study habits, attitude, and skill inventories • Placement and guidance • Biodata • Verifiable content • Situational judgments tests

  46. Fairness and Bias • Extensive research on standardized test scores for bias • Social class has also been scrutinized. • Data for letters of recommendation and personal statements in academic settings is thinner. • A summary of current knowledge on the next slide…

  47. New Alternatives Should be Subjugated to Extensive Study for Bias Before Implementation

More Related