360 likes | 565 Views
Web-based Screening of Applicant Personal Characteristics Can Now Be Done Reliably and Validly Should You Use It? The Development and Real-world Implementation of Computer-based Assessment for Sampling Personal Characteristics. Harold Reiter Kelly Dore. Stemmler Fund.
E N D
Web-based Screening of Applicant Personal CharacteristicsCan Now Be Done Reliably and ValidlyShould You Use It?The Development and Real-world Implementation of Computer-based Assessment for Sampling Personal Characteristics • Harold Reiter • Kelly Dore
Stemmler Fund Web-based Screening of Applicant Personal Characteristics
Why I’m Worried • Length of talk • It’s the economy, stupid
combining Two Concepts “If you nail two things together that have never been nailed together before… some s****k will buy it”
Thing 1 & Thing 2 THING 1 – situational judgment testing THING 2 – online testing
Situational Judgment Testing Scores responses to hypothetical but realistic scenarios 1950’s, 1960’s used for managerial assessments 1990’s – for workforce, medical and dental schools
Play first casper video here • Question 1 – What is your decision and why? • Question 2 – Would your response be any different if this family has four younger children expected to attend the same school in future? • Question 3 – Would your response be any different if the school was in a district with a wide range of income – some very privileged families, some underprivileged?
Personal/Professional Characteristics – financial cost $50,000,000,000
Standard measures of personal/professional characteristics ESSAYLIB.com High-quality custom papers starting from $10/page All papers written from scratch We hire professional writers only Original and creative work Custom Personal Statements
Predictive Validity COGNITIVE PERSONAL/PROFESSIONAL CHARACTERISTICS 3rd yr GPA GPAMMI .6 .4 .2 0 USMLE Step 3 MCCQE I cogn Clinical Clerk MCCQE I C2LEO MCCQE II pass
Characteristics of the desired instrument • Absolute Requirements • Multiple samples • Withhold control from examinees • Relative Requirements • Highly context dependent • Score hard with hard, soft with soft • Societal Requirements • Cheap • Fair to the less privileged
How small problems loom large when changing medium • OPTION # 1 • Moving from MMI to online video-based SJT
Belgian Medical & Dental Studies Admissions Examination 1999 – 2002 5,444 test-takers Video stems 30 MCQ responses 723 admitted Internship performance R = 0.22, p < 0.05 Stopped after 2002 - financial and technical constraints
2003 – 2008: Psychometric research 2008 – 2010: Technological development 2010 – 2012: Implementation CASPer Computer-based Assessment for Sampling Personal characteristics
8 Video Vignettes & 4 Self-Descriptive Non-clinical 3 follow-up questions 5 min short answer Scored by two raters CASPer Computer-based Assessment for Sampling Personal characteristics
Students cheat • OPTION # 2 • How can a non-proctored online test be secure?
CASPer as a pre-interview Screen 2006 – MCC – 110 applicants 2007 - NBME – 167 applicants Reliable Inter-RaterCorrelation with MMI .9 .6 .3 0
Predictive Validity COGNITIVE PERSONAL/PROFESSIONAL CHARACTERISTICS 3rd yr GPA GPAMMI .6 .4 .2 0 USMLE Step 3 MCCQE I cogn Clinical Clerk MCCQE I C2LEO MCCQE II pass
Stemmler Fund 2006 MCC & 2007 NBME studies: • MCC 110 – 63 MCCQE Part I, 53 MCCQE Part II • NBME 167 – 46 MCCQE Part I, 12 MCCQE Part II
Predictive Validity COGNITIVE PERSONAL/PROFESSIONAL CHARACTERISTICS CASPer .6 .4 .2 0 MCCQE Part I med expert MCCQE I C2LEO MCCQE I PHELO p < 0.05 MCCQE II C2LEO
Michael G. DeGroote School of Medicine TARGET < 5% Incomplete
Ongoing research • OPTION # 3 • What further research can be done/is being done?
What’s important to me • OPTION # 4 • Under what circumstances, if any, does CASPer become useful to me?
Thanks for Participating! • Harold Reiter MD MEd FRCPC DABR • reiterh@mcmaster.ca • Kelly Dore PhD • dore@mcmaster.ca
How small problems loom large when changing medium • OPTION # 1 • Moving from MMI to online video-based SJT
Students cheat • OPTION # 2 • How can a non-proctored online test be secure?
Ongoing research • OPTION # 3 • What further research can be done/is being done?
What’s important to me • OPTION # 4 • Under what circumstances, if any, does CASPer become useful to me?