1 / 36

Harold Reiter Kelly Dore

Web-based Screening of Applicant Personal Characteristics Can Now Be Done Reliably and Validly Should You Use It? The Development and Real-world Implementation of Computer-based Assessment for Sampling Personal Characteristics. Harold Reiter Kelly Dore. Stemmler Fund.

marlee
Download Presentation

Harold Reiter Kelly Dore

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Web-based Screening of Applicant Personal CharacteristicsCan Now Be Done Reliably and ValidlyShould You Use It?The Development and Real-world Implementation of Computer-based Assessment for Sampling Personal Characteristics • Harold Reiter • Kelly Dore

  2. Stemmler Fund Web-based Screening of Applicant Personal Characteristics

  3. Why I’m Worried • Length of talk • It’s the economy, stupid

  4. combining Two Concepts “If you nail two things together that have never been nailed together before… some s****k will buy it”

  5. Thing 1 & Thing 2 THING 1 – situational judgment testing THING 2 – online testing

  6. Situational Judgment Testing Scores responses to hypothetical but realistic scenarios 1950’s, 1960’s used for managerial assessments 1990’s – for workforce, medical and dental schools

  7. Play first casper video here • Question 1 – What is your decision and why? • Question 2 – Would your response be any different if this family has four younger children expected to attend the same school in future? • Question 3 – Would your response be any different if the school was in a district with a wide range of income – some very privileged families, some underprivileged?

  8. Personal/Professional Characteristics – financial cost $50,000,000,000

  9. Standardized Tests in 2011

  10. Standard measures of personal/professional characteristics ESSAYLIB.com High-quality custom papers starting from $10/page All papers written from scratch We hire professional writers only Original and creative work Custom Personal Statements

  11. Personality Testing – Big 5 Factor, CPI…

  12. Predictive Validity COGNITIVE PERSONAL/PROFESSIONAL CHARACTERISTICS 3rd yr GPA GPAMMI .6 .4 .2 0 USMLE Step 3 MCCQE I cogn Clinical Clerk MCCQE I C2LEO MCCQE II pass

  13. Characteristics of the desired instrument • Absolute Requirements • Multiple samples • Withhold control from examinees • Relative Requirements • Highly context dependent • Score hard with hard, soft with soft • Societal Requirements • Cheap • Fair to the less privileged

  14. Medical College Admissions Test – 94,000

  15. How small problems loom large when changing medium • OPTION # 1 • Moving from MMI to online video-based SJT

  16. Belgian Medical & Dental Studies Admissions Examination 1999 – 2002 5,444 test-takers Video stems 30 MCQ responses 723 admitted Internship performance R = 0.22, p < 0.05 Stopped after 2002 - financial and technical constraints

  17. 2003 – 2008: Psychometric research 2008 – 2010: Technological development 2010 – 2012: Implementation CASPer Computer-based Assessment for Sampling Personal characteristics

  18. 8 Video Vignettes & 4 Self-Descriptive Non-clinical 3 follow-up questions 5 min short answer Scored by two raters CASPer Computer-based Assessment for Sampling Personal characteristics

  19. Play second casper video here

  20. Students cheat • OPTION # 2 • How can a non-proctored online test be secure?

  21. CASPer as a pre-interview Screen 2006 – MCC – 110 applicants 2007 - NBME – 167 applicants Reliable Inter-RaterCorrelation with MMI .9 .6 .3 0

  22. Predictive Validity COGNITIVE PERSONAL/PROFESSIONAL CHARACTERISTICS 3rd yr GPA GPAMMI .6 .4 .2 0 USMLE Step 3 MCCQE I cogn Clinical Clerk MCCQE I C2LEO MCCQE II pass

  23. Stemmler Fund 2006 MCC & 2007 NBME studies: • MCC 110 – 63 MCCQE Part I, 53 MCCQE Part II • NBME 167 – 46 MCCQE Part I, 12 MCCQE Part II

  24. Predictive Validity COGNITIVE PERSONAL/PROFESSIONAL CHARACTERISTICS CASPer .6 .4 .2 0 MCCQE Part I med expert MCCQE I C2LEO MCCQE I PHELO p < 0.05 MCCQE II C2LEO

  25. Michael G. DeGroote School of Medicine TARGET < 5% Incomplete

  26. TARGET < 5% Incomplete

  27. Ongoing research • OPTION # 3 • What further research can be done/is being done?

  28. What’s important to me • OPTION # 4 • Under what circumstances, if any, does CASPer become useful to me?

  29. Thanks for Participating! • Harold Reiter MD MEd FRCPC DABR • reiterh@mcmaster.ca • Kelly Dore PhD • dore@mcmaster.ca

  30. How small problems loom large when changing medium • OPTION # 1 • Moving from MMI to online video-based SJT

  31. Students cheat • OPTION # 2 • How can a non-proctored online test be secure?

  32. Ongoing research • OPTION # 3 • What further research can be done/is being done?

  33. What’s important to me • OPTION # 4 • Under what circumstances, if any, does CASPer become useful to me?

More Related