Reflections on the Migration of Medical Simulators from Training Tools to Assessment Tools

1. Reflections on the Migration of Medical Simulators from Training Tools to Assessment Tools Rich Shavelson Stanford Education Assessment Lab Invited Talk Simulation in Medical Education Seminar Stanford Hospital March 14, 2007

2. Overview • Motivation for Talk • Analogy and Implications: Medical Simulators and Job Performance Measurement • Evaluation of Technical Quality of Simulator Assessments • Open Discussion

3. Motivation For Talk Before • Requests for advice on how to score performance collected with medical training simulators • SUMMIT—Surgical laparoscopic simulators • Creating scores from data reported by simulator • Defining reference groups • Defining performance benchmarks • Technical quality • Pediatric airway management simulator • Separating training and assessment • Developing a profile of performance • Task sampling • Technical quality Ideal & Possible Solution After Imple-menting Solution

4. Analogy and Implications: Medical Simulators and Job Performance Measurement • A great deal of work has been done on job & education performance measurement that applies to medical simulators as assessment tools. • My enduring interest

5. Analogy Continued • Approaches to performance measurement • Construct definition • Task sampling definition • Rapprochement • Cold reality Universe of Performance of Interest Universe of Possible Tasks for Assessment Universe of “Do-Able” Tasks Universe of “Do-Able” Task Formats Tasks & Scoring on Test

6. Evaluation Of Technical Quality • Reliability • Classical test theory (“Cronbach’s Alpha”)—simple task sampling • Generalizability theory—task sampling • Item response theory—construct driven • Validity • Construct • Content • Predictive

7. Thanks…Your Turn!