650 likes | 663 Views
Explore assessment methodologies and outcomes in healthcare simulation education, including tools for success, performance measurement, and assessment reliability and validity.
E N D
From Learning Objectives to OutcomesAssessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER) University of Pittsburgh
Assessment • Why • What • When • Where • How
What is a simulation for? • Education • Assessment • Fun • 1 and 2
TESTING TEACHING Individuals Teams The Simulation Advantage Psychomotor Skills • Decision Making Skills and Decisions
Psychomotor Skills Decision Making Base Knowledge Communications Skills Teamwork Skills Professionalism Skills The Job of Healthcare Educators
Learning Objective • You have to have a clear reason of why you are running any simulation.
Where does a simulation fit? • It depends on what your doing!
Global Objectives • Course Objectives • 1. Demonstrate a proper patient assessment • 2. Demonstrate following the emergency protocol • 3. Demonstrate Airway Management Skills
Course Schedule / Flow • Sim 1 • Sim 2 • Lecture • Skills Lab • Sim 3 • Sim 4 Course Objectives • Session Objective • Ventilate w BVM • Recognize Narcotic Overdose • Activate Protocol
How do you determine “success” of a simulation? • Students liked it • Learning objectives achieved • Students passed the scenario • Not Sure
How do you determine “success” of a simulation? Instructor Number 1 Instructor Number 2 Instructor Number 3
How do you determine “success” of a simulation? Instructor Number 1 Instructor Number 2 Instructor Number 3 Measure Here
Tools for Success • Make it easy for your faculty to stay on track • “In my opinion…..”
Assessment Methodology • Identify methods for measuring core aspects in simulation education / assessment • Discuss development and deployment of measurement tools
Student Learning Outcomes Teaching & Learning Assessment Evaluation Curriculum
Assessment and Evaluation • Assessment and Evaluation often used interchangeably • Assessment – learner outcomes • Evaluation – course or program outcomes
Rate the Performance • Pass or Fail • Score Performance • Procedural Check
Assessment • Why • What • When • Where • How Rosen, MA et al. Measuring Team Performance in Simulation-Based Training: Adopting Best Practices for Healthcare. Sim Healthcare 3:2008;33–41.
Assessment of clinical skills/competence/performance ACTION PERFORMANCE COMPETENCE KNOWLEDGE Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.
Performance Assessment • Basic to performance – Do they know it and know how? • Competence – Can they do it? • Performance – Do they do it?
DOES SHOWS HOW KNOWS HOW Knows When KNOWS Assessment of clinical skills/competence/performance Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.
DOES Practice Review Audits, Video, UC-SP, 360O SHOWS HOW Performance Tests OSCE, TaskDemo, SIM/SP KNOWS HOW Clinical Based Tests Pt Mgt Qs, Essay, Oral Knows When KNOWS Knowledge Tests MCQs, Essay, Oral exam Assessment of clinical skills/competence/performance Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.
Assessment • Formative Assessment • Lower stakes assessment • One of several assessments over time of course or program • May be evaluative, diagnostic, or prescriptive • Often results in remediation or progression to next level • Summative Assessment • Higher stakes assessment • Generally final course or program assessment • Primary purpose is performance evaluation • Often results in a Go-No Go outcome
Assessment • Assessment data sources • Live observation • Video recording • SIM device logs • Assessors • Instructors • Observers • SIM/Patients • Peers • Participants • Instruments • AV annotation • Pencil/Paper • Scantron form • TabletPC or PDA
Variables in Assessment STUDENT PATIENT ASSESSMENT EVALUATOR
Assessment Validity • Validity – • Degree to which an assessment tool measures what it is intended to measure • Construct validity • Accuracy of tool to measure intended construct • Concurrent validity • Content validity • Predicative validity
Assessment Reliability • Test Reliability – • consistency of a test to yield the same measure over time (does not assure validity) • Test-Retest Reliabilityconsistency among different administrations • Parallel Forms Reliabilityto assure that memory effects do not occur use a different pre- and posttest • Inter-Rater Reliabilitytwo or more observers rate the same subjects with strong positive correlation of their ratings
Variables in Assessment STUDENT PATIENT ASSESSMENT EVALUATOR
Assessment Improvement • Improving Reliability and Validity • Base assessment on outcome/objectives > event triggers > observable behavior > behavioral rating > assess against competencies • Define <low – medium – high> performance • Use of Rubric or rating metric • Use (video) training examples of performance • Employ a quality assurance/improvement system
Assessment Based on Objectives and Events Rosen, MA, et al A measurement Tool for Simulation –Based Training in Emergency Medicine: The Simulation Module for Assessment of Resident Targeted Event Responses (SMARTER) Approach. Sim Healthcare 3:2008;170-179.
Event – Behavior – Scale - Report Rosen, MA, et al. Measuring Team Performance in Simulation –Based Training: Adapting Best Practices for Healthcare. Sim Healthcare 3:2008;33-41.
Assessment Metrics • Procedural or Check List assessment • Global Rating assessment
Assessment Metrics • Procedural or Check List assessment *Assist
Assessment Metrics • Global Rating assessment Pts.
Assessment Metrics • Procedural or Check List assessment • Global Rating assessment < EXAMPLES> • Pros and Cons to each • Multiple modes & points vs. Single assessment • QA/QI Metric – collective vs single assessment
Brett-Fleegler, MB et al. A simulator-based tool that assesses pediatric resident resuscitation competency. Pediatrics 121(3):2008;597-603.
Gaba, DM et al. Assessment of Clinical Performance during Simulated Crises Using Both Technical and Behavioral Ratings. Anesthesiology, 89(1)July 1998:8-18.
Guise, JM et al. Validation of a Tool to Measure and Promote Clinical Teamwork. Sim Healthcare 3:2008;217–223.