1 / 65

From Learning Objectives to Outcomes Assessment During Simulation

Explore assessment methodologies and outcomes in healthcare simulation education, including tools for success, performance measurement, and assessment reliability and validity.

mestas
Download Presentation

From Learning Objectives to Outcomes Assessment During Simulation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. From Learning Objectives to OutcomesAssessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER) University of Pittsburgh

  2. Assessment • Why • What • When • Where • How

  3. The Future of Simulation………

  4. What is a simulation for? • Education • Assessment • Fun • 1 and 2

  5. TESTING TEACHING Individuals  Teams The Simulation Advantage Psychomotor Skills • Decision Making Skills and Decisions

  6. What is “Simulation”….. Really…..

  7. Psychomotor Skills Decision Making Base Knowledge Communications Skills Teamwork Skills Professionalism Skills The Job of Healthcare Educators

  8. Why is this so hard?

  9. Learning Objective • You have to have a clear reason of why you are running any simulation.

  10. Lets Watch

  11. What is most important aspect to debrief?

  12. Where does a simulation fit? • It depends on what your doing!

  13. EACH Sim is part of a bigger plan!

  14. What is this?

  15. Does it belong here?

  16. Course (Global) objectives v sim objectives

  17. Global Objectives • Course Objectives • 1. Demonstrate a proper patient assessment • 2. Demonstrate following the emergency protocol • 3. Demonstrate Airway Management Skills

  18. Course Schedule / Flow • Sim 1 • Sim 2 • Lecture • Skills Lab • Sim 3 • Sim 4 Course Objectives • Session Objective • Ventilate w BVM • Recognize Narcotic Overdose • Activate Protocol

  19. How do you determine “success” of a simulation? • Students liked it • Learning objectives achieved • Students passed the scenario • Not Sure

  20. How do you determine “success” of a simulation? Instructor Number 1 Instructor Number 2 Instructor Number 3

  21. How do you determine “success” of a simulation? Instructor Number 1 Instructor Number 2 Instructor Number 3 Measure Here

  22. You can only keep track of so many things at one time

  23. How many of you feel like this ?

  24. Tools for Success • Make it easy for your faculty to stay on track • “In my opinion…..”

  25. Assessment Methodology • Identify methods for measuring core aspects in simulation education / assessment • Discuss development and deployment of measurement tools

  26. Student Learning Outcomes Teaching & Learning Assessment Evaluation Curriculum

  27. Assessment and Evaluation • Assessment and Evaluation often used interchangeably • Assessment – learner outcomes • Evaluation – course or program outcomes

  28. Performance Assessment

  29. Rate the Performance • Pass or Fail • Score Performance • Procedural Check

  30. Assessment • Why • What • When • Where • How Rosen, MA et al. Measuring Team Performance in Simulation-Based Training: Adopting Best Practices for Healthcare. Sim Healthcare 3:2008;33–41.

  31. Assessment of clinical skills/competence/performance ACTION PERFORMANCE COMPETENCE KNOWLEDGE Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.

  32. Performance Assessment • Basic to performance – Do they know it and know how? • Competence – Can they do it? • Performance – Do they do it?

  33. DOES SHOWS HOW KNOWS HOW Knows When KNOWS Assessment of clinical skills/competence/performance Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.

  34. DOES Practice Review Audits, Video, UC-SP, 360O SHOWS HOW Performance Tests OSCE, TaskDemo, SIM/SP KNOWS HOW Clinical Based Tests Pt Mgt Qs, Essay, Oral Knows When KNOWS Knowledge Tests MCQs, Essay, Oral exam Assessment of clinical skills/competence/performance Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.

  35. Assessment • Formative Assessment • Lower stakes assessment • One of several assessments over time of course or program • May be evaluative, diagnostic, or prescriptive • Often results in remediation or progression to next level • Summative Assessment • Higher stakes assessment • Generally final course or program assessment • Primary purpose is performance evaluation • Often results in a Go-No Go outcome

  36. Assessment • Assessment data sources • Live observation • Video recording • SIM device logs • Assessors • Instructors • Observers • SIM/Patients • Peers • Participants • Instruments • AV annotation • Pencil/Paper • Scantron form • TabletPC or PDA

  37. Variables in Assessment STUDENT PATIENT ASSESSMENT EVALUATOR

  38. Assessment Validity • Validity – • Degree to which an assessment tool measures what it is intended to measure • Construct validity • Accuracy of tool to measure intended construct • Concurrent validity • Content validity • Predicative validity

  39. Assessment Reliability • Test Reliability – • consistency of a test to yield the same measure over time (does not assure validity) • Test-Retest Reliabilityconsistency among different administrations • Parallel Forms Reliabilityto assure that memory effects do not occur use a different pre- and posttest • Inter-Rater Reliabilitytwo or more observers rate the same subjects with strong positive correlation of their ratings

  40. Variables in Assessment STUDENT PATIENT ASSESSMENT EVALUATOR

  41. Assessment Improvement • Improving Reliability and Validity • Base assessment on outcome/objectives > event triggers > observable behavior > behavioral rating > assess against competencies • Define <low – medium – high> performance • Use of Rubric or rating metric • Use (video) training examples of performance • Employ a quality assurance/improvement system

  42. Assessment Based on Objectives and Events Rosen, MA, et al A measurement Tool for Simulation –Based Training in Emergency Medicine: The Simulation Module for Assessment of Resident Targeted Event Responses (SMARTER) Approach. Sim Healthcare 3:2008;170-179.

  43. Event – Behavior – Scale - Report Rosen, MA, et al. Measuring Team Performance in Simulation –Based Training: Adapting Best Practices for Healthcare. Sim Healthcare 3:2008;33-41.

  44. Assessment Metrics • Procedural or Check List assessment • Global Rating assessment

  45. Assessment Metrics • Procedural or Check List assessment *Assist

  46. Assessment Metrics • Global Rating assessment Pts.

  47. Assessment Metrics • Procedural or Check List assessment • Global Rating assessment < EXAMPLES> • Pros and Cons to each • Multiple modes & points vs. Single assessment • QA/QI Metric – collective vs single assessment

  48. Brett-Fleegler, MB et al. A simulator-based tool that assesses pediatric resident resuscitation competency. Pediatrics 121(3):2008;597-603.

  49. Gaba, DM et al. Assessment of Clinical Performance during Simulated Crises Using Both Technical and Behavioral Ratings. Anesthesiology, 89(1)July 1998:8-18.

  50. Guise, JM et al. Validation of a Tool to Measure and Promote Clinical Teamwork. Sim Healthcare 3:2008;217–223.

More Related