1 / 19

Evaluating Simulation Technologies

Evaluating Simulation Technologies. developed by Patricia Youngblood, PhD Associate Director for Evaluation, and Parvati Dev, PhD Director SUMMIT Research Lab. The SUMMIT Evaluation Framework. Beta-testing (system reliability) Review by content experts Usability testing

diep
Download Presentation

Evaluating Simulation Technologies

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluating Simulation Technologies developed by Patricia Youngblood, PhD Associate Director for Evaluation, and Parvati Dev, PhD Director SUMMIT Research Lab

  2. The SUMMIT Evaluation Framework • Beta-testing (system reliability) • Review by content experts • Usability testing • Validity testing • Assessing learning outcomes • Integration into the curriculum • Transfer of learning to clinical practice

  3. 6.Curriculum Integration 5.Learning Outcomes 1.Beta testing 3.Usability testing 4.Validity testing 2.Content review 7.Practice          n.a.    n.a.        n.a.            n.a.     Educational Evaluation at SUMMIT Pelvic Exam CBRNE Event CathSim ShoulderSim Virtual ER  WebSP PharmaPaC n.a.  Virtual Labs  LapSim

  4. Pelvic Exam Trainer: Beta Testing Is student performance data captured consistently? • Basic Functionality • Measurement Capabilities

  5. Content Experts Medical Students Pelvic Exam Trainer: Validity Testing • Face validity • Content validity • Construct validity

  6. 1. Pelvic exam trainer 2. Standard mannequin 3. Control Evaluation Design: Randomized Controlled Trial Do those trained on the simulator perform better than those trained on the mannequin? viewed pelvic exam video 87 medical students Simulator Assessment Patient Educator Assessment

  7. Simulation of a CBRNE Event(photos & screenshots courtesy of Forterra Systems, Inc.) Characters in VR world ED doctors & nurses walking wounded triage decon tent

  8. Review by content experts What are content experts’ opinions of the VR training? Formative Evaluation of VR Training(photos courtesy of Forterra Systems, Inc.) Usability Do trainees have any problems using it?

  9. A Construct Validity Study: Mentice VR Shoulder Arthroscopy Simulator Is the performance of: surgeons > trainees > medical students? Results (p<.05) Significant differences on: • total time • manipulation of hook • number of tacks Mentice VR Shoulder Arthroscopy Simulator

  10. Learning Outcomes: CathSim Study Do those trained with the simulator perform better than those trained with the mannequin arm? • Simulator group performed better than Mannequin arm group (p=.045*). IV Insertion (CathSim)

  11. Comparison of HPS & Virtual ED:A Learning Outcomes Study Does training on the simulator improve performance on the posttest? • four training cases; • pre & post assessment cases; • simulation experience plus • debrief with instructor.

  12. Virtual ED Simulator (n=16) pretest case  4 training cases  posttest case Human Patient Simulator “HPS” (n=14) pretest case  4 training cases  posttest case Evaluation Design: ED Team Training Is there a significant difference between the two student groups? 30 medical students & interns

  13. WebSP: Interactive Patient Cases What are the best ways to integrate the simulation into the curriculum? What are the barriers to implementation? • Beta testing • Content review of new cases • Usability testing • Learning outcomes • Integration into curriculum

  14. Curriculum Integration PharmaPaC Virtual Labs

  15. Box trainer Lapsim simulator VR to OR Study: LapSim Simulator & Box Trainer Do those trained on LapSim perform better than those trained on the Box trainer or those with no training? • 3 treatment groups • 4 x 45 min training sessions • 3 laparoscopic tasks • Assessment in animal lab

  16. Evaluation Design: VR to OR Study 46 surgically naïve medical students LapSim Surgical Simulator (n=17) Box Trainer Simulator (n=16) Control Group (n=13) Perform tasks on an anesthetized animal in the lab

  17. VR to OR: Final assessment in animal lab Findings (p<.05): • LapSim VR group outperformed the Box Trainer group on 3 measures; • LapSim VR group outperformed the Control Group on 2 measures; • Box Trainer group outperformed the Control Group on only 1 measure.

  18. The SUMMIT Evaluation Framework • Beta-testing (system reliability) • Review by content experts • Usability testing • Validity testing • Assessing learning outcomes • Integration into the curriculum • Transfer of learning to clinical practice

  19. Student-Centered Learning No matter how information is presented, it will be of little use if the students are not given sufficient time to interact with it and discuss it. But a quality lesson, designed well and taught well, will be effective no matter which technology is used. From Diamond, RM (1998) Chapter 12: Selecting & Using Technology in Designing & Assessing Courses & Curricula; Jossey-Bass Publishers; San Francisco.

More Related