1 / 81

Measuring Healthcare Outcomes using Serious Games, Gamification, and Virtual Reality

This session outlines the importance of data capture and using sound research methodology to determine the best use of your game-based learning intervention. We explore pertinent learning and behavioral theories, including outcome levels, elements of fidelity, and appraise different strategies that have both succeeded and failed in the use of games in healthcare education. More importantly, we explore the practical aspects of embedding data collection to prove the game’s impact to healthcare education and health.

Download Presentation

Measuring Healthcare Outcomes using Serious Games, Gamification, and Virtual Reality

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Measuring healthcare outcomes using serious games, gamification, and virtual reality Todd P Chang, MD MAcM serious play conference 2018.07.11 (Manassas, VA)

  2. Disclosures / Conflicts of Interest American Heart Association National Board of Medical Examiners Oculus from FaceBook • • • Independent Contractor with SBC Med Sim, LLC • 2

  3. WHY MAKE HEALTHCARE GAMES AT ALL? 3

  4. We do it for the patients 4

  5. Objectives 1 2 3 Map out measurable Outcomes for both Mediating & Moderating games Define the 4 Outcome levels for measuring Effectiveness of a Game Discuss three case studies measuring effectiveness of Games on patient outcomes

  6. HOW CAN GAMES INFLUENCE PATIENT OUTCOMES?

  7. 7

  8. Landers et al. 2014 Sim & Gaming Mediating Moderating • Knowledge • Skill • Attitude 8

  9. MEDIATING Going through the game improves KSA 9

  10. 10

  11. 1000 CUT JOURNEY Virtually experience life as a black child

  12. Landers et al. 2014 Sim & Gaming Mediating Moderating • Knowledge • Skill • Attitude 12

  13. The game or gamification elements improve engagement of an otherwise effective tool MODERATING

  14. 14

  15. Landers et al. 2014 Sim & Gaming Mediating Moderating Going through the game improves KSA The game improves engagement of an effective tool 15

  16. Outcome 00 Mood prior to VR Outcome 01 Use Frequency Outcome 02 Change in Mood 16

  17. Outcome 04 Health Metric (Result) Outcome 03 Knowledge Application (Behavior) Outcome 00 Prior Outcomes Outcome 01 Use Scores Outcome 02 Knowledge Gain 17

  18. Outcome 03 Health Metric (Result) Outcome 01 Use (Behavior) Outcome 02 Attitude 18

  19. What if the game user is the healthcare provider, not the patient? 19

  20. 20

  21. OUTCOME LEVELS Objective 2: Kirkpatrick’s 4 Levels of Evaluation

  22. Kirkpatrick’s Levels of Evaluation Evaluation and Feedback Sullivan GM. J Med Educ. 2011;3(2):121-4.

  23. Kirkpatrick’s Levels of Evaluation Evaluation and Feedback Reaction Sullivan GM. J Med Educ. 2011;3(2):121-4.

  24. Kirkpatrick’s Levels of Evaluation Evaluation and Feedback Learning Reaction Sullivan GM. J Med Educ. 2011;3(2):121-4.

  25. Kirkpatrick’s Levels of Evaluation Evaluation and Feedback Behavior Learning Reaction Sullivan GM. J Med Educ. 2011;3(2):121-4.

  26. Kirkpatrick’s Levels of Evaluation Evaluation and Feedback Results Behavior Learning Reaction Sullivan GM. J Med Educ. 2011;3(2):121-4.

  27. Barsuk et al. Crit Care Med 2009;37(10):2697-701. Barsuk et al. Acad Med 2010;85(10 Suppl):S9-12. Barsuk et al. MJ Qual Saf 2014;23(9):749-56. • • •

  28. Outcome C Prior Knowledge Outcome A Use Scores Outcome B Knowledge Gain

  29. Outcome C Prior Knowledge Outcome A Use Scores Outcome B Knowledge Gain Outcome 04 Health Metric (Result) Outcome 03 Prior Patient State (Result) Outcome 01 Provider Action (Behavior) Outcome 02 Patient State (Result)

  30. CAN MY GAME PREDICT HOW MUCH IMPROVEMENT WILL OCCUR? A question of Validity

  31. Discriminant Validity Testing Expected Low Performers Expected High Performers Uncovers any bias due to: • Differences in Video Game Experience • “Game Cheating” behavior • Lack of Functional Fidelity in a Game • A different target audience

  32. Gerard et al. Validity Evidence for a Serious Game to Assess Performance on Critical Pediatric Emergency Medicine Scenarios. Simul Healthc 2018;13(3):168-80.

  33. 3 CASE STUDIE S

  34. HOW TO MEASURE ‘MULTI-PATIENT CARE’? A discriminant validity study

  35. Background • With higher patient volumes but fewer providers, multi-patient care is now common • Poor multi-patient care leads to increased errors • Multi-patient care is not routinely taught nor universally assessed

  36. Purpose To validate a serious games version of a pediatric ED that measures multi-patient care based on experience: 1. Undergraduate students (jrs, srs) 2. Medical students (MS-II) 3. Resident physicians (PGY-1 through 3) 4. Attending / Fellow physicians (PGY-4+)

  37. Acknowledgments to the Stemmler Fund & BreakAway Games, Ltd

  38. 43

  39. Outcomes & Analyses Predictors Experience Multi-tasking ability Video Game frequency Game Outcomes % Sentinel Orders Time to 1stSentinel Order Time to 2ndSentinel Order Time to Discharge % Patients Seen # correct Diagnoses # Differential Diagnoses • • • • • • • • • • 45

  40. Results Undergrads 11 MSIIs 22 Residents 15 Attgs 23 n 46

  41. Results # Patients Seen 100 75 50 25 0 Undergrad MSII Residents Attgs 47

  42. Results Time-to-Critical Action (game seconds) 750 500 250 0 Time to SO1 Time to SO2 Time to d/c Undergrad MSII Resident Attg 48

  43. Results • MANCOVA analyses revealed a statistically significant difference in game performance after controlling for video game experience & multi- tasking ability. • The two outcomes that the game was able to distinguish expertise in were: – # sentinel orders – % correct diagnoses

  44. The one-way MANCOVA showed that there was statistically significant difference (p=0.03) between the skill groups on the combined dependent variables after controlling for game play frequency, F(21, 153) = 0.700, p = .828, Wilks' Λ = .768, partial η2 = .084, and for MTAT placement score, F(21, 153) = 1.365, p = .144, Wilks' Λ = .610, partial η2 = .152. Follow up univariate one-way ANCOVAs were performed. A Bonferroni adjustment was made such that statistical significance was accepted at p < .008. There were statistically significant differences in adjusted means for total percentage of sentinel orders completed (F(3, 59) = 31.702, p < .0005, partial η2 = .617), and number of correct diagnoses made per patient seen(F(3, 59) = 21.441, p < .0005, partial η2 = .522). 50

More Related