820 likes | 852 Views
This session outlines the importance of data capture and using sound research methodology to determine the best use of your game-based learning intervention. We explore pertinent learning and behavioral theories, including outcome levels, elements of fidelity, and appraise different strategies that have both succeeded and failed in the use of games in healthcare education. More importantly, we explore the practical aspects of embedding data collection to prove the game’s impact to healthcare education and health.
E N D
Measuring healthcare outcomes using serious games, gamification, and virtual reality Todd P Chang, MD MAcM serious play conference 2018.07.11 (Manassas, VA)
Disclosures / Conflicts of Interest American Heart Association National Board of Medical Examiners Oculus from FaceBook • • • Independent Contractor with SBC Med Sim, LLC • 2
WHY MAKE HEALTHCARE GAMES AT ALL? 3
Objectives 1 2 3 Map out measurable Outcomes for both Mediating & Moderating games Define the 4 Outcome levels for measuring Effectiveness of a Game Discuss three case studies measuring effectiveness of Games on patient outcomes
HOW CAN GAMES INFLUENCE PATIENT OUTCOMES?
Landers et al. 2014 Sim & Gaming Mediating Moderating • Knowledge • Skill • Attitude 8
MEDIATING Going through the game improves KSA 9
1000 CUT JOURNEY Virtually experience life as a black child
Landers et al. 2014 Sim & Gaming Mediating Moderating • Knowledge • Skill • Attitude 12
The game or gamification elements improve engagement of an otherwise effective tool MODERATING
Landers et al. 2014 Sim & Gaming Mediating Moderating Going through the game improves KSA The game improves engagement of an effective tool 15
Outcome 00 Mood prior to VR Outcome 01 Use Frequency Outcome 02 Change in Mood 16
Outcome 04 Health Metric (Result) Outcome 03 Knowledge Application (Behavior) Outcome 00 Prior Outcomes Outcome 01 Use Scores Outcome 02 Knowledge Gain 17
Outcome 03 Health Metric (Result) Outcome 01 Use (Behavior) Outcome 02 Attitude 18
What if the game user is the healthcare provider, not the patient? 19
OUTCOME LEVELS Objective 2: Kirkpatrick’s 4 Levels of Evaluation
Kirkpatrick’s Levels of Evaluation Evaluation and Feedback Sullivan GM. J Med Educ. 2011;3(2):121-4.
Kirkpatrick’s Levels of Evaluation Evaluation and Feedback Reaction Sullivan GM. J Med Educ. 2011;3(2):121-4.
Kirkpatrick’s Levels of Evaluation Evaluation and Feedback Learning Reaction Sullivan GM. J Med Educ. 2011;3(2):121-4.
Kirkpatrick’s Levels of Evaluation Evaluation and Feedback Behavior Learning Reaction Sullivan GM. J Med Educ. 2011;3(2):121-4.
Kirkpatrick’s Levels of Evaluation Evaluation and Feedback Results Behavior Learning Reaction Sullivan GM. J Med Educ. 2011;3(2):121-4.
Barsuk et al. Crit Care Med 2009;37(10):2697-701. Barsuk et al. Acad Med 2010;85(10 Suppl):S9-12. Barsuk et al. MJ Qual Saf 2014;23(9):749-56. • • •
Outcome C Prior Knowledge Outcome A Use Scores Outcome B Knowledge Gain
Outcome C Prior Knowledge Outcome A Use Scores Outcome B Knowledge Gain Outcome 04 Health Metric (Result) Outcome 03 Prior Patient State (Result) Outcome 01 Provider Action (Behavior) Outcome 02 Patient State (Result)
CAN MY GAME PREDICT HOW MUCH IMPROVEMENT WILL OCCUR? A question of Validity
Discriminant Validity Testing Expected Low Performers Expected High Performers Uncovers any bias due to: • Differences in Video Game Experience • “Game Cheating” behavior • Lack of Functional Fidelity in a Game • A different target audience
Gerard et al. Validity Evidence for a Serious Game to Assess Performance on Critical Pediatric Emergency Medicine Scenarios. Simul Healthc 2018;13(3):168-80.
3 CASE STUDIE S
HOW TO MEASURE ‘MULTI-PATIENT CARE’? A discriminant validity study
Background • With higher patient volumes but fewer providers, multi-patient care is now common • Poor multi-patient care leads to increased errors • Multi-patient care is not routinely taught nor universally assessed
Purpose To validate a serious games version of a pediatric ED that measures multi-patient care based on experience: 1. Undergraduate students (jrs, srs) 2. Medical students (MS-II) 3. Resident physicians (PGY-1 through 3) 4. Attending / Fellow physicians (PGY-4+)
Acknowledgments to the Stemmler Fund & BreakAway Games, Ltd
Outcomes & Analyses Predictors Experience Multi-tasking ability Video Game frequency Game Outcomes % Sentinel Orders Time to 1stSentinel Order Time to 2ndSentinel Order Time to Discharge % Patients Seen # correct Diagnoses # Differential Diagnoses • • • • • • • • • • 45
Results Undergrads 11 MSIIs 22 Residents 15 Attgs 23 n 46
Results # Patients Seen 100 75 50 25 0 Undergrad MSII Residents Attgs 47
Results Time-to-Critical Action (game seconds) 750 500 250 0 Time to SO1 Time to SO2 Time to d/c Undergrad MSII Resident Attg 48
Results • MANCOVA analyses revealed a statistically significant difference in game performance after controlling for video game experience & multi- tasking ability. • The two outcomes that the game was able to distinguish expertise in were: – # sentinel orders – % correct diagnoses
The one-way MANCOVA showed that there was statistically significant difference (p=0.03) between the skill groups on the combined dependent variables after controlling for game play frequency, F(21, 153) = 0.700, p = .828, Wilks' Λ = .768, partial η2 = .084, and for MTAT placement score, F(21, 153) = 1.365, p = .144, Wilks' Λ = .610, partial η2 = .152. Follow up univariate one-way ANCOVAs were performed. A Bonferroni adjustment was made such that statistical significance was accepted at p < .008. There were statistically significant differences in adjusted means for total percentage of sentinel orders completed (F(3, 59) = 31.702, p < .0005, partial η2 = .617), and number of correct diagnoses made per patient seen(F(3, 59) = 21.441, p < .0005, partial η2 = .522). 50