Embracing The Future Of Serious Gaming and Immersive Technologies In Medical Education - Bill Kapralos

1. Embracing The Future Of Immersive Technologies In Medical Education Bill Kapralos (PhD) University of Ontario Institute of Technology, Oshawa, Ontario, Canada.

2. Overview: Before We Start Immersive Technologies Immersive Technologies: Examples Moving Forward Before We Start Fear of Technology (1): Not a New Phenomenon

3. (Socrates, 470-399 BC) → “This discovery of yours will create forgetfulness in the learners’ souls, because they will not use their memories; they will trust to the external written characters and not remember of themselves.” Conrad Gessner (1516 – 1565) → information overload Must regulate the trade so public won’t suffer “confusing and abundance of books.” Fear of Technology (2): Not a New Phenomenon (cont.) with harmful

4. More recently (very recently!!) CNN → “Email hurts IQ more than pot” Telegraph article → “Twitter and Facebook could harm moral values” and “Facebook and MySpace generation 'cannot form relationships’” Daily Mail → “How using Facebook could raise your cancer.” Something to Consider (1): risk of

5. “Anyone who tries to make a distinction between education and entertainment doesn't know the first thing about either” Marshall McLuhan (1911 - 1980)

6. Immersive Technologies Simulation (1): Thousands of Years Current Emphasis (e.g., U.S. Military) Immersive technologies (virtual reality, augmented reality, and

8. What is Immersive Technology (1): Blurring the Line Technology that blurs the line between the physical world and digital or simulated world, thereby creating a sense of Augmented reality (AR) and virtual reality (VR) Coupled with serious games and virtual sims very engaging learning applications

9. What is Immersive Technology (2): Single person could perceive prerecorded experience via visual, sound, smell, vibration, and wind Why All the Hype Recently? Increase in computational power and decrease in size of electronic components has led to decreasing cost and rising availability of consumer-level immersive tech Helped advance recent adoption of virtual simulation hand and arm tracking technologies using controllers such Leap Motion hand sensor, Microsoft Kinect, and Thalmic Labs

10. What is Immersive Technology (3): Myo are allowing for development of novel interaction methods and techniques. Why All the Hype Recently? (cont.) Such devices provide more natural and immersive interactions → help overcome the limitations associated with traditional keyboard and mouse-based humancomputer interactions where tasks are performed very differently to a real-life situation

11. What is Immersive Technology (4): Providing designers/developers of virtual sims great freedom to develop highly immersive applications VR industry → $1.37 billion (USD) in 2015 to $33.90 billion (USD) by 2022 (Markets and Markets 2016) Goldman Sachs (2016) → AR and VR to be $95 billion industry by 2025 (Hall and Takabashi 2017).

12. What is Immersive Technology (5): What is Driving the Advances? changing Has been suggested that VR/AR will “drive a revolution in role under the following health applications Education and training Surgical planning Telemonitoring Patient experience Treatment and therapies AR-enhanced surgery and patient care. https://1189 medtechboston.medstro.com/blog/2016/05/24/16045/

13. Why Bother (1): Why Use Immersive Technologies (AR/VR)? Cost effective alternative → sim labs are expensive! Offers trainees opportunity to safely and cost-efficiently train until they reach a specific competency level Convenient → mobile VR… Team training → users can be located anywhere Laparoscopic surgeons err less by playing video games

14. (NY study) → hand-eye coordination (more later) Potential Applications: Cognitive → decision making … Technical skill development ? Current Buzz (1): Virtual and Augmented Reality It is all about virtual and augmented reality (VR/AR) Computing speed has increased tremendously and

15. Immersive Technologies: Examples VR and Cardiology Education (1): VR Headset to See Inside the Infant Heart

16. Stanford University, School of Medicine "The heart is a complicated three-dimensional organ, and it's really hard to describe what's going on inside of it — especially when something is going wrong . . . Virtual reality eliminates a lot of that complexity by letting people go inside the heart and see what's happening themselves —it's worth way more than a thousand words.“ Dr. David M. Axelrod, clinical Assistant Professor of Pediatric Cardiology, Stanford Uni. Room Scale VR Medical Education (1): Anesthesia Training

17. Serious game for epidural procedure training Virtual Patients (VPs) (1): Medical Education Environment of Future Low risk educational tool

18. “Virtual patients allow students to learn without putting real patients at risk… No actual patients are harmed in the process of learning from virtual patients.” Norm Berman, MD, Professor of pediatrics at the Geisel School of Medicine at Dartmouth. Framework to facilitate creation of VPs and dialogue scenarios to teach about verbal/nonverbal comm. Clinicians can create complex dialogue scenarios and link it directly to a VP → lip-sync, facial expressions More on VPs in next presentation… Haptics (1): But is it All About Viewing?

19. Big component of immersive technology is interaction Traditional input/interaction devices are not ideal -machine touch interactions High quality haptic devices capable of simulating the sense of touch with high fidelity are cost prohibitive Consumer-level haptic devices have recently become Novint Falcon Geomagic Sculpt Haptics (2):

20. Perceptual-Based Rendering Can sound “trick” the perceptual system into believing haptic fidelity provided by low-end haptic devices is better than it Conducted preliminary studies that examined this Electroencephalogram (EEG) monitoring with aim of correlating brain signals to haptic-sound interaction

21. Designing Immersive Experiences

22. Overview (1):

23. Not a Trivial Task!

24. Overview (2): The Development Team Immersive sims/serious game development assumes an interdisciplinary team Medical professionals → content experts Educational experts → instructional designers Artists “Techies” → computer scientists, game developers Can lead to problems Content experts may not be aware of technical limits

25. Overview (3): Game developers may not understand importance of subtle details Combining Instruction and Game Design Requires a synergy between instructional design and game design Seemingly opposed approaches that have radically different histories Educational aspect must be primary consideration The “book of content” already exists before the designer starts → the game must be about the “book” Two “prime directives” to follow simultaneously

26. Overview (4): 1.To make the game/application engaging 2.To follow the “book” → to be accurate Combining Instruction and Game Design (cont.) Bumpy past → according to one game designer, “the instructional designers brought into the creation of a learning game because they supposedly know how to get people to learn, typically suck the fun out of any game they get their hands on” Many examples of “bad serious games” Edutainment era of the late 1970s and 1980s

27. Overview (5): Typically, a bad serious game is due to a lack of proper design and often times, lacking appropriate instructional design → US Army study The Development Team (cont.) Very important to the design and development process for the game developers to experience first-hand “a day in the life” of the practitioner Agile Instructional Design Basic idea is that it is emergent, adaptive, iterative, and model- based → in the sense of creating prototypes rather than in the sense of following prescriptive models

28. Overview (6): Agile (rapid) design → iterative and incremental development where functional, even if incomplete version is created as soon as possible

29. Moving Forward Show Me the Proof (1): Greater Evidence of Effectiveness Greater studies are required that methodically verify the effectiveness of virtual simulations and serious gaming Some proof does exist but it is scattered and sporadic fMRI studies have validated effectiveness of VR and games in treatment of phobias and in distracting patients in process of burn treatment, chemotherapy Seymour et al. compared performance of surgery residents who trained with VR t better performance for residents who trained with

30. VR during laparoscopic cholecystectomy procedure Dig Rush → “therapeutic video game to treat Amblyopia” by prescription

31. Something to Consider (1): How Can Virtual Sims be Combined with Other Ed. Techniques to Enhance Learning? Open area of research → many questions remain to be answered here! Current Generation of Learners Challenging the current (outdated) teaching methods Prefer “learning by doing” and require interaction Spend considerable amount of time playing games motivated by surpassing their peers and being at top of their peer group throughout most of their student life”

32. Something to Consider (2): It’s Not Only About the Technology Need to think about how we use the technology Technology isn’t the answer to everything → it has its limitations and there are many things we can’t do Proper design required → majority of such applications are lacking instructional design and design in general Must carefully consider team and resources to create effective product that makes good use of technology Most students lack computing power to run latest VR applications → computationally expensive Many unanswered questions, so ongoing research is needed to inform what we do → many opportunities

33. Something to Consider (3): How Much Realism/Fidelity is Needed ? (cont.) Given that perfect multi-sensory fidelity is impossible to achieve, with current technology How much realism is needed to maximize learning? Unclear if high realism is needed for either enjoyment or to maximize learning & when some cues are too realistic

34. Something to Consider (4): → “Uncanny valley” effect

35. The Future (1): Technology Will Continue to Improve Imagine the opportunities in five years, let alone 10, 15 Microsoft HoloLens 2 will include AI coprocessor for implementing deep neural networks (DNNs) → algorithms for pattern recognition and machine learning (possibly key to solving some of the biggest challenges facing AR today) Electronic skin (e-skin) → soft, bendable, and wearable tech that allows user to manipulate objects that exist only in virtual world

36. The Future (2): Technology Will Continue to Improve (cont.) Interfaces and interaction methods are becoming more natural → optimized for humans rather than computers Advancements in gesture tracking, motion tracking, eye tracking, and other technologies → laying groundwork for natural interaction methods that will be essential for the success of augmented reality Gesture and motion tracking Eye tracking Voice Facial tracking Brain control

37. The Future (3): Technology Will Continue to Improve (cont.) IBM Watson → Q & A computer system capable of answering questions posed in natural language IBM & Unity partner to bring the power of AI to developers with IBM Watson Unity SDK (Feb 2018) Allow developers to easily integrate Watson cloud services into Unity applications → configure projects to understand speech, talk with users, and understand the intent of a user in natural language Easy for developers to take advantage of modern AI techniques through set of cloud-based services

38. The Future (4): Coupling Simulation with Makerspaces

40. Think of the opportunities! Boston Children’s Hospital Simulator Program’s Engineering Workshop (Makerspace) 3D printed surgical drill attached to consumer-level haptic device with simple electronic components to simulate surgical drilling “Food for Thought” (1):

41. Chocolate Covered Brocolli & Brussel Sprouts the outside it looks nice Not just about using the latest and greatest technology → proper design and testing is a must!

42. And Finally… Greater Work Remains so This is Definitely not the End… One Final Quote (1)

43. “We don’t stop playing because we grow old; we grow old because we stop playing” George Bernard Shaw (1856 – 1950))