Biofeedback in Virtual Reality applications and Gaming

1. Biofeedback inVirtual Reality applications and Gaming Bonie Rosario, Jr. Sebastian Osorio Tom Iancovici University of Massachusetts Lowell Intro to Biosensors 16.441.201 Prof. Xingwei Wang

2. Outline • What is Biofeedback? • Electroencephalogram • The Galvanic Skin Response and Heart Rate Variability • Biosensor Research • Research findings • Our Conclusions

3. What is Biofeedback? • Biofeedback = the technique of using monitoring devices to furnish information regarding an automatic bodily function, such as heart rate or blood pressure, in an attempt to gain some voluntary control over these functions[1].

4. Electroencephalogram • EEG activity can be subdivided into various types of frequency rhythm bands. Research has indicated that different EEG frequency bands are associated with different mental states. The major five are alpha, theta, beta delta and gamma. • EEG signals tend to have amplitude in the range of 2V to 100 V The electroencephalography is defined as a graphic representation of the potential difference between two different cerebral locations plotted over time[3].

5. EEG Commercial Products NeuroSky MindWave There are several affordable commercial products EEG Systems. Two of the most popular and user-friendly are produced by NeuroSky and Emotiv. • Emotiv EPOC A recent study compared the quality score of an EEG System based on price, functionality and usability. In terms of usability, Emotiv’s EPOC scored the highest.

6. An example of a Two-channel EEG amplifier How EEG Signals are Measured: Hardware An example of an EEG device prototype

7. Emotive EPOC Hardware and Software • The Emotive EPOC EEG Headset samples the EEG signal at a rate of 120 Samples per Second (SPS) using an Analog-to-Digital Converter (ADC). It samples signals from 14 different electrodes and filters out artifacts, or noise, outside the bandwidth range of 0.2 – 45 Hz. It then is Emotiv EPOC headset Specification

8. How EEG Signals are Measured: Software • Expressive Suite • Cognitive Suite • The Emotiv EPOC Headset I. Expressive Suite: User facial recognition II. Affective Suite: Monitors player emotional states in real-time II. Cognitive Suite: Reads and interprets a player’s conscious thoughts and intent

9. How EEG Signals are Measured: Software SDK of the Research Edition Affective Suite

10. GSR and HRV Biosensor • The heart rate and the galvanic skin response are very accurate manifestations of human reaction such as boredom, stress, and excitement. They are also very easy to measure using non-intrusive techniques. http://www.healthy-heart-meditation.com/images/IOM-Technology.jpg http://www.hsphysicsteacher.com/electriccurres/13%20Electric%20Current%20&%20Resistance/13.06%20galvanic%20skin%20response.jpg

11. How GSR and HRV are Measured HRV= Heart Rate Variable GSR=Voltage/Current Also known as Instantaneous Skin Resistance

12. What About Current Input Devices? • Mouse and Keyboard are relatively easy to use and are recognizable by the public • “Typing” and “Clicking” seem to be the most efficient ways to interact with a computer.

13. Multi-Biosensor Gaming System • Research by the University of Saskatchewan, Canada • Combines • EMG (Electrical Activation of Muscle Tissue) • GSR (Galvanic Skin Response) • HRS (Heart Rate Variability) • Respiration • Temperature • Eye Movement / Gaze Source: Nacke, L.E., Kalyn, M., Lough, C., Mandryk, R.L. 2011. “Biofeedback Game Design: Using Direct and Indirect Physiological Control to Enhance Game Interaction,” CHI 2011, Vancouver, BC, Canada.

14. Purpose of Research • Evaluate how feasible it is to incorporate biofeedback peripherals with current computer gaming hardware and software • Determine which kinds of biosensors feel natural to what types of activities

15. Study • 10 participants who are not affluent with video games • Play a Two-Dimensional Shooting Game • Using only a traditional video game controller • Using a controller enhanced with biofeedback peripherals • Using a controller enhanced with biofeedback peripherals mapped to a different set of functions

16. Gameplay Increasing enemy sprite size Movement and flamethrower length

17. Gameplay Snowfall and final boss battle Medusa’s Gaze

18. Testing Different Biosensors • Mechanic Cond. 1 Cond. 2 • Target size RESP GSR • Speed/jump EKG EMG • Weather/boss TEMP EKG • Flamethrower GSR RESP • Avatar control Gamepad Gamepad • Medusa’s Gaze Gaze Gaze

19. General Results • 9 out of the 10 like the biosensors • “made for a very immersive game, out of what is basically just a very simple platform shooter.” • “the sensors added a new dimension to the game, and gave a greater sense of involvement.” • “variation, and more enjoyment while playing because there are always new skills to improve on.” • “sensors [made the] game complicated.”

20. More General Results • “I like the idea of using multiple physiological inputs. Distributing the functions around the body is intuitive in some cases.” • “[It] basically boils down to an "extended controller" where the buttons are not buttons but other actions, similar to waving a Wii remote.” [Preferred biosensor for each activity]

21. GSR Feedback • “I liked that it was always a challenge to control just with my thoughts […] and forced me to use a part of my brain I wouldn't normally use in a video game.” • “I disliked the fact that one of the only ways that I found I was able to use the GSR was by biting my lip which isn't actually all that fun after it starts hurting.” • -overreaction

22. EKG Feedback • I couldn't control as instantly as the others - the effect from it tended to last over longer periods of time.” • “better suited to changing the game context than what the character is doing.”

23. EMG Feedback • “It was fairly easy to use. It was effective and worked.” • “[…] having this sensor tied to jumping/speed felt natural” • - muscle strain during repeated use

24. Respiratory Feedback • “It was neat to see the immediate reaction from my body to the game.” • “[It] felt very natural, particularly when it was tied to target size in the game […]. It's one I felt I could control to a fine degree.” • Immediate Reaction

25. Temperature Feedback • “It was easy to use for short periods of time but hard to remember to breathe deep into the sensor […].” • “I like […] when it was tied to weather […], because it felt like a natural thing to do. […]” • “Breathing rapidly to increase temperature also brought my heart rate up.” • -intrusiveness

26. Gaze Feedback • “Now that was just cool […] I liked being able to roast one frozen combatant while immobilizing another.” • “I found it frustrating when you would look at a platform intending to jump on it and end up freezing it.” -Problems of overfocusing

27. Additional Feedback • “[The] muscle and breathing sensors were simple enough that they were practically like a new button on the controller for me, but very awesome ones since rather than tapping a button, it was an instinctive action.” • “The breathing sensor and GSR sensor felt like controllers though, because I was very aware they were attached to me. The EKG and EMG were completely unnoticeable and fun to use…”

28. Researchers’ Conclusions • Biofeedback devices made games more interactive and more fun • Preferred sensors were one that were easy to control and produces noticeable results in real-time • Sensors were appreciated when mapped to activities that felt natural

29. Our Findings • EEG allows direct control without physical contact • Assistive Technology • Accuracy Issues + set up • Cost • GSR and HRV reflect users’ state of mind • More intelligent gaming systems • Guides gamers to feel particular emotions • Personalizes each user’s gaming experience

30. Conclusions • Innovative • Not enough research • Not cost effective yet • Unlimited potential • Fun and “out of the box” applications of biomedical theory

31. Demo • Demo • Campus Catalyst Competition • Entrepreneurship Showcase • May 5, 2011 at Merrimack College

32. References • Nacke, L.E., Kalyn, M., Lough, C., Mandryk, R.L. 2011. “Biofeedback Game Design: Using Direct and Indirect Physiological Control to Enhance Game Interaction,” CHI 2011, Vancouver, BC, Canada. • [1]"Biofeedback - Definition of Biofeedback by the Free Online Dictionary, Thesaurus and Encyclopedia." Dictionary, Encyclopedia and Thesaurus - The Free Dictionary. Web. 26 Feb. 2011. <http://www.thefreedictionary.com/biofeedback>. • [2]Tarvainen, M.P.; Karjalainen, P.A.; Koistinen, A.S.; Valkonen-Korhonen, M.V.; , "Principal component analysis of galvanic skin responses," Engineering in Medicine and Biology Society, 2000. Proceedings of the 22nd Annual International Conference of the IEEE , vol.4, no., pp.3011-3014 vol.4, 2000doi: 10.1109/IEMBS.2000.901513URL: http://xplorebcpaz.ieee.org/stamp/stamp.jsp?tp=&arnumber=901513&isnumber=19495