Augmented reality based affect adaptive neurocognitive rehabilitation

1. Augmented reality based affect adaptive neurocognitive rehabilitation Intermediate report of Msc. Thesis project A.S.Panic

2. Student Media & Knowledge Engineering (EEMCS) - Man Machine Interaction • 5 months project work at ETH Zurich • Financial support provided by • Imperial College, London, TU Delft, ETH Zurich, RWTH Aachen, ParisTech http://www.idealeague.org/

3. In this presentation Augmented reality based affect adaptive neurocognitive rehabilitation

4. Thesis position • Neurocognitive rehabilitation is possible to a higher extent than previously assumed • Therapeutic exercises can be made more motivating for the patient • In the near future more people will need cognitive (re)habilitation than have access to, or can be treated in the existing clinics • Human factors play a significant and undervalued role when designing technology Augmented reality based affect adaptive neurocognitive rehabilitation

5. Surveying the literature • Three questions drive the literature survey: • Which concepts are used to understand cognitive processes and neurocognitive rehabilitation? • How can neurocognitive rehabilitation be supported by virtual reality applications? • How can therapeutic games be created so that they promote a motivated learner? Augmented reality based affect adaptive neurocognitive rehabilitation

6. Main topics for literature survey Augmented reality based affect adaptive neurocognitive rehabilitation

7. Basic stages of cognition • Lack or loss of language severely impairs cognition • Sound and music can significantly influence emotion • Emotion significantly influences cognition Source: Groome & Dewart, 1999, An introduction to cognitive psychology: processes and disorders Augmented reality based affect adaptive neurocognitive rehabilitation

8. Emotion and cognition One example of cognitive influences of emotion: • In a study at Tufts University: • Test subjects were blindfolded and physically moved a random distance between two houses • The test subject were then asked to which of the two houses he or she was closer to The conclusion: the reported distance estimates were significantly different if the two houses were religious in nature (e.g. church, mosque) versus neutral Source: Qui Wang at the Psychology department of Tufts University Augmented reality based affect adaptive neurocognitive rehabilitation

9. Stages not easily summarizable • Stages of visual perception include processing horizontal lines, vertical lines, arcs, edge detection, shape detection, • Stages of auditory perception include pitch, timbre, rhythm, melody, localization, so… A multi-stage model of memory • In reality cognitive processes are not identifiable by independent and distinct stages, but composed of many (overlapping) substages • Cognition operates at the levels of basic skills, executive functions and general intelligence (fluid and crystallized) Source: Groome & Dewart, 1999, An introduction to cognitive psychology: processes and disorders Augmented reality based affect adaptive neurocognitive rehabilitation

10. Loss of cognitive skills • A loss of cognitive skills can be caused by: • Typical aging process • Neurodegenerative diseases (see picture) • Traumatic brain injury • Similar symptoms exist in young and old people, but with different names • Mostly pathological causes Source: Fisk et al, Designing for older adults Augmented reality based affect adaptive neurocognitive rehabilitation

11. Cognitive (behavioral) therapy • Cognitive therapy (CT) aims at rehabilitating any of the basic cognitive skills • Cognitive behavioral therapy (CBT) aims at rehabilitating one or more Activities of Daily Living (ADL) • Issues with CT and CBT: • Treatment or assessment can be dependant on administrator (therapist) • Lack of standardized stimuli or treatment content • Therapeutic exercises can be boring Source: Buschert, 2009, KognizionsbezogeneInterventionenbei Alzheimer-Krankheit Augmented reality based affect adaptive neurocognitive rehabilitation

12. Cognitive rehabilitation strategies Cognitive rehabilitation can aim to: • Reinforce, strengthen or reestablish previously learned skills or behavior • Establish patterns of compensatory cognitive activity to cope with impairment of cognitive systems • Establish environmental compensation that allow new patterns of activity • Enable the patient to adapt to their cognitive impairment in order to improve overall level of functioning and quality of life Source: Cicerone, 2000, 2005, Evidence based cognitive rehabilitation, recommendations for clinical practice Augmented reality based affect adaptive neurocognitive rehabilitation

13. Brain plasticity and cognitive reserve • Since the 1980s two (theoretical) constructs became adopted when explaining how a loss of cognitive skills can be rehabilitated: • Brain plasticity = the changing of neurons, the organization of their networks, and their function via new experiences. • Cognitive reserve = the brain’s resilience to neuro pathological damage Augmented reality based affect adaptive neurocognitive rehabilitation

14. Gaming stimulates brain plasticity • MRI assessments show that structural engagement with the Tetris game increases formation of ‘gray matter’ • Lack of conclusive evidence for other games, e.g. Dr Kawashima’s “Brain Training” for Nintendo DS. • However many of these games are based on standard instruments of cognitive assessment. Source: Haier et al, 2009, MRI Assessment of cortical thickness and functional activity changes in adolescent girls, following three months of practice on a visual-spatial task Augmented reality based affect adaptive neurocognitive rehabilitation

15. Cognitive reserve • Cognitive reserve is based on two mechanisms: • Neural reserve • Neural compensation • Evidence for cognitive reserve: Nuns have been diagnosed with Alzheimer’s Disease (neurodegenerative!), but without any noticeable loss of cognitive skills Sources: Stern, 2006, Cognitive reserve and Alzheimer’s disease Snowdon, 2004, Healthy aging and dementia: findings from the Nun study Augmented reality based affect adaptive neurocognitive rehabilitation

16. Case studies and explanations Brain plasticity, Discovery channel clip (5 min) This video clip, from the Discovery Channel and McGraw-Hill Higher Education, details how the brain can "repair" itself. It introduces a case study of young Jody Miller, who had half of her brain removed, in an attempt to control her epileptic seizures. http://www.youtube.com/watch?v=TSu9HGnlMV0 Re-wiring the brain, 2008 TED talk (20 min) Neuroscientist Michael Merzenich looks at one of the secrets of the brain's incredible power: its ability to actively re-wire itself. He's researching ways to harness the brain's plasticity to enhance our skills and recover lost function. http://www.ted.com/talks/michael_merzenich_on_the_elastic_brain.html Augmented reality based affect adaptive neurocognitive rehabilitation

17. Brain plasticity – a case study stroke of insight, 2008 TED talk (20 min) Brain researcher Jill Bolte Taylor studied her own stroke as it happened. She has become a spokesperson for stroke recovery and for the possibility of coming back from severe brain injury stronger than before. http://www.ted.com/talks/jill_bolte_taylor_s_powerful_stroke_of_insight.html Augmented reality based affect adaptive neurocognitive rehabilitation

18. Stimulating cognitive reserve • At any point in one’s lifetime, cognitive reserve results from a combination of exposures: • Socio-economic status (e.g. occupation) • Educational attainment • Leisure activities • Physical activity stimulates neurogenesis Source: Stern, 2009, Cognitive reserve Augmented reality based affect adaptive neurocognitive rehabilitation

19. The virtuality continuum • Allows the classification of applications that create an illusion of an alternate reality Reality Virtual reality Mixed Reality Augmented reality, more real than virtual (e.g. HMD) Augmented virtuality, more virtual than real Non or partially-immersive VR (e.g computer monitor) Source: Milgram & Kishino, 1994, A taxonomy of mixed reality visual displays Augmented reality based affect adaptive neurocognitive rehabilitation

20. Creating a fake reality • One to four simultaneous users • 3D motion picture, smell, stereo sound, vibrations, wind • The holy grail of virtual reality is to fool all the senses and create a ‘suspension of disbelief’ for the virtual environment Sensorama (1957) by the ‘godfather of virtual reality’ Source: Morten Heilig, 1962, Sensorama patent Augmented reality based affect adaptive neurocognitive rehabilitation

21. VR based cognitive rehabilitation • When using virtual reality for neurocognitive rehabilitation, what are the: • Strengths? • Weakness? • Opportunities? • Threats? • Assumptions: • Administration in home environment Source: Adapted from Rizzo & Kim, 2005, A SWOT analysis of the field of virtual reality rehabilitation and therapy Augmented reality based affect adaptive neurocognitive rehabilitation

22. SWOT of particular interest • Strength: • High fidelity training and assessment environments can be created • Economy of scale • Gaming factors can be used to increase patient motivation and adherence • Weakness: • Interface challenge: wires, displays and peripherals • Side effects of exposure to virtual environments may not be unavoidable • Engineering challenge: proper support for data mining, extraction & analysis (for therapists!) Source: Adapted from Rizzo & Kim, 2005, A SWOT analysis of the field of virtual reality rehabilitation and therapy Augmented reality based affect adaptive neurocognitive rehabilitation

23. SWOT of particular interest • Opportunities: • Emerging technologies: • Unobtrusive and natural interaction devices, wires • Wearable computing, sensor networks • Situated cognitive rehabilitation / telerehabilitation • Preventive training during preclinical stage increases cognitive reserve and resilience to pathology • Threats: • Privacy issues with telerehabilitation and medical data • Potential for lawsuits because of side-effects • Ethical challenges Source: Adapted from Rizzo & Kim, 2005, A SWOT analysis of the field of virtual reality rehabilitation and therapy Augmented reality based affect adaptive neurocognitive rehabilitation

24. Motivation in instructional games • The goal is to create a game cycle which: • fulfills learning objectives and • creates and sustains a motivated learner Source: Paras, 2005, Games, motivation and effective learning: an integrated model for educational game design Augmented reality based affect adaptive neurocognitive rehabilitation

25. Affective gaming • Affective computing: computers and programs which take the player’s affective (emotional) state into account • Affective games: affect-adaptive games adapt their content based on the player’s emotional state. E.g. adapt gameplay difficulty, offer more or less feedback and support on progress, offer tips and suggestions on how to continue, increase or decrease number of concurrent tasks etc. • High level design heuristics: “Emote me, assist me, challenge me” Sources: Picard, 2000, Affective Computing Gilleade et al, 2005, Affective videogames and modes of affective gaming Augmented reality based affect adaptive neurocognitive rehabilitation

26. Affective game engines • Core requirements: • A knowledge base which (hierarchically) describes emotions, used for emotion generation or expression. • An affective user model (AUM) of the player, which facilitates recognition of emotion and transitions between them. • Modeling of game characters’ emotions. • No computational model of cognitive processes required, a few simple rules can already have a measurable effect Source: Hudlicka, 2009, Affective game engines: motivation & requirements Augmented reality based affect adaptive neurocognitive rehabilitation

27. Thesis position (revisited) • Neurocognitive rehabilitation is possible to a higher extent than previously assumed • This may be explained by theories of brain plasticity and cognitive reserve • Therapeutic exercises can be made more motivating for the patient • This can be achieved by designing therapeutic games using criteria from affective computing and affective gaming Augmented reality based affect adaptive neurocognitive rehabilitation

28. Thesis position (revisited) • In the near future more people will need cognitive (re)habilitation than have access to, or can be treated in the existing clinics • Using telerehabilitation and commonly available (cheap) computing technology, virtual reality based therapy can be offered to a larger part of the patient population • Human factors play a significant and undervalued role when designing technology • Human Computer Interaction design should target natural and embodied interaction as much as possible. • Differences in technological aptitude should not prevent people from accessing healthcare Augmented reality based affect adaptive neurocognitive rehabilitation

29. Experimental research questions • Principal research questions: • Does the use of affect-adaptive training influence the rate of learning and motivation on a commonly used instrument of cognitive performance, when compared to a standard computer based training? • Does the use of human computer interaction devices which support more natural and embodied interaction lead to a higher rate of acceptance by the targeted population? Augmented reality based affect adaptive neurocognitive rehabilitation

30. The mental rotation task • Research vehicle: the mental rotation task (MRT) • Response time depends on: • Difference in rotation between object pair • Object complexity (but this diminishes with practice!) Are these two objects the same? Sources: Shepard & Metzler, 1972, Mental rotation of three dimensional objects, Bethell-Fox & Shepard, 1988, Mental Rotation: effects of stimulus complexity and familiarity Augmented reality based affect adaptive neurocognitive rehabilitation

31. Why use the mental rotation task? Rationale for using the MRT in this experiment: • As a widely used instrument for cognitive assessment it is particularly suitable for investigating gaming factors that influence motivation • As a visuo-spatial task it is particularly suitable for virtual and augmented reality • Stimulus complexity can be controlled precisely Sources: Rizzo et al, 1998, the virtual reality mental rotation spatial skills project Augmented reality based affect adaptive neurocognitive rehabilitation

32. Experiment setup IR LED glasses allow tracking of head position & gaze direction Training, Assessment Wii remote allows gestural interaction with virtual objects (pointing, dragging, selecting) and tactile and auditory output With headtracking support, a TV screen allows virtual objects to appear in front or behind of it Augmented reality based affect adaptive neurocognitive rehabilitation

33. Experiment protocol • Targeted population: • Elderly people with mild cognitive impairment • Vision corrected to standard, no history of postural instabilities, no motion sickness • In a clinical session the participants complete: • A training round (using 1 of 3 game modes) • A feedback round (motivation questionnaire) • An assessment round (of task performance) Augmented reality based affect adaptive neurocognitive rehabilitation

34. Project status • Dec 2009 – Mar 2010 • Project work @ ETH • Mar 2010 – Jul 2010 • Finish writing Msc. Thesis @ DUT • Thesis defense • Complete 2 remaining human factors in aerospace courses from my honors program Augmented reality based affect adaptive neurocognitive rehabilitation

35. Do you want to know more? Y. Stern,“Cognitive Reserve”, Neuropsychologia vol 47, 2009 http://www.cumc.columbia.edu/dept/sergievsky/cnd/pdfs/sdarticle-1.pdf Rizzo & G. Kim, “A SWOT analysis of the Field of Virtual Reality Rehabilitation and Therapy”, Presence vol 14, 2005 http://vrpsych.ict.usc.edu/PDF/1Rizzo_2005_A%20swot%20analysis%20of%20the%20field%20of%20VR.pdf E. Hudlicka, “Affective Game Engines: Motivation and Requirements”, ICDFG proceedings, 2009 http://www.imgd.wpi.edu/speakers/0910/Hudlicka_ICFDG_09_Dist.pdf