Psychophysiology-Based Affective Communication for Implicit Human-Robot Interaction

1. Ph.D. Defense Psychophysiology-Based Affective Communication for Implicit Human-Robot Interaction Pramila Rani, 2005 October 24, 2005 Committee: Dr. Nilanjan Sarkar (Chair) Dr. Mitch Wilkes, Dr. Richard Shiavi, Dr. Eric Vanman, and Dr. Michael Goldfarb

2. Some Definitions • Human-Robot Interaction • The study of humans, robots and the ways in which they influence each other • Psychophysiology • Science of understanding the link between psychology and physiology • Affective Communication • Communication relating to, arising from, or influencing feelings or emotions Pramila Rani

3. Research Focus This dissertation involves developing an intuitive affect*-sensitive human-robot interaction framework where • robot interacts with a human based on his/her probable affectivestate • affective states are inferred from the human's physiological signals • robot adapts its behavior in response to the human's affective state *emotion Pramila Rani

4. Outline • Motivation • Research Hypotheses • Main Components • Results • Discussion • Conclusion Pramila Rani

5. Motivation • The Robot “Invasion” • There is a projected increase of 1,145% in the number of personal service robots in use within a year • According to World robotics 2004 report, at the end of 2003, about 610,000 autonomous vacuum cleaners and lawn-mowing robots were in operation • In 2004-2007, more than 4 million new units are forecasted to be added!!! • Need for Natural and Intuitive Human-Robot Communication • Unlike industrial robots, personal and professional service robots will need to communicate more naturally and spontaneously with people around • Robots will be expected to be understanding, emphatic and intelligent Pramila Rani

6. Motivation • Attempt to mimic Human-Human Interaction • More than 70% of communication is non-verbal or implicit • Emotions are a significant part of communication • 7% percent of the emotional meaning of a message is communicated verbally. About 38% by paralanguage and 55% via nonverbal channels [1] • Most Significant Channels of Implicit Communication in Humans • Facial Expressions • Vocal Intonation • Gestures and Postures • Physiology [1] Mehrabian, A. (1971). Silent Messages. Wadsworth, Belmont, California Pramila Rani

7. Motivation Giving Robots Emotional Intelligence • Robots should be capable of implicit communication with humans • They should detect human emotions • They should modify their behavior to adapt to human emotions Pramila Rani

8. Application Areas Some Potential Application Areas of Affect-Sensitive Robots Pramila Rani

9. Research Challenges • Human–Centric Technology • Affective Robots- Emotion Expression and Perception • Physiology-Based Affective Computing • Challenges of Affect Recognition • Robot Control Architecture • Robots and Real-Time Affective Feedback Pramila Rani

10. Human–Centric Technology • Technological advancement so far has been more machine-centric • Now, new areas of robot application are emerging (e.g., battlefield, space, personal assistance, search & rescue) • There is a need to synergistically combine various capabilities of robotic systems with human intelligence • Most robots lack implicit channel of communication with humans There is an evident need for technological innovation in HRI that permits implicit communication between humans and robots so that we can begin to build affective or emotionally-intelligent robots. Pramila Rani

11. Affective Robots • Two-fold capability of an affective robot • Perceive emotions in humans, • Express its own emotions in a manner understandable to humans. • There exist robots that can express their emotions using human-like facial expressions and affective speech • Need for real understanding of human emotions • detecting anxiety, frustration, engagement, boredom etc. • reacting to these emotions For intelligent and intuitive human-robot interaction, it is imperative that the robot should be capable of perceiving human psychological states and adapting its behavior appropriately to address such a perception. Pramila Rani

12. Physiology-Based Affective Computing • Affect Recognition via Facial Expressions and Vocal Intonation • Work under highly constrained conditions • Dependent on gender, age, culture • Under voluntary control, hence manipulable • Computationally expensive and not designed for real-time affect recognition. • Advantages of using Physiology for Affect Recognition • Largely involuntary • Reasonably independent of cultural, gender and age related biases. • Continuously available and are not dependent on overt emotion expression • Technological Advancement in Physiological Sensing • Smaller, noninvasive , better sensors • Wireless communication • High-speed signal processing and pattern recognition capabilities Given the strong relationship between physiology and affective states, and the continuous and involuntary nature of physiological phenomena, advanced signal processing and machine learning techniques can be effectively employed to determine an individual's underlying affective states in real-time. Pramila Rani

13. Physiology and Affect • Current Physiology-Based Affect-Recognition Systems • Vyzas et. al., Kim et. Al., Nasoz et.al, Hayakawa et. al. • Limitations of Current Systems • Distinguish between discrete affective states • Affect elicitation usually involves audio/visual stimuli, or in some cases deliberate emotion expression • Very few systems work online • No systematic investigation of the relationship between a comprehensive set of physiological signals, their features and the affective states It would be useful to develop an online affect-recognition system based on features derived from multiple physiological signals, that can detect arousal of specific emotions of individuals while they are engaged in real-life task experiments. Pramila Rani

14. Machine Learning • The data sets are extremely constrained: • Noisy • Small size • Missing predictor variables • High input dimensionality • Possible redundancy in the input domain • Machine learning techniques employed by other works • Fuzzy Logic, Neural Network, Hidden Markov Models, and Bayesian learning • Regression Tree based affect-recognition not been investigated till now It would be worthwhile to empirically study the classification performance, advantages and disadvantages of few key machine learning techniques when applied to the domain of affect recognition using physiological signals. Pramila Rani

15. Real-Time Affective Feedback Requirements of Robot Control Architecture • Support channels for Explicit and Implicit Communication • Interpret affective input in the task context • Adapt Robot functionality to accommodate the affective states of the human • Allows mixed-initiative interaction between the human and robot Till date there is no human-robot interaction system available in which real-time physiology-based feedback is utilized by a robot to interpret the underlying psychological state of the human and modify or adapt its (robot's) behavior as a result. Pramila Rani

16. Research Hypotheses • It is possible to detect distinct affective states and further differentiate within varying levels of each affective state using multiple indices derived from physiological signals in real-time • Such a channel of implicit-communication can be integrated within a machine's control architecture to make it capable of detecting human affective states and responding to them appropriately • Such systems are expected to improve human performance, while lowering the user's anxiety and increasing task challenge. Pramila Rani

17. Research Components • Theoretical • Psychological states relevant in implicit communication • Physiological signals to be monitored • Control Architecture to accommodate implicit communication • Task design for training (Phase I) and validation (Phase II) phases • Computational • Signal conditioning and processing • Machine learning for affect recognition • System Development • Phase I and Phase II • Experimental • Phase I & Phase II Pramila Rani

18. Psychological States Selected Anxiety, Engagement, Boredom, Frustration, and Anger • These psychological states play an important role in human-machine interaction. • The affective states identified above were mainly chosen from the domain of negative affective states since they can be more closely related to performance and mental health of humans while working with machines. • Discussion with Psychologists, review of research works done in psychophysiology and human factors, and preliminary piloting was instrumental in this selection. Pramila Rani

19. Physiological Signals • Impedance Cardiogram (ICG) • Electrocardiogram (PCG) • Pulseplethysmogram (PPG) • Phonocardiogram (PCG) • Electromyogram (EMG) • Corrugator Supercilii • Zygomaticus Major • Upper Trapezius • Peripheral Temperature • Electrodermal Activity (EDA) Pramila Rani

20. Impedance Cardiogram Q Point PEP B Point • Relationship with Affective States • Pre-Ejection Period (PEP) is most heavily influenced by sympathetic innervation of the heart. • Reduced PEP is a marker of negative affect states – specifically anxiety • Features Extracted • Mean PEP • Mean IBI ECG Signal dZ/dt, where Z = ICG Signal Pramila Rani

21. ECG and PPG BVP ECG PTT • Relationship with Affective States • ECG influenced by frustration, anger and anxiety • PPG modulated by anxiety, fear of harm • Negative affect dimension specifically associated with increased sympathetic arousal • Features extracted • Mean Interbeat Interval (IBI) • Std. of IBI • Sympathetic power • Parasympathetic power • Ratio of Sympathetic to Parasympathetic power • Mean amp. of the peak values of the BVP signal • Standard deviation (Std.) of the peak values of the BVP signal • Mean Pulse Transit Time ECG Signal Pulse Transit Time Pramila Rani

22. Phonocardiogram (Heart Sound) Features extracted • Mean of the 3rd,4th, and 5th level coefficients of the Daubechies wavelet transform of heart sound signal • Standard deviation of the 3rd,4th, and 5th level coefficients of the Daubechies wavelet transform of heart sound signal http://www.biologymad.com/HeartExercise/HeartE3.gif Pramila Rani

23. Electromyogram • Relationship with Affective States • Facial displays (frowns, grimaces, smiles etc.) of affective reactions are obvious overt behaviors associated with expression of emotions • The Corrugator Supercilii muscles (responsible for lowering and contraction of the brows) considered as a measure of distress • EMG activity in the Zygomaticus Major occurs when the cheek is drawn back or tightened. This activity has been found to increase with expression of pleasure. • Features Extracted • Mean of EMG activity • Std. of EMG activity • Slope. of EMG activity • Mean Interbeat Interval of blink activity • Mean amplitude of blink activity • Mean and Median frequency of Corrugator, Zygomaticus and Trapezius EMG Signal classes.midlandstech.com/ Bio112/muscles%20fac... Pramila Rani

24. Electrodermal Activity • Relationship with Affective States • Tonic SC can be a useful index of a process related to energy mobilization or regulation • SC response is produced by social stimulation that invokes stress, tension, anxiety or cognitive reactions. • Significantly smaller values of SC response associated with neutral states than with sadness, anger, fear, disgust, and amusement • Features Extracted: • Mean tonic activity level • Slope of tonic activity • Mean amplitude of skin conductance response (phasic activity) • Maximum amplitude of skin conductance response • Rate of phasic activity Typical Skin Conductance Response Skin Conductance Signal Pramila Rani

25. Peripheral Temperature • Relationship with Affective States • Peripheral temperature is an indirect index of peripheral vasoconstriction. • Skin temperature can vary by 1-2 degrees Fahrenheit depending upon the emotional state of a person • In the flight/fight stress response peripheral nervous system shunts the blood away from one’s extremities and into the brain, heart and lungs, to aid in optimum performance in order to eliminate the acute stress. • Features Extracted • Mean • Slope, and • Standard deviation of temperature recording Pramila Rani

26. Mixed-Initiative Interaction Pramila Rani

27. Task Design (Phase I) Anagram • The anagram solving task has been previously employed to explore relationships between both electrodermal and cardiovascular activity with mental anxiety. • In this task, emotional responses were manipulated by presenting the participant with anagrams of varying difficulty levels, as established through pilot work. • Affective states such as engagement, boredom, anger, frustration and anxiety were induced by manipulating the difficulty of anagrams • All these conditions were well tested during the task design and development stage and piloting. Pramila Rani

28. Task Design (Phase I) Pong • Pong game has been used in the past by researchers to study anxiety, performance, and gender differences • Various parameters of the game were manipulated to elicit the required affective responses. These included: • ball speed and size, • paddle speed and size, • sluggish or over-responsive keyboard, • random keyboard response. • The relative difficulties of various trial configurations were established through pilot work. Pramila Rani

29. Task Design (Phase II) • Pong • Interactive Pong • Real-time feedback regarding player anxiety provided to machine • Performance-based game adaptation • Anxiety-based game adaptation Pramila Rani

30. Task Design (Phase II) • Robot Basketball Game • A basketball hoop attached to a robotic manipulator • The difficulty of the task varied by controlling parameters such as robot arm speed and direction of motion. • Performance Based Game Adaptation • Anxiety-Based Game Adaptation Pramila Rani

31. Computational • Signal conditioning and processing • Algorithms for artifact-rejection, adaptive thresholding, signal conditioning and feature-extraction for various signals • Wavelet transform, Fourier transform and statistical analysis and were extensively used in order to perform signal processing • Machine learning for affect recognition • A systematic comparison of the strengths and weaknesses of four machine learning methods - K-Nearest Neighbor, Regression Tree, Bayesian Network and Support Vector Machine* was performed * SVM analysis was done by Mr. Changchun Liu Pramila Rani

32. Signal Processing • Adaptive Thresholding • An adaptive or continuously changing threshold value was used to determine whether candidate for peaks qualified to be valid peaks. • The need for this arose from the fact that for signals such as PPG (pulseplethysmogram), the average peak amplitude shows a large deviation over a given period of time. Pramila Rani

33. Adaptive Thresholding • A moving window was used to determine the threshold • The peaks in a given window were weighed so that the most recent peaks had higher weight values than the older peaks. • Where C = Scaling factor, wi = weight for peak (k-i), and Ak-I is the amplitude of peak (k-i). The values of wi were such that smaller the value of i, greater the value of wi. The values of wi, k, and C were determined by Monte Carlo Simulations. Pramila Rani

34. Fourier Transform • Powerful signal analysis technique to study the frequency components of the physiological signals • Time series waveforms do not capture frequency-related variabilities easily • Frequency domain analysis has proven valuable in linking physiological abnormalities and variability to specific frequency bands. • Fourier Transform of the interbeat interval (IBI) derived from ECG Pramila Rani

35. Wavelet Transform • Signals such as PCG are complex and highly non-stationary • FFT analysis has limited analysis capabilities for such signals • Wavelets can be a very powerful tool for performing time-frequency analysis of non-stationary signals such as heart sounds. • It allows simultaneous localization in time and frequency domain • It has inbuilt noise filtering Pramila Rani

36. Wavelet Transform (cont…) Pramila Rani

37. Filtering & Artifact Rejection • Filtering of the signal is required to focus on a narrow band of electrical energy that is of interest • It removes noise and artifact such as that commonly found at 50 or 60 Hz (emitted into the recording • environment by devices such as florescent lights , computer power supplies) • Other elements that need to be filtered out are the artifacts caused by limb motions • Band pass, low pass and high pass were employed depending upon the frequency of interest EMG Signal Filtered in Different Ways http://www.thoughttechnology.com/pdf/MAR656-00%20Tech%20Note%20024.pdf Pramila Rani

38. Machine learning for affect recognition • KNN • description • BNT • description • RT • description • SVM • description Pramila Rani

39. System Development PhaseI System Set-up Pramila Rani

40. System Development Sensors Task in Progress Room 1- Experimental Set-up Sensors Wearable Sensors Baselining Pramila Rani

41. System Development Phase II System Set-up for Pong Pramila Rani

42. System Development Phase II System Set-up for Robot Basketball Pramila Rani

43. Experimental • Model building and Verification Model Verification and Analysis Phase I Experiments Data Analysis for Model Building Phase II Experiments Models for Affective States (I will make a better figure) Pramila Rani

44. Sliding Window Technique • For New Participants Pramila Rani

45. Experimental • Phase I • Fifteen participants took part in a 2-month study during which each person completed six sessions (three sessions of playing Pong and three sessions of solving anagrams) • Phase II • In the verification experiments for Pong nine participants who also took part in Phase I experiments volunteered • Fifteen participants took part in the robot-based basketball game. None of them had participated in Phase I and four of them were new. Pramila Rani

46. Pong • Experiment Procedure Pramila Rani

47. Robot Basketball • Experiment Procedure Pramila Rani

48. Results • Relationship between physiological signals and affective states • Accuracy of Regression Tree based affect recognition • Comparison between Regression Tree, KNN, Bayesian Networks and Support Vector Machines • Results of Pong game and robot-based basketball game with real-time affective feedback Pramila Rani

49. Results Relationship Between Physiological Signals and Affective States • High Correlations found between physiological signals and affective states • Extent of correlation was different for different affective states Pramila Rani

50. Results Physiological Signals and Affective States • Person Stereotypy • The highly correlated physiological indices vary from individual to individual Pramila Rani