Exploring the role of affective feedback in Interactive IR

1. Exploring the role of affective feedback in Interactive IR Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

2. Questions? • What is the role of emotions in the information seeking process? • Do they correspond to any form of relevance feedback? • How can we effectively employ them in information retrieval scenarios? Affective Feedback

3. Relevance Feedback • Relevance assessments can contribute in the disambiguation of the user’s information need • This is achieved through the application of various feedback techniques  Affective Feedback

4. Explicit Relevance Feedback • Feedback which is obtained through the explicit and intended indication of documents as relevant (positive feedback) or irrelevant (negative feedback)  Affective Feedback

5. Explicit Relevance Feedback  Affective Feedback

6. Implicit Relevance Feedback • Implicit Feedback: a passive form of feedback, which is applied in an intelligent and unobtrusive manner • Can be used to individualize a system’s responses or develop user models (UM)  Affective Feedback

7. Implicit Relevance Feedback  Affective Feedback

8. Common aspects • Both categories of feedback techniques determine relevance by considering what occurs on the cognitive and situational level of interaction • However, they do not account for the affective dimension of the conversational interplay between the user and the system  Affective Feedback

9. Affective Computing • Affective computing aims in the development of more natural and flexible systems. • Human-machine interactive systems capable of sensing affect states (stress, inattention, etc) and capable of adapting and responding appropriately to these are likely to be perceived as more natural, efficient and trustworthy (Pantic, Sebe, Cohn, Huang, 2005). • Can we build a multimodal retrieval system that exploits more than one modality? Affective Feedback

10. Affective Feedback • Can affective feedback be of any value to IR? • Likely yes, since it is considered a qualitatively rich source of human affect indications, which can be potentially exploited to enhance the information retrieval process. • Affective feedback can be defined as • the sum of all the human affective expression/indications, which are communicated implicitly to (or identified by) a computer system and can be therefore used to facilitate a more natural, effective and robust interaction. Affective Feedback

11. Affective Interaction • Users interact with intentions, motivations and feelings besides real-life problems and information objects… • Intentions, motivations and emotions are all critical aspects of cognition and decision-making  Affective Feedback

12. Affective Interaction • Information systems equipped with the ability to detect and respond to user emotions could potentially: • Improve the naturalness of human-computer interaction • Progressively optimize their retrieval strategy • Offer a more personalized experience • Determine more accurately the relevancy of an information object  Affective Feedback

13. Affective Interaction • What are the possible reasons of emotion? • System? • Search strategy & search results? • Content design and aesthetics? • Other  Affective Feedback

14. Emotion in IR – Some Conclusions • The co-occurrence of emotions during an information seeking process, among other physiological, psychological and cognitive processes • Patterns of emotional variance, which reveal a progressive transition from positive to negative valence as the degree of task difficulty increases • Depending on their frequency of occurrence the value of the conveyed affective information may potentially vary?  Affective Feedback

15. Test Collection • For the indexing we used TREC 9 (2000) Web Track • 1.69 million document subset of the VLC2 collection • We retained the original content of the TREC topics, but presented them using the structural framework of the simulated information need situations • Introduce a layer of realism, while preserving well-defined relevance criteria  Affective Feedback

16. Search Tasks  Affective Feedback

17. Facial Expression Analysis • Facial expression analysis was applied on the video recordings of each session • For each key-frame of the video eMotion calculated the probability of the detected facial expression (assuming there was one) corresponding to any of the seven detectable emotion categories (Neutral, Happiness, Surprise, Anger, Disgust, Fear, Sadness)  Affective Feedback

18. eMotion • eMotion is an automatic facial expression recognition system • Developed by Nicu Sebe’s group in Amsterdam/Trento • It follows a model-based approach, in which a 3-dimensional wireframe model of the face is constructed, once certain facial landmark features are detected • Head motion of facial deformation can then tracked and measured in terms of motion-units (MU’s), which are eventually classified into one (or more) of the seven detectable emotion categories  Affective Feedback

19. eMotion  Affective Feedback

20. Classifier • eMotion has been trained using a generic static classifier • The classifier has been developed from a subset of the Cohn-Kanade database • It performs reasonably well across all individuals, independently of ethnicity-specific features  Affective Feedback

21. Tools & Modalities • Tools: • eMotion (Facial Expression Recognition System) + 2d camera • Pasion (Facial Expression Recognition System) + 3d camera • Polar RS800 Heart Rate Monitor • BodyMediaSenseWear Pro3 Armband • Modalities: • Facial Expressions (emotion categories)1 • Facial Expressions (motion units)1 • HR3 • GSR4 • Heat Flux4 • Skin Temperature4 • Acceleration4

22. Facial Expressions- Affective Feedback

23. Facial Expressions- Affective Feedback

24. Biometrics Affective Feedback

25. Findings • users' affective responses will vary across the relevance of perused information items. • the results also indicate that prediction of topical relevance is possible and • to a certain extent models can benefit from taking into account user affective behaviour. Affective Feedback

26. Open Questions • How to select different modalities? • Large-scale body movements; Hand-gesture recognition; Gaze-detection; Speech/voice analysis • How to integrate multiple modalities? • Modelling challenge? • How to develop a practical system that respond to users emotional behaviour? Affective Feedback