1 / 17

Computation Approaches to Emotional Speech

Computation Approaches to Emotional Speech . Julia Hirschberg julia@cs.columbia.edu. Why Study Emotional Speech?. Recognition Anger/frustration in call centers Confidence/uncertainty in online tutoring systems “Hot spots” in meetings Generation TTS for Computer games IVR systems

connie
Download Presentation

Computation Approaches to Emotional Speech

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Computation Approaches to Emotional Speech Julia Hirschberg julia@cs.columbia.edu

  2. Why Study Emotional Speech? • Recognition • Anger/frustration in call centers • Confidence/uncertainty in online tutoring systems • “Hot spots” in meetings • Generation • TTS for • Computer games • IVR systems • Other applications: Speaker State • Deception, Charisma, Sleepiness, Interest… • The Love Detector (available for Skype )…

  3. Assessing Health-Related Conditions • Assessing intoxication levels (Levit et al ‘01) • Distinguishing between active and passive coping responses in patients with breastcancer (Zei Pollermann ’02) • Assessing schizophrenia (Bitouk et al ‘09) • Classifying degree of autistic behavior (Columbia) • Suicide notes

  4. Hard Questions in Emotion Recognition • How do we know what emotional speech is? • Acted speech vs. natural (hand labeled) corpora • What can we classify? • Distinguish among multiple ‘classic’ emotions • Distinguish • Valence: is it positive or negative? • Activation: how strongly is it felt? (sad/despair) • What features best predict emotions? • What techniques best to use in classification?

  5. Acted Speech: LDC Emotional Speech Corpus happy sad angry confident frustrated friendly interested anxious bored encouraging

  6. Neutral July 30 Yes Disappointed/tired No Amused/surprised No Annoyed Yes Late morning Frustrated Yes No No, I am … …no Manila... Is Natural Emotion Different? (thanks to Liz Shriberg)

  7. Major Problems for Classification:Different Valence/Different Activation

  8. But….Different Valence/ Same Activation

  9. Good Features Can be Hard to Find • Useful features: • Automatically extracted pitch, intensity, rate, VQ • Hand-labeled, automatically stylized pitchcontours • Context • Lexical information: Dictionary of Affect • But….individual and cultural differences • Algorithms for classification: • Machine learning (Decision trees, Support Vector Machines, Rule induction algorithms, HMMs,…)

  10. Results: Different Emotions, Different Success Rates

  11. Open Questions • New features and algorithms • New types of emotion/speaker state to identify • New ways of finding/collecting useful data • New applications of more-or-less successful emotion classification • Interspeech Paralinguistic Challenges

  12. This Class • Goals: • Learn what we know about: readings and discussion participation • Learn how to analyze speech, how to design a speech experiment, how to classify speaker states • Try to contribute something new: term project • Practice doing research • Syllabus: • http://www.cs.columbia.edu/~julia/courses/CS6998/syllabus11.htm

  13. Readings and Discussion • Weekly readings • Everyone prepares/hands in 3 discussion questions on each assigned paper or website • If you read an optional paper, submit questions on that as well if you want ‘credit’ • Everyone participates in class discussion • Each week one person leads discussion on one paper • Submit pdf in courseworks shared files

  14. Term Project • Everyone prepares a term project on a topic of their choice • You may work alone or in teams of 2 • Deliverables • Proposal • Interim progress report • Final report • Short presentation/demo

  15. Possible Topics • Collect audio from children of different ages winning and losing a game and see if adults can distinguish those who win (happy speech) from those who lose (sad speech). • Create hybrid speech stimuli from tokens uttered with different emotions (mixing pitch, loudness, duration, speaking rate,...) and see which features of emotional speech are most reliably associated with emotions. • Detect different emotions from Cantonese and Mandarin speakers and compare performance of an automatic program to performance of human judges. • Train Machine Learning algorithms on emotional speech corpora and see if you can improve over other approaches on the same corpora • Develop an email reader that detects emotion from text and uses the appropriate emotional TTS system to read it to the use

  16. Important Details • Read the academic integrity paragraph in the syllabus and understand it. • Do all the readings when they are due, turn in all discussion questions by noon on the day of class, come to every class

  17. Questions?

More Related