Mixed Feelings About Using Phoneme-Level Models in Emotion Recognition

1. Hannes Pirker Austrian Research Institute for Artificial Intelligence (OFAI) Vienna Mixed Feelings About Using Phoneme-Level Models in Emotion Recognition Posterpresented atACII-2007, September16-19,Lisbon, Portugal

2. This study deals with the application of MFCC based models for both the recognition of emotional speech and the recognition of emotions in speech. More specifically it investigates the performance of phone-level models. First, results from performing forced alignment for the phonetic segmentation on GEMEP, a novel multimodal corpus of acted emotional utterances are presented, then the newly acquired segmentations are used for experiments with emotion recognition comparing phone-level models with sentence-level models.. Abstract 1 Motivation The Geneva Multimodal Emotion Portrayals corpus (GEMEP) [1] is a novel corpus of highly controlled and uniform content, containing 2 pseudo-linguistic utterances produced with 18 different emotions and varying levels of intensity. This uniform content provides a promising basis for investigating both the acoustic correlates of emotions as well as timing-issues in the synchronization between speech, facial expressions and body movements. In order to provide a sound basis for the investigation of fine-grained temporal issues, phonetic segmentation at the level of individual speech sounds was to be performed. Apart from this practical goal it also was to be investigated on how well standard segmentation techniques, i.e. Hidden Markov Models (HMM) with Mel Frequency Cepstral Coefficients (MFCC) [4] would cope with the amount of variability of manner and voice quality typically found in emotional speech. The third goal was to shed some light on the conflicting requirements on MFCCs. They are used in speech recognition for discriminating speech sounds and for their robustness against variation in intonation and voice quality. But they are also became popular in emotion recognition [3], though then they typically should be indifferent to the underlying speech sounds and sensitive to voice quality. As the identity of the speech sounds is a major influencing factor on MFCCs, we are comparing emotion classifiers that rely on phoneme-level models with sentence-level models.

3. 2 Database Description • The Geneva Multimodal Emotion Portrayals corpus (GEMEP) consists of audio and video recordings of 10 speakers (professional French speaking actors), 5 of which are female. Verbal content is restricted to only 2 different pseudo-linguistic sentences: • (type1): Ne kal ibam soud mol'en! • (type2): Koun s'e mina lod belam? • These were uttered with 18 different emotions which were chosen for an extensive coverage of both activation (high-low) and evaluation dimensions (positive-negative) • Utterances were also produced in less intense, more intense and 'masked' manner (i.e. unsuccessfully ‘hide’ the actual emotion from the audience). In average utterances were repeated 8 times per condition. • This results in 3815 sentences: 2739 of type1 and 1076 of type2. Emotion categories in the GEMEP corpus. Subset of “selected 6” set in bold. 3. Phonetic Segmentation 3.2 Workflow Starting with 50 manually segmented sentences for training an initial set of HMM, several cycles of training-alignment-manual correction and re-training with the increased set were performed. By now 1313 manually validated sentences are available which are currently split into 892 training- and 421 test-samples. To make use of the constrained content of the corpus HMM where trained for each individual sound, i.e. 4 different models for /a/ where trained. Apart from unconstrained training (Global-models) specific models for male and female speakers (per Gender), for each actor (per Speaker) and each emotion (per Emotion) where constructed and evaluated.

4. 3.1. Technical Procedure • The original audio data was downsampled to 22.05 kHz and high-pass filtered at 55 Hz. MFCC where calculated with a frame size of 30 ms and a window shift of 2.5 ms (i.e. resulting in 400 frames per second). 12 MFCC, energy, delta and acceleration were included resulting in a 39 dimensional feature vector. • For the phoneme-level models Hidden Markov Models with 3 states and 5 Gaussian mixtures in each state were employed. Baum-Welch algorithm was used for training the HMMs and Viterbi decoding was performed in order to retrieve the segment boundaries. Extraction of MFCC features, training and application of HMMs was performed with the respective tools provided by HTK [4]. • 3.2. Evaluation • The absolute position error at the initial phone boundaries was used for evaluation. Quantiles and error thresholds provide a meaningful measure: approx. 85% of segments are located with less an error less than 20ms and should not require manual correction. Below the performance of two differently sized training sets (N=503 vs. N=892) and when using gender-, speaker- and emotion-specific phoneme-models is illustrated. • The results indicate that the performance of the aligner seems to already level out, i.e. for type1 the increase in the size of the training data from 395 to 685 does not show much of an effect anymore while the less frequent type 2 still benefits from the increase in training-size. Also further fractionating the training set by using speaker-specific models etc. does not decrease the performance dramatically anymore.

5. This is in line with the subjective impression that ‘normal’ cases were remarkably well handled by models trained on small sets but persistently problematic classes exist, e.g. flustered speech with its weakly articulated formant structures. Soft speech, even of very low intensity, as well as loud or even shouted speech are less critical. The most problematic cases are intermingled laughter, hesitations etc. which pose severe problems to any procedure based on forced alignment. • 4 Emotion Classification • 4.1 Technical Procedure • In order to test the capabilities of phone-level-MFCC modeling for emotion classification virtually the same methods as used for phonetic segmentation were re-applied. For phone-level modelling emotion-specific 3 state left-to-right HMMs with 5 Gaussian Mixtures were trained. Alternatively two sentence-level HMM typologies were tested: an 'elongated' version of the phone-level model with 22 states/5 mixtures and several 1-state HMMs, which are equivalent to a Gaussian Mixture Model. The “recognition grammar” of the aligner was adapted as shown in Fig.2. A majority vote was performed on the outcome of the Viterbi decoding. • For the experiments with emotion recognition a test-set of 875 samples was reserved, leaving a maximum of 2940 samples for training (i.e. 77%/23%). Currently only sentence-level models could make use of the whole training-set, as they do not require any pre-segmented data. Phone-based models where for now trained on a smaller set of 892 though, as it was necessary to use manually segmented data. Fig.1: 85% Error quantile [ms] per segment for the two sentences in the corpus (N train=892)

6. 4.2 Evaluation • The table below provides a comparison between the phone-based model and a sentence-based model on the same train- and test-set as it was also used in the evaluation of the phonetic-alignment task above. Again the smaller number of training-samples for type2 results in a significant drop in performance. 18 categories is an untypical high numbee, for better comparability all experiments where also performed with a sub-set of 6 emotions (anger, joy, fear, sadness, pleasure, interest) For sentence-level HMMs different typologies where compared. For the full set of 2940 a single state model with an exceedingly high number of Gaussian Mixtures (512) performed best. This model was compared with phone-level modelling. Fig.2 Recognition grammar for phone-level and sentence-level modelling.

8. The recognition quality is likely to still benefit from increased training size as the results from the experiments with sentence-level models indicate: 5. Conclusion We presented a study on phone-based MFCC-models for emotion recognition which originated in the ‘practical task’ of phonetic segmentation of the GEMEP corpus. The results for the automatic segmentation probably have reached a certain ceiling, but provide a valid basis for manual correction of further samples. Results for emotion classification are still in flux, i.e., results with differently sized training-sets still show significant volatility. Ultimate conclusions on the relationship of phone-based vs. global models are difficult to obtain. Because the segmental content in GEMEP is so restricted, phoneme-based models lose their expected implicit advantages in this context. On the other hand errors in the automatic alignment have a strong influence on the classification, e.g. surprisingly the sound /s/ provided the best classification results of all phonemes, which probably is due to the fact that fricatives are the most suitable sounds for the aligner. The study on emotion recognition was not at all aimed at coming up with ‘impressive’ recognition rates, which could be easily boosted by using e.g. 10fold-cross-validation, a-priori probabilities, more equally balanced training sets etc., but to test the relative performance of different MFCC-based models.

9. Acknowledgements I am very indebted to Klaus Scherer and his group in Geneva for designing, creating and sharing the GEMEP corpus and especially to Tanja Baenziger for long standing and ongoing interaction and support. This work has been funded by the EU Network of Excellence HUMAINE (IST 507422) and by the Austrian Funds for Research and Technology Promotion for Industry (FFF 808818/2970 KA/SA). Financial support for OFAI is provided by the Austrian Federal Ministry of Science and Research and by the Federal Ministry of Transport, Innovation and Technology. • References • Baenziger T., Pirker H., Scherer K.: GEMEP - GEneva Multimodal Emotion Portrayals: A corpus for the study of multimodal emotional expressions, in Devillers L. et al. (eds.), Proceedings of LREC'06 Workshop on Corpora for Research on Emotion and Affect, May 23, Genoa, Italy, pp.15-19, 2006. • Lee C.M., Yildirim S., Bulut M., Kazemzadeh A., Busso C., Deng Z., Lee S., Narayanan S.: Emotion Recognition based on Phonem Classes, Proceedings of ICSLP 04, Jeju, Korea, 2004. • Schuller B., Rigoll G.: Timing Levels in Segment-Based Speech Emotion Recognition, in Proceedings of INTERSPEECH 2006 - ICSLP, 17-21 September, Pittsburgh, PA, USA, pp.1818-21, 2006. • Young S., Evermann G., Kershaw D., Moore G., Odell J., Ollason D., Povey D., Valtchev V., Woodland P.: The HTK Book (version 3.4), Cambridge University Engineering Department, Cambridge UK, 2006.