Automatic Classification of Married Couples’ Behavior using Audio Features

1. Automatic Classification of Married Couples’ Behavior using Audio Features Matthew Black, AthanasiosKatsamanis, Chi-Chun Lee, Adam C. Lammert, Brian R. Baucom, Andrew Christensen, Panayiotis G. Georgiou, and Shrikanth Narayanan September 29, 2010

2. (Common Slides) • Insert here … Automatic Classification of Married Couples' Behavior using Audio Features

3. Overview of Study Married couples discussing a problem in their relationship Data Coding Trained Human Evaluator Available Data (e.g., audio, video, text) Judgments (e.g., how much is one spouse blaming the other spouse?) Signal Processing (e.g., feature extraction) Examples Computational Modeling (e.g., machine learning) Feedback Automatic Classification of Married Couples' Behavior using Audio Features

4. Motivation • Psychology research depends on perceptual judgments • Interaction recorded for offline hand coding • Rely on a variety of established coding standards[Margolin et al. 1998] • Manual coding process expensive and time consuming • Creating coding manual • Training coders • Coder reliability • Technology can help code audio-visual data • Certain measurements are difficult for humans to make • Computers can extract these low-level descriptors (LLDs) [Schuller et al. 2007] • Consistent way to quantify human behavior from objective signals Automatic Classification of Married Couples' Behavior using Audio Features

5. Corpus • Real couples in 10-minute problem-solving dyadic interactions • Longitudinal study at UCLA and U. of Washington [Christensen et al. 2004] • 134 distressed couples received couples therapy for 1 year • 574 sessions (96 hours) • Split-screen video (704x480 pixels, 30 fps) • Single channel of far-field audio • Data originally only intended for manual coding • Recording conditions not ideal • Video angle, microphone placement, and background noise varied • Access to word transcriptions with speaker explicitly labeled • No indications of timing or speech overlap regions Automatic Classification of Married Couples' Behavior using Audio Features

6. Sample Transcript Husband: what did I tell you about you can spend uh everything that we uh earn Wife: then why did you ask Husband: and spend more and get us into debt Wife: yeah why did you ask see my question is Husband: mm hmmm Wife: if if you told me this and I agree I would keep the books and all my expenses and everything Automatic Classification of Married Couples' Behavior using Audio Features

7. Manual Coding • Each spouse evaluated by 3-4 trained coders • 33 session-level codes (all on 1 to 9 scale) • Utterance- and turn-level ratings were not obtained • Social Support Interaction Rating System • Couples Interaction Rating System • All evaluators underwent a training period to standardize the coding process • We analyzed 6 codes for this study • Level of acceptance (“acc”) • Level of blame (“bla”) • Global positive affect (“pos”) • Global negative affect (“neg”) • Level of sadness (“sad”) • Use of humor (“hum”) Automatic Classification of Married Couples' Behavior using Audio Features

8. Speaker Segmentation • Segment the sessions into meaningful regions • Exploit the known lexical content (transcriptions with speaker labels) • Recursive automatic speech-text alignment technique [Moreno 1998] • Session split into regions: wife/husband/unknown • Aligned >60% of sessions’ words for 293/574 sessions AM = Acoustic Model LM = Language Model Dict= Dictionary MFCC = Mel-Frequency Cepstral Coefficients ASR = Automatic Speech Recognition HYP = ASR Hypothesized Transcript Automatic Classification of Married Couples' Behavior using Audio Features

9. Goals • Goals • Separate extreme cases of session-level perceptual judgments using speaker-dependent features derived from (noisy) audio signal • Extraction and analysis of relevant acoustic (prosodic/spectral) features • Relevance • Automatic coding of real data using objective features • Extraction of high-level speaker information from complex interactions Automatic Classification of Married Couples' Behavior using Audio Features

10. Feature Extraction (1/3) • Explore the use of 10 acoustic low-level descriptors (LLDs) • Broadly useful in psychology and engineering research 1) Speaking rate • Extracted for each aligned word [words/sec, letters/sec] 2) Voice Activity Detection (VAD) [Ghosh et al. 2010] • Trained on 30-second clip from held-out session • Separated speech from non-speech regions • Extracted 2 session-level first-order Markov chain features Pr(xi = speech | xi-1 = speech) Pr(xi= non-speech | xi-1 = speech) • Extracted durations of each speech and non-speech segment to use as LLD for later feature extraction Automatic Classification of Married Couples' Behavior using Audio Features

11. Feature Extraction (2/3) • Extracted over each voiced region (every 10ms with 25ms window) 3) Pitch 4) Root-Mean-Square (RMS) Energy 5) Harmonics-to-Noise Ratio (HNR) 6) “Voice Quality” (zero-crossing rate of autocorrelation function) 7) 13 MFCCs 8) 26 magnitude of Mel-frequency bands (MFBs) 9) Magnitude of the spectral centroid 10) Magnitude of the spectral flux • LLDs 4-10 extracted with openSMILE[Eyben et al. 2009] • Pitch extracted with Praat[Boersma 2001] • Median filtered (N=5) and linearly interpolated • No interpolation across speaker-change points (using automatic alignment) Automatic Classification of Married Couples' Behavior using Audio Features

12. Pitch Example & Normalization • Normalized the pitch stream 2 ways: • Mean pitch value, µFo, computed across session using automatic alignment • Unknown regions treated as coming from one “unknown” speaker Automatic Classification of Married Couples' Behavior using Audio Features

13. Feature Extraction (3/3) • Each session split into 3 “domains” • Wife (when wife was speaker) • Husband (when husband was speaker) • Speaker-independent (full session) • Extracted 13 functionals across each domain for each LLD • Mean, standard deviation, skewness, kurtosis, range, minimum, minimum location, maximum, maximum location, lower quartile, median, upper quartile, interquartile range • Final set of 2007 features • To capture global acoustic properties of the spouses and interaction Automatic Classification of Married Couples' Behavior using Audio Features

14. Classification Experiment • Binary classification task • Only analyzed sessions that had mean scores evaluator scores in the top/bottom 20% of the code range • Goal: separate the 2 extremes automatically • Leave-one-couple-out cross-validation • Trained wife and husband models separately [Christensen 1990] • Error metric: % of misclassified sessions • Classifier: Fisher’s linear discriminant analysis (LDA) • Forward feature selection to choose which features to train the LDA • Empirically better than other common classifiers (SVM, logistic regression) Automatic Classification of Married Couples' Behavior using Audio Features

15. Classification Results (1/2) • Separated extreme behaviors better than chance (50% error) for 3 of the 6 codes • Acceptance, global positive affect, global negative affect • Global speaker-dependent cues captured evaluators’ perception well Automatic Classification of Married Couples' Behavior using Audio Features

16. Classification Results (2/2) • Other codes • Need to be modeled with more dynamic methods • Depend more crucially on non-acoustic cues • Inherently less separable Automatic Classification of Married Couples' Behavior using Audio Features

17. Results: Feature Selection • LDA classifier feature selection • Mean = 3.4 features, Std. Dev. = 0.81 features • Domain breakdown • Wifemodels: Independent=56%, Wife=32%, Husband=12% • Husbandmodels: Independent=41%, Husband=37%, Wife=22% • Feature breakdown • Pitch=30%, MFCC=26%, MFB=26%, VAD=12%, Speaking Rate=4%, Other=2% • Also explored using features from a single domain (make bar graphs!) As expected, performance decreases in mismatched conditions Good performance using speaker-independent features: session is inherently interactive (need to use more dynamic methods) Automatic Classification of Married Couples' Behavior using Audio Features

18. Conclusions Work represents initial analysis of a novel and challenging corpus consisting of real couples interacting about problems in their relationship Showed we could train binary classifiers using only audio features that separated spouses’ behavior significantly better than chance for 3 of the 6 codes we analyzed Provides a partial explanation of coders’ subjective judgments by objective acoustic signals Automatic Classification of Married Couples' Behavior using Audio Features

19. Future Work • More automation • Front-end: speaker segmentation using only audio signal • Multimodal fusion • Acoustic features + Lexical features • Saliency detection • Using supervised methods (manual coding at a finer temporal scale) • Using unsupervised “signal-driven” methods Automatic Classification of Married Couples' Behavior using Audio Features

20. References P. Boersma, “Praat, a system for doing phonetics by computer,” Glot International, vol. 5, no. 9/10, pp. 341–345, 2001. A. Christensen, D.C. Atkins, S. Berns, J. Wheeler, D. H. Baucom, and L.E. Simpson. “Traditional versus integrative behavioral couple therapy for significantly and chronically distressed married couples.” J. of Consulting and Clinical Psychology, 72:176-191, 2004. F. Eyben, M. W¨ollmer, and B. Schuller, “openEAR–Introducing the Munich open-source emotion and affect recognition toolkit,” in Proc. IEEE ACII, 2009. P. K. Ghosh, A. Tsiartas, and S. S. Narayanan, “Robust voice activity detection using long-term signal variability,” IEEE Trans. Audio, Speech, and Language Processing, 2010, accepted. C. Heavey, D. Gill, and A. Christensen. Couples interaction rating system 2 (CIRS2). University of California, Los Angeles, 2002. J. Jones and A. Christensen. Couples interaction study: Social support interaction rating system. University of California, Los Angeles, 1998. G. Margolin, P.H. Oliver, E.B. Gordis, H.G. O'Hearn, A.M. Medina, C.M. Ghosh, and L. Morland. “The nuts and bolts of behavioral observation of marital and family interaction.” Clinical Child and Family Psychology Review, 1(4):195-213, 1998. P.J. Moreno, C. Joerg, J.-M. van Thong, and O. Glickman. “A recursive algorithm for the forced alignment of very long audio segments.” In Proc. ICSLP, 1998. V. Rozgić, B. Xiao, A. Katsamanis, B. Baucom, P. G. Georgiou, and S. Narayanan, “A new multichannel multimodal dyadic interaction database,” in Proc. Interspeech, 2010. B. Schuller, A. Batliner, D. Seppi, S. Steidl, T. Vogt, J. Wagner, L. Devillers, L. Vidrascu, N. Amir, and L. Kessous. “The relevance of feature type for automatic classification of emotional user states: Low level descriptors and functionals.” In Proc. Interspeech, 2007. A. Vinciarelli, M. Pantic, and H. Bourlard. “Social signal processing: Survey of an emerging domain.” Image and Vision Computing, 27:1743-1759, 2009. Automatic Classification of Married Couples' Behavior using Audio Features

21. Thank you! Questions?