Introduction Our goal

1. Automated Analysis of Interactional Synchrony using Robust Facial Tracking and Expression Recognition Xiang Yu1, Shaoting Zhang1, Yang Yu1,Norah Dunbar2, Matthew Jensen2, Judee K. Burgoon3, DimitrisN. Metaxas1 1CBIM, Rutgers Univ., NJ 2Oklahoma University, Ok 3Univ. of Arizona, AZ

2. IntroductionOur goal • Predict whether an interactantis truthful or deceptive by analyzing interactional synchrony • Propose a computational framework to do so

3. IntroductionWhat is interactional synchrony? • Interpersonal communication is contingent upon some form of mutual adaptation such as: • Accommodation • Interpersonal coordination • Matching • Mirroring • Compensation • Divergence • Complementarity

4. IntroductionWhy investigating interactional synchrony? • Practitioners have suggested using interactional synchrony for detecting deception: • with terrorists (Turvey, 2008) • in FBI interviews (Navarro, 2003) • and in police investigations (Kassin et al., 2007) • Assumption: interviews with deceivers are less synchronous than interviews with truth tellers. • However, few systematic studies of coordination, synchrony or reciprocity have examined the effects of synchrony on deception.

5. IntroductionRelevant techniques • Face Tracking - Active Shape Model (Cootes et al. CVIU 1995) - Active Appearance Model (Cootes et al. ECCV 1998) - Constrained Local Model (Cristinacce and Cootes, BMVC 2006) • Gesture and Expression recognition - Head nodding, shaking, HMM(Kapoor and Picard PUI 2001) - Gesture recognition, CRF (Morency et al. CVPR 2007) - Image based expression recognition (Pantic et al. PAMI 2000) - Video based (Black and Yacoob, IJCV1997; Cohen et al. CVIU 2003)

6. Proposed MethodSystem framework

7. Module 1: Robust Facial Tracking • Robust facial tracking is the foundation of interactional synchrony analysis • Challenges: • Partial occlusion • Multiple poses • Poor lighting conditions

8. Active Shape Models • Active Shape Model Shape vector: Shape matrix: PCA: Covariance matrix: Shape representation:

9. Active Shape Models • Using KLT tracker to estimate shape locally. • Optimize shape , not only similar to , but also follows the shape distribution: • Alternately optimize and , until they converge.

10. Limitations • Partial occlusion: • Bayesian inference [Y. Zhou CVPR’03] • Pictorial structures [P. Felzenszwalb IJCV’05] • Sparse outliers [F. Yang FG’11] • Multiple poses: • Mixture of Gaussian distribution [T. Cootes IVC’99] • Kernel PCA [S. Romdhani BMVC’07] • Hierarchical multi-state [Y. Tong PR’07]

11. Face Tracking: Handle Occlusions • We explicitly model the large errors as a sparse vector : sparsity number of

12. Face Tracking: Handle Multi-pose • Instead of a low dimensional subspace, we model the shape as a sparse linear combination of training shapes: ... ≈

13. Face Tracking: Handle Multi-pose • Instead of a low dimensional subspace, we model the shape as a sparse linear combination of training shapes: sparsity number of sparsity number of

14. Synthetic data • Shape optimization from side pose with outliers. Blue line: detection results. Red line: fitting result. E-ASM Sparse shape registration (Handle gross error) Our method (Handle gross error and multi-pose)

15. Quantitative Comparison • The point error by pixel

16. Occlusion by Hat Extended ASM Sparse Shape Registration Our method

17. Occlusion by Scarf Extended ASM Sparse Shape Registration Our method

18. Multiple Pose Sparse shape Registration Our Method

19. Multiple Pose with Occlusions Sparse shape Registration Our Method

20. Face Tracking Results • Tracking result from CMC dataset

21. Module 2 and 3: Expression Detection and Head Pose Demo of the successful detection of events (both are smiling and with the same head pose)

22. Module 2: Expression Recognition Use the relative intensity order of facial expressions to learn a ranking model (RankBoost) for recog. and intensity estimation 6 universal facial expressions (i.e., fear, anger, sadness, disgust, happiness, surprise) Trained ranking model gives a score for the prob. of smiling in real time.

23. Expression Recognition(1) • Facial feature representation.

24. Expression Recognition(2) • Ordinal pair-wise data organization.

25. Expression Recognition(3) • Ranking Model. - sparse Rankboost (weak classifier) lost function: - Adaboost strong classifier

26. Smile Detection Illustration of synchrony for a smiling event (both the subject and the interviewer are smiling at about the same time instant)

27. Smile Synchrony Left: smiling scores curves for both the subject (in blue) and interviewer (in red). Here we see that there is synchrony. Right: Snapshots from the actual tracked footage illustrating smiling synchrony.

28. Module 3: Head Pose Module: Nodding A 3D head nodding sequence. The green lines show head pitch angle. The black lines show head yaw angle. Head pose pitch curve along time axis. The pattern of the blue plot is characteristic of head nodding

29. Head Pose Module: Shaking A 3D head shaking sequence. The green lines show head pitch angle. The black lines show head yaw angle. head pose yaw curve. The pattern of the red plot is characteristic of a head shake.

30. Head Pitch Synchrony (nodding) head pitch curves for the subject (in blue) and the interviewer (in red). The patterns are characteristic of head nodding. Here we see that there is synchrony (both plots show nodding pattern).

31. Head Yaw Dissynchrony (shaking) head yaw curves for the subject (in blue) and the interviewer (in red). The distinct pattern of the red plot is characteristic of a head shake. This head shaking event is NOT in synchrony.

32. Module 4: Synchrony Feature ExtractionCross correlation • Generate higher level synchrony feature from Lower level features • Correlation based strategy

33. Synchrony Feature Extractionhit-miss rate • Synchrony definition. • If event A is detected for subject X at time T and for subject Y at time T ± w  we say we have synchrony • Some attributes can be extracted from detected Synchrony: • 1. Smile_Hit_Subject_Rate_S(x): Rate of smiling synchrony (led by subject and followed by interviewer) in segment (x) • 2. Smile_Hit_Interviewer_Rate_S(x): Rate of smiling synchrony (led by interviewer and followed by subject) Subject Interviewer Smile Smile_Hit_Interviewer Smile_Hit_Subject Yes Subject Interviewer Smile Yes No Time No Time

34. Synchrony Feature Extractionhit-miss rate Subject Interviewer Smile Yes No Time Smile_Miss_Subject Smile_Miss_Interviewer Subject Interviewer Smile Yes No Time • Some attributes can be extracted from detected Synchrony: • 3. Smile_Miss_Subject_Rate_S(x): The rate of no smiling synchrony (led by subject and not followed by interviewer) in segment (x) • 4.Smile_Miss_Interviewer_Rate_S(x): The rate of no smiling synchrony (led by subject and not followed by interviewer) in segment (x)

35. Module 5: Feature Selection and Classification • Feature Selection, Genetic Algorithm. • Classification Two-class problem: Truthful vs. Deceptive Three-class problem: Truthful, Sanctioned and Unsanctioned cheating

36. Experimental Databases • Computer-Mediated Communication Dataset (CMC) • Face to Face Communication Dataset (FtF)

37. Evaluation of Synchrony feature

38. Evaluation of Two-class Classification • Confusion Matrices • Performance

39. Evaluation of Three-class Classification • Confusion Matrices • Performance

40. Conclusions • We investigated how the degree of synchrony effects the result of deception detection. • Automatic methods provide an important way to evaluate synchrony other than manual coding • Some observations: • Different lower level features contribute differently to the deception detection. • Modalities have subtle influence in detecting deception. (CMC vs. FtF)

41. Thanks!