Audio-visual speaker verification using continuous fused HMMs

1. Audio-visual speaker verification using continuous fused HMMs David Dean*, Sridha Sridharan*, and Tim Wark*† Presented by David Dean Slides will be available at http://www.davidbdean.com/category/publications

2. Why audio-visual speaker recognition Bimodal recognition exploits the synergy between acoustic speech and visual speech, particularly under adverse conditions. It is motivated by the need—in many potential applications of speech-based recognition—for robustness to speech variability, high recognition accuracy, and protection against impersonation. (Chibelushi, Deravi and Mason 2002)

3. A1 A1 V1 V1 A2 A2 V2 V2 A3 A3 V3 V3 A4 A4 V4 V4 Visual Speaker Models Fusion Speaker Decision Acoustic Speaker Models Speaker Decision Speaker Models Early and late fusion • Most early approaches to audio-visual speaker recognition (AVSPR) used either early or late fusion (feature or output) • Problems • Output fusion cannot model temporal dependencies • Feature fusion suffers from problems with noise Early Fusion Late Fusion

4. Middle fusion models can accept two streams of input and the combination is done within the classifier Most middle fusion is performed using coupled HMMs (shown here) Can be difficult to train Dependencies between hidden states are not strong (Brand 1999) Middle fusion - coupled HMMs

5. Pan et al. (2004) used probabilistic models to investigate the optimal multi-stream HMM design Maximise mutual information in audio and video They found that linking the observations of one modality to the hidden states of the other was more optimal than linking just the hidden states (i.e. Coupled HMM) Middle fusion – fused HMMs Acoustic Biased FHMM

6. Choosing the dominant modility • The fused HMM designed results in two designs, acoustic, or video biased • The choice of the dominant modality (the one biased towards) should be based upon which individual HMM can more reliably estimate the hidden state sequence for a particular application • Generally audio • Alternatively, both versions can be used concurrently and decision fused (as in Pan et al. 2004)

7. The original Fused HMM implementation treated the secondary domain as discrete (Pan et al. 2004) This caused problems with within-speaker variation Work fine on single session (CUAVE – Dean et al. 2006) Fail on multi-session (XM2VTS) Continuous FHMMs model both modalities with GMMs Discrete FHMM Continuous FHMM Continuous fused HMMs

8. Training FHMMs • Both biased FHMM (if needed) are trained independently • Train the dominant (audio for acoustic-biased, video for video-biased) HMM independently upon the training observation sequences for that modality • The best hidden state sequence of the trained HMM is found for each training observation using the Viterbi process • Model the relationship between the dominant hidden state sequence and the training observation sequences for the subordinate modality • i.e. model the probability of getting certain subordinate observation whilst within a particular dominant hidden state

9. Decoding FHMMs • The dominant FHMM can be viewed as a special type of HMM that outputs observations in two streams • This does not affect the decoding lattice, and the Viterbi algorithm can be used to decode

10. Experimental setup HMM/GMM Output Fusion Acoustic HMM/GMM Output Fusion Speaker Score Acoustic Feature Extraction Visual HMM/GMM Visual Feature Extraction Acoustic-Biased FHMM Acoustic Biased FHMM Speaker Score Video-Biased FHMM Visual Biased FHMM Speaker Score Manual Lip Tracking

11. Training and testing datasets • Training and testing configuration was based on the XM2VTSDB protocol (Messer et al. 1999) • 12 configurations were generated based on the single XM2VTSDB configuration • For each configuration • 400 client tests • 8000 imposter tests

12. Feature extraction • Audio • MFCC – 12 + 1 energy, + deltas and accelerations = 43 features • Video • Lip ROI manually tracked every 50 frames • 120x80 pixels • Grayscale • Down-sampled to 24x16 • DCT – 20 coefficients + deltas and accelerations = 60 features

13. Output fusion training • Two classifier types for each modality • Gaussian mixture models (GMMs) • Trained over entire sequences • Hidden Markov models (HMMs) • Trained for each word • Speaker models adapted from background models using maximum a posterior (MAP) adaption (Lee & Gauvin 1999) • Topology of HMMs and GMMs determined from testing evaluation partition in first configuration

14. Fused HMM performance is compared to output fusion of normal HMMs and GMMs in each modality Audio HMM + Video HMM Audio HMM + Video GMM Audio GMM + Video HMM Audio GMM + Video GMM Evaluation session used to estimate each modalities output score distribution to normalise scores within each modality Background model score subtracted from speaker scores to normalise for environment and length Output Fusion Normalisation + Normalisation Output fusion testing

15. HMM-based models takes advantage of temporal information to improve performance over GMM-based models in both modalities Audio GMM is near HMM, but with large number of Gaussians Video GMM does not improve with more Gaussians Output fusion results

16. Advantage of video HMM does not carry over to output fusion Little difference between video HMM and GMM in output fusion Output fusion performance affected mostly by choice of audio classifier Output fusion doesn’t take advantage of video temporal information Output fusion results Audio GMM Audio HMM

17. Fused HMM training • Both acoustic- and visual-biased FHMMs are examined • Individual audio and video HMMs used as basis for FHMMs • Secondary models adapted from individual speaker’s GMMs for each state of the underlying HMM • Background FHMM was formed similarly

18. Fused HMM testing • Subordinate observations are up/down sampled to the same rate as the dominant HMM • Evaluation session used to estimate each modalities frame-score distribution to normalise scores within each modality • Similar to output-fusion, but on a frame-by-frame basis rather than using final output score • As well as using subordinate models adapted to the states of the dominant HMM, testing is performed with • Word subordinate models • (same secondary model for entire word) • Global subordinate models • (same secondary for all words) • Finally, background FHMM score is subtracted to normalise for environment and length

19. Comparison with output-fusion • If the same subordinate model is used for each dominant state, the FHMM model can be viewed as functionally equivalent to HMM-GMM output fusion • Although in practice this is not the case due to resampling of the subordinate observations and where modality-normalisation occurs • Word and State subordinate models can also be viewed as functionally equivalent to HMM-GMM output fusion • Choose the subordinate GMM based on the dominant state for each frame • Provided that the FHMM design doesn’t affect the best path through the lattice

20. There is some benefit in using state-based FHMM models for audio State or word-based FHMM models are better than global for most of the plot Acoustic-biased FHMM results Best Performing Output Fusion

21. No benefit in using word or state-based FHMM models for video Therefore, no use in using FHMM models at all Video-biased FHMM results Best Performing Output Fusion

22. Acoustic vs. video-biased FHMM • The acoustic-biased FHMM shows that the audio can be used to segment the video into visually-similar sequences • However, the video-biased FHMM cannot use video to segment the audio into acoustically-similar sequences • Whilst the performance increase is small, it appears that the acoustic FHMM is benefiting from a temporal relationship between the acoustic states and the visual observations

23. Conclusion • Video HMM improves performance over video GMM, but not when used not in output fusion • Output fusion performance based mainly on acoustic classifier chosen • Audio-biased continuous FHMMs can take advantage of the temporal relationship between audio states and video observations • However, the video-biased continuous FHMM performance appears to show no corresponding relationship between video states and audio observations

24. Continuing/Future Research • Secondary GMMs are recognising a large amount of static video information • Skin or lip colour, facial hair, etc. • This information has no temporal relationship with the audio states, and may be swamping the more dynamic information available in facial movements • A more efficient structure may be realised by using more dynamic video features (mean-removed DCT, contour-based or optical flow) and output fusion with a face GMM • This would take advantage of the temporal audio-visual relationship, in addition to static face-recognition

25. Speech Recognition with Fused HMMs • FHMMs improve single-modal speech processing in two ways: • 2nd modality improves scores within states • 2nd modality improves state sequence • Text-dependent speaker recognition only benefits from the first improvement • State sequences is fixed • However, speech recognition can take advantage of both improvements

26. Using first XM2VTS configuration (XM2VTSDB) Speaker-independent, continuous-speech, digit recognition PLP-based Audio Features, Hierarchical LDA-based (of mean-removed DCT) video features We believe this is comparable performance to coupled and asynchronous HMMs But simpler to train and decode Speech Recognition with Fused HMMs

27. FHMMs, synchronous HMMs and feature fusion • The FHMM structure can be implemented as a multi-modal synchronous HMM, and therefore with minor simplification as a feature-fusion HMM • The difference is in how the structure is trained • In synchronous HMMs and feature-fusion, both modalities are used to train the HMMs • FHMMs can be viewed as adapting a multi-modal synchronous HMM from the dominant single-modal HMM • If the same number of Gaussians are used for both modalities, a FHMM can be implemented within a single-modal HMM decoder • Decoding is exactly the same as with feature-fusion

28. References • Brand, M. (1999), A bayesian computer vision system for modeling human interactions, in `ICVS'99', Gran Canaria, Spain. • Chibelushi, C., Deravi, F. & Mason, J. (2002), `A review of speech-based bimodal recognition', Multimedia, IEEE Transactions on 4(1), 2337. • Dean, D., Wark, T. & Sridharan, S. (2006), An examination of audio-visual fused HMMs for speaker recognition, in `MMUA 2006', Toulouse, France. • Lee, C.-H. & Gauvain, J.-L. (1993), Speaker adaptation based on MAP estimation of HMM parameters, in `Acoustics, Speech, and Signal Processing, 1993. ICASSP-93., 1993 IEEE International Conference on', Vol. 2, pp. 558561 vol.2. • Luettin, J. & Maitre, G. (1998), Evaluation protocol for the extended M2VTS database (XM2VTSDB), Technical report, IDIAP. • Messer, K., Matas, J., Kittler, J., Luettin, J. & Maitre, G. (1999), XM2VTSDB: The extended M2VTS database, in `Audio and Video-based Biometric Person Authentication (AVBPA '99), Second International Conference on', Washington D.C., pp. 7277. • Pan, H., Levinson, S., Huang, T. & Liang, Z.-P. (2004), `A fused hidden markov model with application to bimodal speech processing', IEEE Transactions on Signal Processing 52(3), 573-581.

29. Questions?