1 / 40

Audio-Visual Speech and Speaker Recognition

Audio-Visual Speech and Speaker Recognition. Gérard Chollet, Guido Aversano, Hervé Bredin, Fabian Brugger, Maurice Charbit, Jerôme Darbon, Walid Karam, Chafic Mokbel, Santa Rossi, Eduardo Sanchez, Marc Sigelle, Georges Yazbek, Leila Zouari. Talking Faces.

johnathan
Download Presentation

Audio-Visual Speech and Speaker Recognition

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Audio-Visual Speech and Speaker Recognition Gérard Chollet, Guido Aversano, Hervé Bredin, Fabian Brugger, Maurice Charbit, Jerôme Darbon, Walid Karam, Chafic Mokbel, Santa Rossi, Eduardo Sanchez, Marc Sigelle, Georges Yazbek, Leila Zouari

  2. Talking Faces • Recognition of face features (lips, jaws, eyebrows, gaze, eye-blinkings,...) in synchrony with speech, • Tracking of lip movements, • Recognition of visemes, • Lip reading : how well do hard-of-hearing people perform ?

  3. F. J. Huang and T. Chen, "Real-Time Lip-Synch Face Animation driven by human voice", IEEE Workshop on Multimedia Signal Processing, Los Angeles, California, Dec 1998 Audio-visual recognition of spectrally reduced speech Frédéric Berthommier

  4. SpeechReading A human listener can use visual cues, such as lip and tongue movements, to enhance the level of speech understanding, especially in a noisy environment. The process of combining the audio modality and the visual modality is referred to as speechreading, or lipreading. There are many applications in which it is desired to recognize speech under extremely adverse acoustic environments. Detecting a person's speech from a distance or through a glass window, understanding a person speaking among a very noisy crowd of people, and monitoring a speech over TV broadcast when the audio link is weak or corrupted, are some examples.

  5. 2001: a Space Odyssee

  6. Audio-Visual Speech Recognition(Ref ?)

  7. Audio-Visual Speech Recognition(Ref ?)

  8. Audio-Visual Speech Recognition(Ref ?)

  9. Audio-Visual Speech Recognition(Ref ?)

  10. Coupled HMM

  11. OpenCV Open source code for AVCSR can be downloaded from http://sourceforge.net/projects/opencvlibrary/ **.

  12. Publications 1 • - Ara V Nefian, Lu Hong Liang, Xiao Xing Liu, Xiaobo Pi and • Kevin Murphy, "Dynamic Bayesian networks for audio-visual • speech recognition", EURASIP, Journal of Applied Signal • Processing , vol. 2002, no 11, p. 1274-1288, 2002. • Xiao Xing Liu, Yibao Zhao, Xiaobo Pi, Lu Hong Liang and • Ara V Nefian, "Audio-visual continuous speech recognition • using a coupled hidden Markov model", IEEE International • Conference on Spoken Language Processing , p. 213-216, • September 2002. • Lu Hong Liang, Xiao Xing Liu, Yibao Zhao, Xiaobo Pi and • Ara V Nefian, "Speaker independent audio-visual continuous • speech recognition", IEEE International Conference on • Multimedia and Expo , vol.2, p. 25-28, August 2002.

  13. Publications 2 • Ara V Nefian, Lu Hong Liang, Xiao Xing Liu, Xiaobo Pi, • Crusoe Mao and Kevin Murphy, "A coupled HMM for • audio-visual speech recognition", International Conference on • Acoustics Speech and Signal Processing , vol II, pp 2013-2016, • Orlando, Florida, May 2002 . • - Gerasimos Potamianos, Chalapathy Neti, Gridharan Iyengar, • Andrew W. Senior and Ashish Verma • A cascade visual front end for speaker independent • automatic speechreading • International Journal of Speech Technology, Special Issue on • Multimedia, 4, 193-208, 2001

  14. Biblio Adjoudani, A. and Benoit, C. (1996) . On the integration of auditory and visual parameters in an HMM-based ASR. In Stork, D.G. and Hennecke, M.E. (Eds.), Speechreading by Humans and Machines. Berlin, Germany: Springer, pp. 461-471. Bregler, C. and Konig, Y. (1994) . `Eigenlips' for robust speech recognition. Proceedings International Conference on Acoustics, Speech, and Signal Processing (ICASSP)'94, Adelaide, Australia, pp. 669-672.

  15. Biblio Brooke, N.M. (1996) . Talking heads and speech recognizers that can see: The computer processing of visual speech signals. In Stork, D.G. and Hennecke, M.E. (Eds.), Speechreading by Humans and Machines. Berlin, Germany: Springer, pp. 351-371. Chen, T. (2001) . Audiovisual speech processing. Lip reading and lip synchronization. IEEE Signal Processing Magazine, 18(1):9-21.

  16. Dupont, S. and Luettin, J. (2000) . Audio-visual speech modeling for continuous speech recognition. IEEE Transactions on Multimedia, 2(3):141-151. Gray, M.S., Movellan, J.R., and Sejnowski, T.J. (1997) . Dynamic features for visual speech-reading: A systematic comparison. In Mozer, M.C., Jordan, M.I., and Petsche, T. (Eds.), Advances in Neural Information Processing Systems 9. Cambridge, MA: MIT Press, pp. 751-757.

  17. Biblio Neti, C., Potamianos, G., Luettin, J., Matthews, I., Glotin, H., Vergyri, D., Sison, J., Mashari, A., and Zhou, J. (2000). Audio-Visual Speech Recognition. Summer Workshop 2000 Final Technical Report, Center for Language and Speech Processing, The Johns Hopkins University, Baltimore, MD (http: //www.clsp.jhu.edu/ws2000/final reports/avsr/). Petajan, E.D. (1984) . Automatic lipreading to enhance speech recognition. Proceedings Global Telecommunications Conference (GLOBCOM)'84, Atlanta, GA, pp. 265-272.

  18. Rogozan, A., Deleglise, P., and Alissali, M. (1997) . Adaptive determination of audio and visual weights for automatic speech recognition. Proceedings European Tutorial Research Workshop on Audio-Visual Speech Processing (AVSP)'97, Rhodes, Greece, pp. 61-64. Summerfield, A.Q. (1987) . Some preliminaries to a comprehensive account of audio-visual speech perception. In Dodd, B. and Campbell, R. (Eds.), Hearing by Eye: The Psychology of Lip-Reading. Hillside, NJ: Lawrence Erlbaum Associates, pp. 97-113.

  19. Summerfield, Q., MacLeod, A., McGrath, M., and Brooke, M. (1989) . Lips, teeth, and the benefits of lipreading. In Young, A.W. and Ellis, H.D. (Eds.), Handbook of Research on Face Processing. Amsterdam, The Netherlands: Elsevier Science Publishers, pp. 223-233. Teissier, P., Robert-Ribes, J., Schwartz, J.-L., and Guerin-Dugue, A. (1999) . Comparing models for audiovisual fusion in a noisy-vowel recognition task. IEEE Transactions on Speech and Audio Processing, 7(6):629-642.

  20. Wark, T. and Sridharan, S. (1998) . A syntactic approach to automatic lip feature extraction for speaker identication. Proceedings International Conference on Acoustics, Speech, and Signal Processing (ICASSP)'98, Seattle, WA, pp. 3693-3696. A HYBRID ANN/HMM AUDIO-VISUAL SPEECH RECOGNITION SYSTEM Martin Heckmann, Frédéric Berthommier , Kristian Kroschel

  21. A HYBRID ANN/HMM AUDIO-VISUAL SPEECH RECOGNITION SYSTEMMartin Heckmann, Frédéric Berthommier , Kristian Kroschel

  22. C. Bregler, S. Manke, H. Hild, and A. Waibel, “Bimodal sensor integration on the example of speech-reading,” in Proc. IEEE Int. Conf. on Neural Networks, 1993, pp. 667– 671. A. Rogozan and P. Deléglise, “Adaptive fusion of acoustic and visual sources for automatic spech recognition,” Speech Communication, vol. 26, pp. 149–161, 1998.

More Related