Lip Feature Extraction Using Red Exclusion

1. Lip Feature Extraction Using Red Exclusion Trent W. Lewis and David M.W. Powers Flinders University of SA VIP2000

2. Overview • Context • Lip Feature Extraction • Related Work (greyscale, horizontal edges, red and hue colour spaces) • Red Exclusion • AVSR: Results and Issues • Summary Lip Feature Extraction Using Red Exlucsion

3. Context • Audio Speech Recognition (ASR) • Psycholinguistic Research • Audio Visual Speech Recognition (AVSR) Lip Feature Extraction Using Red Exlucsion

4. Context - ASR • Up to 99% word accuracy • However, • limited context • limited vocabulary • trained on individual • close microphone, cannot handle noise Lip Feature Extraction Using Red Exlucsion

5. Context - Psycholinguistic • McGurk Effect • A[ba] + V[ga]  [da] • Viseme • visual phonemes • form complementary sets • Demonstrates vision can assist the perception of speech  AVSR Lip Feature Extraction Using Red Exlucsion

6. Acoustic Features Visual Features width height oral cavity Integration Early Late Context - AVSR Lip Feature Extraction Using Red Exlucsion

7. Lip Feature Extraction • Pixel-Based Model Lip Feature Extraction Using Red Exlucsion

8. Lip Feature Extraction • Pixel-Based Model • raw pixels or minimal processing • retain linguistically relevant data • large amounts of data,  time • shift and lighting variant • normalisation and PCA Lip Feature Extraction Using Red Exlucsion

9. Lip Feature Extraction • Pixel-Based Model • reduced input to set of hand-crafted features • width, height, average intensity, etc. • less features,  time • model fitting,  time • lose linguistically relevant features Lip Feature Extraction Using Red Exlucsion

10. Lip Feature Extraction Our Approach • Pixel-Based Model • feature extraction • Steps • preprocess to enhance contrast • locate mouth edges • identify corners, height, and other key features • train recogntion engine Lip Feature Extraction Using Red Exlucsion

11. Lip Feature Extraction • Database 1 2 3 Lip Feature Extraction Using Red Exlucsion

12. Lip Feature Extraction • Preprocessing Techniques • Grey-scale • Horizontal Edges • Red Analysis • Hue, Saturation, and Value (HSV) • Red Exclusion Lip Feature Extraction Using Red Exlucsion

13. Lip Feature Extraction • Grey-scale • vertical position of mouth • minimum row sum • threshold minimum row • average of min and max of row • search for above threshold pixels Lip Feature Extraction Using Red Exlucsion

14. Lip Feature Extraction • Grey-scale Lip Feature Extraction Using Red Exlucsion

15. Lip Feature Extraction • Grey-scale Lip Feature Extraction Using Red Exlucsion

16. Lip Feature Extraction • Horizontal Edges • high horizontal edge content • 3x3, DY Prewitt operator Lip Feature Extraction Using Red Exlucsion

17. Lip Feature Extraction • Horizontal Edges “Found” Corners Binary Image Lip Feature Extraction Using Red Exlucsion

18. Lip Feature Extraction • Red Analysis • overcome bearded subjects • used for face location Lip Feature Extraction Using Red Exlucsion

19. Lip Feature Extraction • Red Analysis Lip Feature Extraction Using Red Exlucsion

20. Lip Feature Extraction • HSV • disentangles illumination from colour • Illumination > Hue Lip Feature Extraction Using Red Exlucsion

21. Lip Feature Extraction • HSV Lip Feature Extraction Using Red Exlucsion

22. Lip Feature Extraction • Red Exclusion • needed extraction method for AVSR • similar to Red Analysis • face predominantly red • variations occur in the blue and green colours Lip Feature Extraction Using Red Exlucsion

23. Lip Feature Extraction • Red Exclusion Lip Feature Extraction Using Red Exlucsion

24. Lip Feature Extraction • Corners found using Red Exclusion Lip Feature Extraction Using Red Exlucsion

25. Lip Feature Extraction • Comparison Lip Feature Extraction Using Red Exlucsion

26. AVSR: Results and Issues • Application for red exclusion • Used in finding lip features • Width • Height • Key pixels Lip Feature Extraction Using Red Exlucsion

27. AVSR: Results and Issues • Visual Speech Recognition Lip Feature Extraction Using Red Exlucsion

28. AVSR: Results and Issues • AVSR - Integration Lip Feature Extraction Using Red Exlucsion

29. Summary • Vision can help ASR • AVSR • Needed good extraction technique • Red Exclusion • AVSR is difficult when both signals degraded Lip Feature Extraction Using Red Exlucsion

30. Questions? Trent W. Lewis BSc (Cognitive Science) Flinders University lewi0146@ist.flinders.edu.au Lip Feature Extraction Using Red Exlucsion