MSc in RSIP Jie Zhang s0567061 10/02/06

1. Advanced Image ProcessingStudent Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi AndoFujitsu Laboratories Ltd., Atsugi, 243-0197 Japan ) MSc in RSIP Jie Zhang s0567061 10/02/06

2. Background • Why Lip reading? • Human computer interaction • Automatically recognize speech contents by processing lip movement images • Potential application for disable facilities • What is the principle? • Dealing with frame images sampled from video • Lip Feature extraction • Lip Feature recognition • Various methodology

3. Lip colour extraction algorithm Colour characteristics of lip feature Roughly mouth detection with colour algorithm Precise mouth position detection Eigen image Eigen template Matching algorithm Eigenwaveform recognition Mouth status Dictionary data Thresholding Training Lip images Test Lip images Colour extraction Colour extraction Eigenvector template build Eigentemplate Detection Eigenwaveform recognition System overview

4. Colour components of face feature Strongly brightness dependence ! Colour system HSV or RGB Evenly luminance Brightness normalization Lip region colour characteristics Three classes: lip, skin, teeth Looking for the separation of spectrum components Colour extraction algorithm Figure 1

5. Colour extraction algorithm Figure 2: Brightness dependent face feature colour distribution of normalised RGB image. Well distinguished in R &G!

6. Colour extraction algorithm • Lip Extraction with colour distribution • Threshold function of R and G to separate skin and mouth • Label and Extract largest bright area to appropriate size as it is deem to be mouth • Remove teeth and lips with second R and G threshold function • Position determination with four key points of the lips Figure 3 Lip edge is the combination of teeth and mouth cavity.

7. Eigentemplate method • Aim ----- precisely detect the location of lip! • Creation of eigen image • Use appropriate colour extracted image (trimmed image) • PCA + one dimension vector for a single trimmed image • Form image series matrix with x vectors

8. Eigentemplate method • Calculate eigenvector • Convert eigenvectors to eigenimage

9. Eigentemplate method Figure 4

10. Eigentemplate method • Eigentemplate • Trimmed test image • Recover eigenimage to template image • Similarity calculation Searching and trimming the image, compare trimmed image will template. Largest similarity gives the exact location.

11. Eigentemplate method Figure 5 Figure 6

12. Eigenwaveform recognition • Brief introduction to preprocessing • A threshold method to detect mouth states and hence recognize particular mouth shape • Define mouth state: i.e. open, wide open, close, tight closed… • Aspect ratio: distance b between upper and lower edge points over distance a between left and right edge points • Projection vector components: project the image of mouth and lip into eigen space through time scale. • Thresholding

13. Eigenwaveform recognition • Dictionary matching Small difference sw recognize the test utterance as the same as the template utterance Figure 7

14. Summary & Analysis • Lip reading system • Colour extraction method roughly classify lip and other face feature • Eigentemplate method precisely detect the location • Eigenwaveform algorithm recognize the utterance • Analysis • Widely used image processing technique • Hard to get high precision • Difficult for language different, culture different, appearance difference user • Potential problems of algorithm: brightness dependence; overlap between lip & skin; similar words; uncommon mouth shape; various speak speed and so on. • Need more improvement!

15. That’s all! Thank you very much! Question?