Wrapping Snakes For Improved Lip Segmentation

1. Wrapping SnakesFor Improved Lip Segmentation Matthew Ramage Dr Euan Lindsay (Supervisor) Department of Mechanical Engineering

2. Presentation Overview • Brief overview of Visual Speech Recognition • Requirements of a good Lip Segmenter • One approach: Traditional Snakes • An improved approach: “Wrapping Snakes”

3. What is Visual Speech Recognition? • Uses video footage to identify spoken words • Does not use any audio information • Audio-only speech recognition accuracy falls dramatically as audio signal degraded • Basic Principle: • Mouth shape is based on the underlying structure of the word being spoken

4. What makes a good Lip Segmenter? • Accurately represents lip shape • Robust against poor initialisation • Can handle multiple enclosed regions • Reliably handles noisy images • Can handle weak lip features near strong noise • Quickly finds the lip shape

5. One Approach: Traditional Snakes • A form of Active Contour Model • Series of connected points • Internal forces: • Tension • Rigidity • External forces: • Constraints • Image Forces: • Based on image features Using snakes to track the outer lip boundary (Kass et al. 1988) Kass, M, Witkin, A & Terzopoulos, D, 1988, 'Snakes: Active contour models', International Journal of Computer Vision, vol. 1, no. 4, pp. 321-31.

6. Traditional Snakes

7. Problems with Traditional Snakes • Not robust against poor initialisation • Cannot handle multiple enclosed regions • Cannot reliably handle noisy images • Cannot handle weak target features with strong noise

8. Multiple Enclosed Regions (Traditional Snake)

9. Weak Features & Strong Noise (Traditional Snake)

10. The Problem? • Traditional Snakes don’t continue along features

11. The Problem? • Traditional Snakes don’t continue along features

12. The Solution: A Wrapping Force • A substitute for the Image Force • Based on the Image Force • Modified by the snake’s shape and position • When a snake curves away from a feature, apply a force along the feature • Pushes snake along the feature

13. Calculating the Wrapping Force • Component of Image Force in the direction of the Snake’s Normal

14. Wrapping Behaviour

15. Cutting the Snake • Cut the snake when separate sections come into contact • Benefits: • Distinguishes between multiple fully enclosed regions • Ignores areas of noise near the lips • Allows the lip shape to be accurately found • Increased robustness of lip segmentation

16. Pinching Forces • A subset of points a made “Pinch Points” • Non-adjacent pinch points attract when close enough • Pulls sections of the snake together when close enough • Helps with cutting

17. Multiple Enclosed Regions (Wrapping Snake)

18. Weak Features & Strong Noise (Wrapping Snake)

19. Properties of Wrapping Snakes • Accurately represents lip shape • Robust against poor initialisation • Can handle multiple enclosed regions • Reliably handles noisy images • Can handle weak lip features near strong noise • Quickly finds the lip shape

20. Wrapping SnakesFor Improved Lip Segmentation Matthew Ramage Dr Euan Lindsay (Supervisor) Department of Mechanical Engineering

21. Handles Very Poor Initial Position (Wrapping Snake)

22. Multiple Enclosed Regions (Traditional Snake)

23. Multiple Enclosed Regions (Wrapping Snake)