Ongoing Challenges in Face Recognition

1. Ongoing Challenges in Face Recognition Peter Belhumeur Columbia University New York City

2. How are people identified? People are identified by three basic means: • Something they have(identity document or token) • Something they know(password, PIN) • Something they are(human body)

3. Iris

4. Retina Every eye has its own totally unique pattern of blood vessels.

5. Hand

6. Fingerprint

7. Ear

8. Face

9. Who are these people? [Sinha and Poggio 1996]

10. Who are these people? [Sinha and Poggio 2002]

11. Images as Points in Euclidean Space x 1 x x n 2 • Let an n-pixel image to be a point in an n-D space, x  Rn. • Each pixel value is a coordinate of x.

12. Face Recognition: Euclidean Distances D1 > 0 D2 > 0 ~ D3= 0

13. Face Recognition: Euclidean Distances D1 > 0 D2 > 0 D3 > D1 or 2 [Hallinan 1994] [Adini, Moses, and Ullman 1994]

14. Same Person or Different People

16. Same Person or Different People

18. Why is Face Recognition Hard?

19. Challenges: Image Variability Long Term • Facial Hair • Makeup • Eyewear • Hairstyle • Piercings • Aging Short Term Expression Illumination Pose

20. Illumination Invariants? Does there exist a function f s.t. f ( ) = f ( ) = f ( ) = a and f ( ) = f ( ) = f ( ) = b ?

21. Can Any Two Images Arise from a Single Surface? n s a I(x,y) = a(x,y) n(x,y) s I(x,y) n Same Albedo and Surface Different Lighting l a J(x,y) = a(x,y) n(x,y) l J(x,y)

22. The Surface PDE I(x,y) = a(x,y) n(x,y) s ( I l – J s )n = 0 J(x,y) = a(x,y) n(x,y) l Linear PDE Nonlinear PDE

23. Non-Existence Theorem for Illumination Invariants Illumination invariants for 3-D objects do not exist. This result does not ignore attached and cast shadows, as well as surface interreflection. [Chen, Belhumeur, and Jacobs 2000]

24. Geometric Invariants? Does there exist a function f s.t. f ( ) = f ( ) = f ( ) = a and f ( ) = f ( ) = f ( ) = b ?

25. Non-Existence Theorem for Geometric Invariants Geometric invariants for rigid transformations of 3-D objects viewed under perspective projective projection do not exist. [Burns, Weiss, and Riseman 1992]

26. Image Variability: Appearance Manifolds x2 xn x1 Lighting x Pose [Murase and Nayar 1993]

27. Modeling Image Variability Can we model illumination and pose variability in images of a face? Yes, if we can determine the shape and texture of the face. But how?

28. Modeling Image Variability: 3-D Faces ] Laser Range Scanners Stereo Cameras Structured Light Photometric Stereo [Atick, Griffin, Redlich 1996] [Georghiades, Belhumeur, Kriegman 1996] [Blanz and Vetter 1999] [Zhao and Chellepa 1999] [Kimmel and Sapiro 2003] [Geometrix 2001] [MERL 2005]

29. Illumination Variation Reveals Object Shape n s2 s1 a s3 I1 I2 I3 [Woodham 1984]

30. Illumination Movie Illumination Movie

31. Shape Movie Shape Movie

32. Image Variability: From Few to Many Lighting x Pose x2 xn x1 Real Synthetic [Georghiades, Belhumeur, and Kriegman 1999]

33. Illumination Dome

34. Real vs. Synthetic Synthetic Real

35. Real vs. Synthetic Real Synthetic

36. A Step Back in Time

37. Albrecht Dürer, “Four Books on Human Proportion” (1528)

38. D’arcy Thompson, “On Growth and Form” (1917)

39. D’arcy Thompson, “On Growth and Form” (1917)

40. D’arcy Thompson, “On Growth and Form” (1917)

41. But what if we could ….? [Blanz and Vetter 1999, 2003]

42. Building a Morphable Face Model [Blanz and Vetter 1999, 2003]

43. 3-D Morphaple Models: Semi-Automatic [Blanz and Vetter 1999, 2003]

44. Building Morphable Face Models [Blanz and Vetter 1999, 2003]

45. Fitting Morphable Face Models [Blanz and Vetter 1999, 2003]