Face Recognition

1. Face Recognition Image Understanding Xuejin Chen

2. Face Recogntion • Good websites • http://www.face-rec.org/ • Eigenface [Turk & Pentland] Image Understanding, Xuejin Chen

3. Eigenface • Projecting a new image into the subspace spanned by the eigenfaces (face space) • Classifying the face by comparing its position in face space with the positions of known individuals Image Understanding, Xuejin Chen

4. Face image used as the training set Image Understanding, Xuejin Chen

5. Average face • 7 eigenfaces calculated from the training images Image Understanding, Xuejin Chen

6. Calculating Eigenfaces Face image : intensity matrix -D vector PCA: to find the vectors that best account for the distribution of face images within the entire image space Training images Average face Each face differs from the average by Seeks a set of M orthogonal vectors that best describe the distribution of the data The kth vector is chosen such that Image Understanding, Xuejin Chen

7. Calculating Eigenfaces • The vector and scalar are the eigenvectors and eigenvalues, respectively, of the covariance matrix Intractable task for typical image size Need a computationally feasible method to find these eigenvectors Image Understanding, Xuejin Chen

8. Calculating Eigenfaces • If image number M < N^2, only M meaningful eigenvectors • Consider eigenvectors vi of A’A such that 16x16  16384*16384 (an 128x128 image) Solve MxM matrix Image Understanding, Xuejin Chen

9. Calculating Eigenfaces • Construct MxM matrix • Compute M eigenvectors of L • The M eigenfaces is the linear combination of the M eigenvectors The computations are greatly reduced Image Understanding, Xuejin Chen

10. Average face • 7 eigenfaces calculated from the training images Image Understanding, Xuejin Chen

11. Classify a Face Image • 40 eigenfaces are sufficient for a good description for an ensemble of M=115 images [Sirovich and Kirby 1987] • A smaller M’ is sufficient for identification • C • hoose M’ significant eigenvectors of L matrix (M=16, M’=7) Image Understanding, Xuejin Chen

12. Classify a Face Image • New image : transformed into its eigenface components Image Understanding, Xuejin Chen

13. Classify a Face Image • describe the contribution of each eigenface in representing the input image, treating the eigenfaces as a basis set for face image • Find a class of the face • A simple distance Average of a class or individual Image Understanding, Xuejin Chen

14. Classify a Face Image • Creating the weight vector is: projecting the original face image onto the low-D face space • Distance from image to face space Image Understanding, Xuejin Chen

15. Four possibilities for an image and pattern vector • Near face space and near a face class • Near face space but not near a known face class • Distant from face space but near a face class • Distant from face space and not near a known face class Image Understanding, Xuejin Chen

16. Distance to face space • 29.8 • 58.5 • 5217.4 Image Understanding, Xuejin Chen

17. Summary of Eigenface Recognition • Collect a set of characteristic face image of the known individuals. • The set should include a number of images for each person, with some variation in expression and lighting (M=40) • Calculate the 40x40 matrix L, find its eigenvalues and eigenvectors, choose M’(~10) eigenvectors with highest eigenvalues • Compute eigenfaces • For each known individual, calculate the class vector by averaging the eigenface pattern vector . • Choose a threshold that defines the maximum allowable distance from any face class and • a threshold that defines the maximum allowable distance from face space Image Understanding, Xuejin Chen

18. Summary of Eigenface Recognition • For each new face image to be identified, calculate • its pattern vector , • the distance to face space , • the distance to each known class . • if the minimum distance • Classify the input face as the individual associated with class vector • If the minimum distance • The image may be classified as unknown, and • Optionally used to begin a new face class • Optionally, if the new image is classified as a known individual, the image may be added to the original set of familiar face image, and the eigenfaces may be recalculated (steps 1-4) Image Understanding, Xuejin Chen

19. Locating and Detecting Faces • Assume a centered face image, the same size as the training images and the eigenfaces • Using face space to locate the face in image • Images of faces do not change radically when projected into the face space, while the projection of nonface images appears quite different • > detect faces in a scene Image Understanding, Xuejin Chen

20. Use face space to locate face • At every location in the image, calculate the distance between the local subimage and face space, which is used as a measure of ‘faceness’  a face map Expensive calculation Image Understanding, Xuejin Chen

21. Face Map • A subimage at (x,y) Image Understanding, Xuejin Chen

22. Face Map • : linear combination of orthogonal eigenface vectors Image Understanding, Xuejin Chen

23. Face Map Correlation operator Image Understanding, Xuejin Chen

24. Face Map Precomputed L+1 correlations Can be implemented by a simple neural networks Image Understanding, Xuejin Chen

25. Learning to Recognize New Faces • An image is close to face space, but not classified as one of the familiar faces, labeled as “unknown” • If a collection of “unknown” pattern vectors cluster in the pattern space, a new face is postulated • Check similarity: the distance from each image to the mean is smaller than a threshold • Add the new face to database (optionally) Image Understanding, Xuejin Chen

26. Background Issue • Eigenface analysis can not distinguish the face from background • Segmentation? • Multiply the image by a 2D Gaussian window centered on the face • Deemphasize the outside of the face • Also practical for hairstyle changing Image Understanding, Xuejin Chen

27. Scale and Orientation Issue • Recognition performance decreases quickly as the size is misjudged • Motion estimation? • Multiscaleeigenfaces / multiscale input image • Non-upright faces • Orientation estimation using symmetry operators Image Understanding, Xuejin Chen

28. Distribution in Face Space • Nearest-neighbor classification assumes Gaussian distribution of an individual feature vector • No prior to assume any distribution • Nonlinear networks to learn the distribution by example [Fleming and Cottrell, 1990] Image Understanding, Xuejin Chen

29. Multiple Views • Define a number of face classes for each person • Frontal view • Side view at ± 45° • Right and left profile views Image Understanding, Xuejin Chen

30. Experiments • Database • Over 2500 face images under controlled conditions • 16 subjects • All combinations of 3 head orientations, 3 head sizes, 3 lighting conditions • Construct 6-level Gaussian pyramid from 512x512 to 16x16 Image Understanding, Xuejin Chen

31. Variation of face images for one individual Image Understanding, Xuejin Chen

32. Experiments with Lighting, Size, Orientation • Training sets • One image of each person, under the same lighting condition, size, orientation • Use seven eigenfaces • Mean accuracy as the difference between the training conditions, test conditions • Difference in illumination • Image size, • Head orientation • Combinations of illumination, size, orientation Image Understanding, Xuejin Chen

33. Changing lighting conditions --- few errors • Image size changing -- performance dramatically drops • Need multiscale approach (a) Lighting 96% (b) Size 85% (c) Orientation 64% (d) Orientation & lighting (e) Orienation & Size 1 (f) Orientation & Size 2 (g) Size & Lighting 1 (h) Size & Lighting 2 Image Understanding, Xuejin Chen

34. Experiments with varying thresholds • Smaller threshold: • Few errors, but more false negative • Larger threshold • More errors • To achieve 100% accurate recognition, boost unknown rate to • 19% while varying lighting • 39% for orientation • 60% for size • Set the unknown rates to 20%, the correct recognition rate • 100% for lighting • 94% for orientation • 74% for size Image Understanding, Xuejin Chen

35. Neural Networks • Can be implemented using parallel computing elements Image Understanding, Xuejin Chen

36. Collection of networks to implement computation of the pattern vector, projection into face space, distance from face space, and identification Image Understanding, Xuejin Chen

37. Image Understanding, Xuejin Chen

38. Conclusion • Not general recognition algorithm • Practical and well fitted to face recognition • Fast and simple • Do not require perfect identification • Low false-positive rate • A small set of likely matches for user-interaction Image Understanding, Xuejin Chen

39. Eigenface • Tutorial Image Understanding, Xuejin Chen

40. Bayesian Face Recognition BabackMoghaddam, Tony Jebaraand Alex Pentland Pattern Recognition 33(11), Nov. 2000

41. Novelty • A direct visual matching of face images • Probabilistic measure of similarity • Bayesian (MAP) analysis of image differences • Simple computation of nonlinear Bayesian similarity

42. A Bayesian Approach • Many face recognition systems rely on similarity metrics • nearest-neighbor, cross-correlation • Template matching • Which types of variation are critical in expressing similarity?

43. Probabilistic Similarity Measure • Intensity difference • Two classes of facial image variations • Intrapersonal variations • Extrapersonal variations • Similarity measure Non-Euclidean similarity measure Can be estimated using likelihoods given by Bayes rule

44. A Bayesian Approach • First instance of non-Euclidean similarity measure for face recognition • A generalized extension of • Linear Discriminant Analysis • FisherFace • Has computational and storage advantages over most linear methods for large database

45. Probabilistic Similarity Measures • Previous Bayesian analysis of facial appearance • 3 different inter-image representations were analyzed using the binary formulation • XYI-warp modal deformation spectra • XY-warp optical flow fields • Simplified I-(intensity)-only image-based difference

46. Probabilistic Similarity Measures • Intrapersonal variations • Images of the same individual with different expression, lighting conditions.. • Extrapersonal variations • Variations when matching two individuals • Both are Gaussian-distributed, learn the likelihoods

47. Probabilistic Similarity Measures • Similarity score • The priors can be set as the portion of image number in the database or specified knowledge • Maximum Posterior (MAP)

48. Probabilistic Similarity Measures • M individuals: M classes • Many classification -> binary pattern classification • Maximum likelihood measure • Almost as effective as MAP in most cases

49. Subspace Density Estimation • Intensity difference vector: high dimensional • No sufficient independent training examples • Computational cost is very large • Intrinsic dimensionality or major degree-of-freedom of intensity difference is significantly smaller than N • PCA • Divides the vector space R^N into two complementary subspaces [Moghaddam & Pentaland]

50. Subspace Density Estimation • Two complementary subspaces A typical eigenvalue spectrum