Using Group Prior to Identify People In Consumer Images

1. Using Group Prior to Identify People In Consumer Images Andrew C. Gallagher Tsuhan Chen Carnegie Mellon University Eastman Kodak Company June 18, 2007 CVPR SLAM 2007

2. The Problem • Consumer image collections are growing exponentially each year. • Consumers want to search for images based on whom the image contains. And they don’t like to label images! • This is more than a face recognition problem. To best understand the semantics of who is in the images, we need to understand the people in the images. CVPR SLAM 2007

3. Traditional Face Recognition • Determines the assignment of each person independently • Extract facial features • Build a classifier that finds the most likely name, given the features. • But this method does not take full advantage of the available information! CVPR SLAM 2007

4. The Group Prior for Learning The Semantics of People in Images • Determine the joint assignment of all people in the image to names, using the group prior. • By the unique object constraint (UOC), an individual can appear only once in the image. • The group prior characterizes the prior probability of certain groups of people appearing together in an image. CVPR SLAM 2007

5. Jonah Holly Holly Jonah Hannah Andy Jonah System Diagram HannahJonahHolly Andy Jonah Holly Jonah Images (Faces) Ambiguous Label Resolution Labeled Faces Ambiguous Labels Group Prior Classifier Training Unlabeled Image Annotated Image Recognize People Hannah Holly CVPR SLAM 2007

6. Posterior Probability Likelihood Individual Prior Recognizing a Person • When a single person is in the image: : the set of all unique names : a member of the set : the features from person image CVPR SLAM 2007

7. p1 pM p2 … f1 fM … f2 Recognizing Multiple People The Group Prior • The graph model represents the features and people in an image. • The graph encodes the independence assumptions of our model. • E.g. given the identity of a person, their features are independent of others in the image. CVPR SLAM 2007

8. p1 pM p2 … f1 fM Likelihood The Group Prior … f2 Recognizing Multiple People The Group Prior • The joint probability function: : an index over the people in the image : the set of all features for all people : the set of people in the image : a subset of ; a particular assignmentof a name to each person in . CVPR SLAM 2007

9. Estimating the Group Prior • For pairs of names, the group prior is estimated by counting the number of images the pair appears, then normalizing. • The group prior for 3 or more people is estimated according to the group prior pairwise graphical model. The Group Prior The Individual Prior CVPR SLAM 2007

10. Jonah Holly Holly Jonah Hannah Andy Jonah System Diagram HannahJonahHolly Andy Jonah Holly Jonah Images (Faces) Ambiguous Label Resolution Labeled Faces Ambiguous Labels Group Prior Classifier Training Unlabeled Image Annotated Image Recognize People Hannah Holly CVPR SLAM 2007

11. Jonah Jonah Jonah Jonah Jonah Hannah Hannah Hannah Hannah Hannah Andy Andy Andy Andy Andy Andy Andy Andy Andy Andy Hannah Hannah Hannah Hannah Hannah Hannah Hannah Hannah Hannah Hannah Jonah Jonah Jonah Jonah Jonah Hannah Hannah Hannah Hannah Hannah Jonah Jonah Jonah Jonah Jonah Hannah Hannah Hannah Hannah Hannah Holly Holly Holly Holly Holly Jonah Jonah Jonah Jonah Jonah Hannah Hannah Hannah Hannah Hannah Holly Holly Holly Holly Holly Jonah Jonah Jonah Jonah Jonah Hannah Hannah Hannah Hannah Hannah Jonah Jonah Jonah Jonah Jonah Hannah Hannah Hannah Hannah Hannah Jonah Jonah Jonah Jonah Jonah Andy Andy Andy Andy Andy Jonah Jonah Jonah Jonah Jonah Holly Holly Holly Holly Holly Andy Andy Andy Andy Andy Andy Andy Andy Andy Andy Hannah Hannah Hannah Hannah Hannah Andy Andy Andy Andy Andy Hannah Hannah Hannah Hannah Hannah Holly Holly Holly Holly Holly Jonah Jonah Jonah Jonah Jonah Andy Andy Andy Andy Andy Jonah Jonah Hannah Jonah Jonah Andy Andy Andy Andy Andy Hannah Hannah Hannah Hannah Hannah Jonah Jonah Jonah Jonah Jonah Hannah Hannah Hannah Hannah Hannah Andy Andy Andy Andy Andy Holly Holly Holly Holly Holly Ambiguous Labels • Ambiguous labels indicate who is in the image, but not which person is which name. • A constrained clustering algorithm is used to ‘resolve’ the labels. • The resolved labels are used to learn the feature distribution for each name. CVPR SLAM 2007

12. Classification with Group Prior • From the joint pdf, inference questions can be answered: • Most Probable Explanation MAP • MAP- Most probable assignment of a particular person . CVPR SLAM 2007

13. Experiment • The image collection: • Facial Features: Active Shape Model [Cootes95] based features, then PCA reduces to 5D. • Ambiguously label a portion of the image collection, classify the identities of all the rest. • Compare 4 Priors: • Group Prior (GP) • UOC Prior A binary version of the GP that respects the UOC. • The individual prior. • No Prior • The performance is quantified for: • MAP • MPE CVPR SLAM 2007

14. Results • Group Prior produces a large benefit. • Note: All images were ambiguously labeled; no people were explicitly labeled. CVPR SLAM 2007

15. Prior Work • Many face recognition methods- most ignore the issue of prior probabilities. [Zhao03] • Face recognition methods have been used to assist the labeling of image collections. [Zhang04] • In news photos, names from captions have been assigned to faces. [Berg04] • The co-occurrence of people in images has been studied, but not combined with image features. [Naaman05] CVPR SLAM 2007

16. Conclusions • The group prior models the social relationships between individuals. • We learn feature distributions and relationships between the labels (people). • By using the group prior, recognition accuracy is significantly improved! CVPR SLAM 2007