Visual Processing for Social Media

1. Visual Processing for Social Media Andrew C. Gallagher Tsuhan Chen September 30, 2012 Cornell University 1 TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAA

2. Outline • Social Media Overview • Visual Processing Overview • Social Media Insights Within the Image • Social Media Insights From Sharing 2 TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAA

3. Now, pictures of people • Examples of how social data has helped understand images of people • Some things I’ve learned about people from computer vision 3

4. Understanding images of people 4

5. What the computer sees 5

6. Faces in the lab [Blanz et al., PAMI 2003] [Belhaumer et al., PAMI 1997] [Turk et al., Cog. Neuro. 1991] [Wiskott et al., PAMI, 1997] [Lucey et al., IJCV 2007] [Kanade, Kyoto U. 1973] 6

7. Is this a family? 8

8. The Loop Images and Computer Vision What we know about people 9

9. Understanding Images of People • Describe people: How tall? How old? • Identify people: Who? • Why are they together? • Exploit the same context humans use! 10

10. Understanding Images of People • Capture Context • Social Context July 2, 20058:27 PMLat: 42.2902Long: 85.5361 June 25, 200510:50 AMLat: 42.3202Long: 85.1261 11

11. Understanding Images of People • Capture Context • Social Context Adult male height: 177 cm Adult female height: 163 cm MLE mother-child: 27 yearsMLE husband-wife: 2 years MLE |sibling-sibling|: 6 years 12

12. What is social context? Social Context: information about people and their society that is useful for understanding images. • Distributions of ages and genders in social groups • Social relationships • Face position in a group image • First name popularity over time • Anthropometric measurements 13

13. Group Images • What we know and learn about people: • Group dynamics • Computer vision task: • Measuring age, gender, of each person in a group 14

14. Images of Groups • Identify age and gender • Recognize certain group events • Consider context and appearance 15 [A. Gallagher, T. Chen, CVPR 2009]

15. Contextual Features • Absolute face position • Size, position relative neighbor and group • Minimal spanning tree degree 16

16. Evidence of Social Context • Relative positions of nearest neighbors depends on the social relationship • Mean distance is 306 mm Neighbors Male to Female Other to Baby 17

17. Evidence of Social Context • Samples of faces based on image location Random samples 19

18. Use All Context • 5080 images with 28,231 faces • Classification improves with more contextual features 20

19. Appearance Features • Project face into Fisher Space • Nearest neighbor density estimation Gender subspace Nearest neighbors 21

20. Gender Estimation 22 Context Appearance Combined

21. Context and Appearance • Context contributes more when appearance is weak. Context Appearance Combined Small Faces All Faces 24

22. Context for Scene Geometry Find the face vanishing line Estimated horizon from face positions Manually labeled horizon 25

23. Context for Dining Event • Group Structure = Activity 26

24. Row Segmentation 27 [A. Gallagher, T. Chen, ICME 2009]

25. Row Segmentation 29

26. Row Segmentation 30

27. Social Relationship Retrieval Mother-Child Spouses 31 • [G. Wang, A. Gallagher, J. Luo, D. Forsyth, ECCV 2010]

28. Names as Context • What we know and learn about people: • Government census data • Computer vision task: • Matching names to faces. Guessing age and gender. 32

29. First names capture information about age and gender First names are social context First Names as Context Person A and Person B Mildred and Lisa Source: U.S. Social Security Administration 33 [A. Gallagher, T. Chen, CVPR 2008]

30. First names and appearance? Tom_101 Ben_165 Caleb_337 Andrew_233 Brian_116 Zachary_431 1953 1996 1956 2003 1984 1962 Abigail_194 Heather_224 Alejandra_152 Juanita_192 Ethel_165 Gertrude_53 2002 1924 1970 1977 1947 1926 34

31. Tom_101 Ben_165 Caleb_337 Andrew_233 Brian_116 Zachary_431 1953 1996 1956 2003 1984 1962 Abigail_194 Heather_224 Alejandra_152 Juanita_192 Ethel_165 Gertrude_53 2002 1924 1970 1977 1947 1926 Sort by Expected Age 35

32. Name Birth Year Gender Age Features Gender Features Image-Based Features First Names as Context Mildred and Lisa 36

33. More Context = Better Results Appearance First Name Full Model 37

34. The model improves name assignment, age estimation, and gender classification Recognition from First Name 38

35. Learning about people What we know about people Images and Computer Vision 39 39

36. Group Images • How close do people stand in group photos? • Computer vision answer: 306 mm • Sociology’s “Personal Space”: 457 mm • Do people suspend personal space needs during photograph? 40

37. Group Images: Gender Prior • How do people end up in a group photo anyhow? 41

38. Group Images: Gender Prior • Bernoulli world? • Implicit prior, IID: • Let’s look at the data! 42

39. Gender Distribution of 6 people “Group Shots” Binomial Distribution ? Number of Females Number of Females “Family” Actual Distributions ? Genders of people in a image are not independent! Number of Females 43

40. Group Shot Analysis • Standing Order Frequency for 4 people (2 male, 2 female): 0.13 0.11 But why? 0.19 0.13 0.30 0.15 44

41. Learning about people What we know about people (what they do and think!) Images and Computer Vision 45 45 45

42. Social Context Data Summary • U.S. Social Security First Name Database • 6693 first names, birth years, gender • U.S. CDC National Center for Health Statistics • Physical growth tables • Birth rates and other birth statistics • Family structure statistics • Farkas, 1994 • Facial anthropometric measurements 46

43. Conclusions • Social context is useful for interpreting single images or image collections • Social context is learned from images or other public sources • Learning about people improves our understanding of images of people 47

44. Related Publications 48