Relative Attributes

1. Relative Attributes Mingxia Liu mingxialiu@nuaa.edu.cn

2. Outline • 1. Introduction • 2. Relative Attributes[1] • 3. Discussion • 4. Our Intent Work [1] Devi Parikh and Kristen Grauman.Relative Attributes. ICCV 2011 Best Paper.

3. Traditional Recognition Tiger ??? Dog Chimpanzee Tiger

4. Attributes-based Recognition Attributes provide a mode of communication between humans and machines! Striped Black White Big Furry White Black Big Striped Yellow Dog Chimpanzee Tiger [Lampert 2009] [Farhadi 2009] [Kumar 2009] [Berg 2010] [Parikh 2010] … Zero-shot learning Describing objects Face verification Attribute discovery Nameable attributes …

5. Binary or Relative Attributes?

6. Binary or Relative Attributes? “Smiling”=0 “Smiling”=1 ??? Smiling less than Gao, more than Ge

7. Binary or Relative Attributes? Horse Donkey Mule

8. Attributes for Mule Has four-legs Is furry Tail longer than donkeys’ Legs shorter than horses’ Has tail [Oliva 2001] [Ferrari 2007] [Lampert 2009] [Farhadi 2009] [Kumar 2009] [Wang 2009] [Wang 2010] [Berg 2010] [Branson 2010] [Parikh 2010] [ICCV 2011] …

9. Binary Has four-legs Is furry Tail longer than donkeys’ Legs shorter than horses’ Has tail

10. Relative Has four-legs Is furry Tail longer than donkeys’ Legs shorter than horses’ Has tail

11. Relative Attributes • Enhanced human-machine communication • More informative • Natural for humans

12. Outline • 1. Introduction • 2. Relative Attributes • 3. Discussion • 4. Our Intent Work

13. Contents • Relative attributes • Allow relating images and categories to each other • Learn ranking function for each attribute • Novel applications • Zero-shot learning from attribute comparisons • Automatically generating relative image descriptions

14. 1) Relative Attributes Annotation Attribute coast (C), forest (F), highway (H), inside-city (I), mountain (M), open-country(O), street (S) and tall-building (T) 8 categories, 6 attributes

15. 2) Learning Relative Attributes For each attribute , supervision is “open”

16. 2) Learning Relative Attributes Learn a scoring function Image features Learned parameters that best satisfies constraints:

17. 2) Learning Relative Attributes Ranking SVM Image  Relative Attribute Score Based on [Joachims 2002]

18. 2) Learning Relative Attributes Wm 1 2 3 Rank Margin 4 5 6

19. 3) Relative Zero-shot Learning • N : total categories • N = S + U • S : ‘seen’ categories • training images are provided relative attribute relation such as “lions are larger than dogs, as large astigers, but less large than elephants” • U : ‘unseen’ categories • training images are not provided

20. 3) Relative Zero-shot Learning • How to relate Seen categories and Unseen categories through relative attributes?

21. 3) Relative Zero-shot Learning • For attribute m •  •  •  • If attribute m is not used to describe  to be the mean of all training image -

22. 3) Relative Zero-shot Learning • Step1: Compute the attribute rank score: • Step2: Compute the mean and covariance of according to relative attribute description. • Step 3: Assign class label for using the Maximum Likelihood Method: 

23. 3) Relative Zero-shot Learning Need not use all attributes, or all seen categories. Age: Scarlett Scarlett Hugh Clive Clive Hugh Miley Jared Miley Jared Smiling:

24. 3) Relative Zero-shot Learning Age: Scarlett Hugh Clive S Miley Jared Miley Clive Smiling: Smiling Clive J Hugh Scarlett H Jared Miley Infer image category using max-likelihood Age Can predict new classes based on their relationships to existing classes – without training images

25. 4) Automatic Relative Image Description Novel image Density Conventional binary description: not dense Dense: Not dense:

26. 4) Automatic Relative Image Description Novel image Density less dense than more dense than

27. Experiments

28. Datasets Public Figures Face (PubFig) [Kumar 2009] 8 classes 800 images 11 attributes: white, chubby, etc. Outdoor Scene Recognition (OSR) [Oliva 2001] 8 classes 2700 images 6 attributes: open, natural, etc.

29. Ranker vs. Classifier – – – + + +

30. Baselines • Zero-shot learning • Binary attributes: Direct Attribute Prediction [Lampert 2009] • Relative attributes via classifier scores • Automatic image-description • Binary attributes

31. Relative Zero-shot Learning Rel. att.(ranker) Rel. att. (classifier) Binary attributes An attribute is more discriminative when used relatively

32. Relative zero-shot learning Classifier score Binary attributes Proposed

33. Automatic Relative Image Description Binary (existing): Not natural Not open Has perspective Relative (ours): More natural than insidecity Less natural than highway More open than street Less open than coast Has more perspective than highway Has less perspective than insidecity

34. Automatic Relative Image Description Binary (existing): Not natural Not open Has perspective Relative (ours): More natural than tallbuilding Less natural than forest More open than tallbuilding Less open than coast Has more perspective than tallbuilding

35. Automatic Relative Image Description Binary (existing): Not Young BushyEyebrows RoundFace Relative (ours): More Young than CliveOwen Less Young than ScarlettJohansson More BushyEyebrows thanZacEfron Less BushyEyebrows than AlexRodriguez More RoundFace than CliveOwen Less RoundFace than ZacEfron (Viggo)

36. Outline • 1. Introduction • 2. Relative Attributes • 3. Discussion • 4. Our Intent work

37. Discussion • Relative attributes representation • Attribution relation learning

38. Outline • 1. Introduction • 2. Relative Attributes • 3. Discussion • 4. Our Intent work

39. Our Intent Work • Learning attributes’ relation automatically • Learning attributes’ relation with noise • Attribute selection • Instance selection for attributes

40. Outline • 1. Introduction • 2. Relative Attributes • 3. Discussion • 4. Our Intent work

41. Thank You