Using Attributes to Describe What People Wear

1. Using Attributes to Describe What People Wear Andy Gallagher October 14, 2013 with Huizhong Chen and Bernd Girod

2. Objective List of attributes Men’s Black color Sweater Long sleeve Solid pattern Low skin exposure … Attribute learning

3. 3

4. Outline • Attributes • Describing Clothing with Attributes • ! Miscellaneous Topics !

5. Attributes

6. Attributes • Describing objects by their attributes, A Farhadi, I Endres, D Hoiem, D ForsythComputer Vision and Pattern Recognition, 2009. CVPR 2009 • Learning To Detect Unseen Object Classes by Between-Class Attribute Transfer, C. Lampert, H. Nickisch, S. Harmeling, CVPR 2009 • Many others

7. Computer Vision image features classification

8. Computer Vision image ? [ .1 -.9 .1 .231 -.1] features classification

9. Computer Vision image What feature representation should we use? features classification

10. Computer Vision image [ .1 -.9 .1 .231 -.1] features Now we can talk… Has hair, has skin, has ear, has eye, has arms attributes classification

11. Attributes • Properties shared by many objects • Explicit semantics • Facilitate human-CPU communication • Materials (glass, fur, wood, etc.) • Parts (has wheel, has tail, etc.) • Shape (boxy, cylindrical, etc.) Based on a slide by David Forsyth 11

12. Example Attributes Face Tracer Image Search “Smiling Asian Men With Glasses” Kumar et al., 2008

13. Example Attributes Farhadi et al. 2009

14. Example Attributes Lampert et al. 2009 Slide credit: Devi Parikh

15. Example Attributes Welinder et al. 2010 Slide credit: Devi Parikh

16. Attribute Models • Classifiers for binary attributes Kumar et al. 2010 Slide credit: Devi Parikh

17. Why attributes? • How humans naturally describe visual concepts • Image search I want elegant silversandals withhigh heels Slide credit: Devi Parikh

18. Example Attributes Verification classifier SAME Kumar et al., 2010

19. Why attributes? • An okapi is a mammal with a reddish dark back, with striking horizontal white stripes on the front and back legs. (Wikipedia)

20. Why attributes? • An okapi is a mammal with a reddishdark back, with striking horizontal white stripes on the front and back legs. (Wikipedia)

21. Why attributes? • An okapi is a mammal with a reddishdark back, with striking horizontal white stripes on the front and back legs. (Wikipedia)

22. Zero-shot Learning • Aye-ayes • Are nocturnal • Live in trees • Have large eyes • Have long middle fingers Which one of these is an aye-aye? Humans can learn from descriptions (zero examples). Slide adapted from ChristophLampert by Devi Parikh

23. Is this a giraffe? No. Is this a giraffe? Yes. Is this a giraffe? No. Slide credit: Devi Parikh

24. Parkash and Parikh, 2012 Focused feedback Knowledge of the world Current belief I think this is a giraffe. What do you think? No, its neck is too short for it to be a giraffe. • Learner learns better from its mistakes • Accelerated discriminative learning with few examples [Animals with even shorter necks] …… Ah! These must not be giraffes either then. Feedback on one, transferred to many Slide credit: Devi Parikh

25. Which Attributes to Describe? (c) (a) (b) (f) (d) (e) Please choose a person to the left of the person who is frowning 25 Sadovnik et al. 2013

26. Describing Clothing with Attributes

27. Objective List of attributes Men’s Black color Sweater Long sleeve Solid pattern Low skin exposure … Attribute learning

28. Recommend and Analyze Recommendations Sport Formal

29. Related Work Person identification with clothing • Bounding box under face [Anguelov, 2007] • Clothing segmentation [Gallagher, 2008]

30. Dataset Preparation • 1856 people from the web. • Images are unconstrained.

31. Dataset Preparation $400 spent for collecting 283,107 labels on Amazon Mechanical Turk (AMT).

32. Dataset Statistics 23 Binary 3 Multiclass

33. The System Feature 1 SVM1 Combine features … Pose estimation … … SVM Attribute classifier 1 Feature N SVMN A: attribute F: feature F1 Feature extraction & quantization Attribute classifier 2 F2 F4 A1 Predictions Blue Solid pattern Outerwear Wear scarf Long sleeve A4 … A2 F3 Attribute classifier M Multi-attribute CRF inference A3 …

34. Pose Estimation [Eichner et. al., 2010] • Perform upper body detection, by using complementary results from face detector and deformable part models. • Foreground highlighting within the enlarged upper body bounding box. • Parse the upper body into head, torso, upper and lower parts of the left and right arms.

35. Feature Extraction • SIFT descriptor extracted over the sampling grid. • Similar procedure for the arm regions.

36. Feature Extraction • Maximum Response Filters [Varma 2005] • LAB color • Skin probability MRF bank Skin probability RGB image

37. Feature Extraction • Raw features are quantized using soft K-means (K=5 in our implementation). • Quantized features are aggregated over various body regions, by max or average pooling. • For learning color attributes, the feature is LAB color aggregated from non-skin regions.

38. Feature Fusion • SVM is a kernel-based classification technique. • Feature fusion solution: combined SVM is trained using weighted sum of the kernels. • Combining features consistently outperforms the single best feature. K1 SVM 1 K1 Predict accuracy 1 K2 SVM Combined Attribute prediction SVM 2 K2 Predict accuracy 2 … KN KN SVM N Predict accuracy N

39. Recap Feature 1 SVM1 Combine features … Pose estimation … … SVM Attribute classifier 1 Feature N SVMN A: attribute F: feature F1 Feature extraction & quantization Attribute classifier 2 F2 F4 A1 Predictions Blue Solid pattern Outerwear Wear scarf Long sleeve A4 … A2 F3 Attribute classifier M Multi-attribute CRF inference A3 …

40. Attribute Dependencies Necktie and T-Shirt?

41. Attribute Inference with CRF F6 • Each attribute is a node. All nodes are pair-wise connected. • The edge connecting 2 nodes corresponds to the joint probability of these 2 attributes. F5 A6 F1 A5 F4 A1 F2 A4 F3 Ai: Attribute i Fi: Features for Ai A2 A3

42. CRF for Attribute Learning [Following CRF model] • For a fully connected CRF, we maximize: • The CRF potential is maximized using standard belief propagation technique [Tappen et. al. 2003] . F2 … F1 FM … A2 A1 AM Node potential Edge potential

43. No necktie (Wear necktie) Has collar Men’s Has placket Low exposure No scarf Solid pattern Black Short sleeve (Long sleeve) V-shape neckline Dress (Suit) Wear necktie Has collar Men’s Has placket High exposure (Low exposure) No scarf Solid pattern Gray & black Long sleeve V-shape neckline Suit No necktie Has collar Men’s Has placket Low exposure Wear scarf Solid pattern Brown & black No sleeve (long sleeve) V-shape neckline Tank top (outerwear)

44. Experimental Results • Questions that we are interested in: • Does combining features improve performance? • Does the pose model help? • Does the CRF work?

45. Pose Vs No Pose - Experiment Setup • Positive and negative examples are balanced. • SVM classification • Chi-squared kernel • Leave-1-out cross validation • Comparison with attribute learning without pose model. • Features are extracted within a scaled clothing mask under the face. • Evaluation performed under the same experiment settings. The clothing mask [Gallagher 2008]

47. Multiclass Confusion Matrix

49. Steve Jobs: “solid pattern, men’s clothing, black color, long sleeves, round neckline, outerwear, wearing scarf”

50. The predicted dressing style of weddings: • Male: “solid pattern, suit, long-sleeves, V-shape neckline, wearing necktie, wearing scarf, has collar, has placket” • Female: “high skin exposure, no sleeves, dress, other neckline shapes, white, >2 colors, floral pattern”