Attributes for Classifier Feedback

1. Attributes for Classifier Feedback AmarParkash and Devi Parikh

2. You can teach a child by examples...

10. And on, and on, and on…

13. Is this a giraffe? No. Is this a giraffe? Yes. Is this a giraffe? No.

14. And on, and on, and on…

16. Proposed Active Learning Scenario

17. Focused feedback Knowledge of the world Current belief I think this is a giraffe. What do you think? No, its neck is too short for it to be a giraffe. • Learner learns better from its mistakes • Accelerated discriminative learning with few examples [Animals with even shorter necks] …… Ah! These must not be giraffes either then. Feedback on one, transferred to many

18. Communication Need a language that is • Machine understandable • Human understandable Attributes! • Mid-level shareable • Visual • Semantic

19. Proposed Active Learning [Label-feedback] [Attributes-based feedback] [Predicted Label] Attribute predictors: a1 a2 … aM Unlabeled pool of images Classifiers: Concepts to teach: h1 h2 … hK C1 C2 … CK Any feature space Any discriminative learning algorithm

20. Relative Attributes[Parikh and Grauman, ICCV 2011] Image features Attribute predictors: Parameters a1 a2 … aM Unlabeled pool of images Openness

21. Attributes-based Feedback No, It is too open to be a forest Forest Attribute predictors: a1 a2 … aM Unlabeled pool of images Unlabeled pool of images Not Forest Openness

22. Attributes-based Feedback No, It is too open to be a forest Forest Attribute predictors: a1 a2 … aM Unlabeled pool of images Unlabeled pool of images Classifiers: h1 h2 … hK Not Forest

23. Proposed Active Learning [Label-feedback] [Attributes-based feedback] [Predicted Label] Attribute predictors: a1 a2 … aM Unlabeled pool of images Classifiers: Concepts to teach: h1 h2 … hK C1 C2 … CK

24. Label-based Feedback • Not our contribution • Experiment with different scenarios • Benefits of attributes-based feedback • Small when label-based is very informative • Large when label-based feedback is weak

25. Label-based Feedback Forest • Accept: Yes, this is a forest. • Strong: It is not anything else Example: Classification • Weak: It can be other things Example: Annotation Unlabeled pool of images

26. Label-based Feedback Forest • Reject: • Strong: No, it is a coast. Example: Classification with few classes • Weak: No, this is not a forest. Example: Large-scale classification Example: Biased binary classification • 4 different scenarios in experiments Unlabeled pool of images

27. Datasets • Datasets and relative attribute predictors from [Parikh and Grauman, ICCV 2011]

28. Datasets • Faces: • 8 celebrity categories • 11 attributes (chubby, white, etc.) [Kumar et al., ICCV 2009]

29. Datasets • Scenes: • 8 categories • 6 attributes (open, natural, etc.) [Oliva and Torralba, IJCV 2001]

30. Settings • Feedback from MTurk • Features: • Raw image features (gist, color) • Attribute scores • Category classifiers: SVM with RBF kernel • Results on • 2 datasets x 2 features x 4 label-feedback scenarios • Show 2 here, rest in paper. This image doesn’t have enough perspective to be a street scene.

31. Results Faces, Attribute features, Strong label-feedback Accuracy # iterations

32. Results Scenes, Image features, Weak label-feedback 1/4th ! Accuracy # iterations More results in the paper.

33. Conclusion • Attributes for providing classifier feedback • Novel learning paradigm with enhanced human-machine communication • Discriminative learning + domain knowledge • Learning with few examples • Connections to semi-supervised learning • Shrivastava, Singh and Gupta: Up Next!

34. Thank you!

35. Backup Slides

36. Negative Feedback • Many reasons come together to make a concept • Hard to describe why an image is a concept • One reason can break a concept • Easier to describe why an image is not a concept • (Arguably) waste to give feedback when right

37. Related Work • Zero-shot learning • Convey domain knowledge • Zero training images • Generative model • Classification in attribute space • Performance compromised • Proposed paradigm • Convey domain knowledge to transfer • Few training images • Discriminative model • Classification in any feature space • Performance maintained • Discriminative • No domain knowledge is conveyed • Many training images • Discriminative model • Classification in any feature space • State-of-art performance – – S – C Smiling + J + H + Cis younger than H C smiles more than H M Age Mis younger than J Msmiles more than J [Lampert et al., CVPR 2009] [Parikh and Grauman, ICCV 2011]

38. Related Work Focused discrimination • Mining of hard negatives [Felzenszwalb 2010] • To understand classifier [Golland 2001] • Here, supervisor provides the discriminative direction by verbalizing semantic knowledge [Felzenszwalb et al., CVPR 2008] [Golland, NIPS 2001]

39. Related Work What we are not doing: • Collecting deeper annotations of images • Segmentation masks [Russell 2008], Parts [Farhadi 2010], Pose [Bourdev 2009], Attributes [Kumar 2009] • We use attributes for broad propagation of category labels to unlabeled images

40. Related Work What we are not doing: • Actively interleaving attribute annotations • Object & attributes [Kovashka 2011], Image, boxes & segments [Vijaynarsimhan 2008], Parts & attributes [Wah 2011], etc. • Human-in-the-loop at test time [Branson 2010] • Our supervisor provides additional information at training time which is leveraged for better category models

41. Related Work • Rationales • Human feature selection in NLP [Raghavan 2005] • Spatial and attribute rationales [Donahue 2011] • Restricted to classification in attribute space • We can operate in any feature space [Donahue and Grauman, ICCV 2011]

42. Imperfect Attribute Predictors • In the end, all images labeled with ground truth • Discriminative training can deal with outliers • Attributes are pre-trained, and so unlikely to be severely flawed • Experiments: used predictors directly from Parikh and Grauman, 2011

43. Large-scale Classification • Categories may require expert knowledge, attributes need not • Can show a few exemplars to verify category

44. Results Faces, Classification, Attribute features (overestimate) Accuracy # iterations

45. Label-based Feedback • Simulate the different scenarios in responses Benefits of attributes-based feedback • Small when label-based is very informative • Large when label-based feedback is weak Classification Large-scale Classification Street Outdoor City Grayscale …. Annotation Biased Binary Classification

46. Results Scenes, Annotation, Image features Accuracy # iterations

47. Results Faces, Biased binary classification, Attribute features Accuracy # iterations More results in the paper.

48. Traditional Active Learning Highest Entropy [Label] Unlabeled pool of images Classifiers: Concepts to teach: h1 h2 … hK C1 C2 … CK