Constrained Semi-Supervised Learning using Attributes and Comparative Attributes

1. Constrained Semi-Supervised Learning using Attributes and Comparative Attributes Abhinav Shrivastava, Saurabh Singh, Abhinav Gupta The Robotics Institute Carnegie Mellon University

2. Supervision Big-Data Active Learning Supervised [Prakash and Parikh, ECCV 2012] [Vijayanarasimhan and Grauman, CVPR 2011] [Kapoor et al., ICCV 2007] [Qi et al., CVPR 2008] [Joshi et al., CVPR 2009] [Siddiquie and Gupta, CVPR 2010] [von Ahn and Dabbish, 2004], [Russell et al., IJCV 2009]

3. Supervision Active Learning Unsupervised Supervised [Russell et al., CVPR 2006] [Kang et al., ICCV 2011]

4. Supervision Semi-Supervised Active Learning Unsupervised Supervised [Zhu, TR, 2005], [Chunsheng Fang, Slides, 2009]

5. Supervision Semi-Supervised Active Learning Unsupervised Supervised Bootstrapping Labeled Seed Examples Retrain Models Train Models Unlabeled Data Select Candidates Amphitheatre Amphitheatre Add to Labeled Set

6. Supervision Semi-Supervised Active Learning Unsupervised Supervised Bootstrapping Labeled Seed Examples Retrain Models Unlabeled Data Select Candidates Amphitheatre Amphitheatre + Auditorium Amphitheatre Add to Labeled Set Semantic Drift 25th Iteration [Curran et al., PACL 2007]

7. Supervision Semi-Supervised Active Learning Unsupervised Supervised Graph-based Methods [Ebert et al., ECCV 2010] [Fergus et al., NIPS 2009]

8. Our approach Amphitheatre Amphitheatre Amphitheatre Amphitheatre Auditorium Auditorium Auditorium Auditorium

9. Our approach Joint Learning Amphitheatre Amphitheatre Auditorium Auditorium Share Data [Carlson et al., NAACL HLT Workshop on SSL for NLP 2009]

10. Binary Attributes (BA) Amphitheatre Amphitheatre Conference Room Conference Room Indoor Indoor Tables and Chairs Tables and Chairs Man-made Man-made Large Seating Capacity Large Seating Capacity Banquet Hall Banquet Hall Auditorium Auditorium [Farhadi et al., CVPR 2009] [Lampert et al., CVPR 2009]

11. Binary Attributes (BA) Tables and Chairs Tables and Chairs Large Seating Capacity Large Seating Capacity Conference Room Conference Room Amphitheatre Amphitheatre Man-made Man-made Indoor Indoor Banquet Hall Banquet Hall Auditorium Auditorium [Patterson and Hays, CVPR 2012]

12. Auditorium Indoor Has Seat Rows   ✗ 

13. Sharing via Dissimilarity Amphitheatre Auditorium Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]

14. Has Larger Circular Structures Amphitheatre Auditorium ?

15. Has Larger Circular Structures Amphitheatre Auditorium  ✗  

16. Dissimilarity Comparative Attributes Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]

17. Dissimilarity Comparative Attributes • Features • GIST • RGB (Tiny Image) • Line Histogram of: • Length • Orientation • LAB histogram Has Larger Circular Structures ………… ………… [Hays and Efros, SIGGRAPH 2007] [Oliva and Torralba, 2006] [Torralba et al., PAMI, 2008] [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]

18. Dissimilarity Comparative Attributes Classifier Boosted Decision Tree [Hoiem et al., IJCV 2007] Has Larger Circular Structures ✗  ………… ………… or Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]

19. Comparative Attributes Is More Open Has Taller Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]

20. Selected Candidates Labeled Seed Examples Bootstrapping Amphitheatre Amphitheatre Auditorium Auditorium

21. Selected Candidates Our Approach (Constrained Bootstrapping) Labeled Seed Examples Bootstrapping Indoor Has Seat Rows Amphitheatre Auditorium Auditorium Amphitheatre Amphitheatre Comparative Attributes Attributes Has Larger Circular Structures Auditorium

22. banquet hall Banquet bedroom Scene Classifiers Bedroom Unlabeled Data has grass indoor Labeled Data Attribute Classifiers has more space more space larger structures has larger structures Comparative Attribute Classifiers Training Pairwise Data Conference Room Banquet Hall Promoted Instances [Gupta and Davis, ECCV 2008]

23. Conference Room Seed Examples Iteration 1 Introspection Iteration 40

24. Amphitheatre Seed Images Bootstrapping BA Constraints Our Approach

25. Experimental Evaluation

26. Control Experiments SUN Database : [Xiao et al., CVPR 2010]

27. Control Experiments

28. Control Experiments

29. Control Experiments

30. Control Experiments

31. Control Experiments \\

32. Control Experiments \\ Eigen Functions: [Fergus et al., NIPS 2009]

33. Banquet Hall Seed Images 1 Iterations 10 40

34. Control Experiments SUN Database : [Xiao et al., CVPR 2010]

35. Co-training (Small Scale) SUN Database : [Xiao et al., CVPR 2010]

36. Bedroom Seed Images Iteration-1 Bootstrapping Iteration-60 Iteration-1 Our Approach Iteration-60

37. Scene Classification Eigen Functions: [Fergus et al., NIPS 2009]

38. Co-training (large Scale) • 15 Scene Categories • 25 Seed images / category • Unlabeled Set • 1Million(SUN Database + ImageNet) • >95% distractors Improve 12 out of 15 scene classifiers SUN Database: [Xiao et al., CVPR 2010] ImageNet: [Deng et al., CVPR 2009]

39. Conclusion Labeled Seed Examples Constrained Bootstrapping Bootstrapping • Sharing via Dissimilarities • Constrained Bootstrapping Amphitheatre Auditorium Amphitheatre Amphitheatre Auditorium Auditorium Amphitheatre Auditorium Has Larger Circular Structures

40. banquet hall Banquet bedroom Scene Classifiers Bedroom Unlabeled Data has grass indoor Labeled Data Attribute Classifiers has more space more space larger structures has larger structures Comparative Attribute Classifiers Training Pairwise Data Conference Room Banquet Hall Promoted Instances

42. Unary Binary

43. Eigen Functions

44. Features • 960D GIST • 75D RGB (image is resized to 5x5) • 30D histogram of line lengths • 200D histogram of orientation of lines • 784D 3D-histogram Lab color space (14x14x4) • Total 2049 [Hays and Efros, SIGGRAPH 2007] [Oliva and Torralba, 2006] [Torralba et al., PAMI, 2008]

45. Categories

46. Binary Attribute Relationship

47. Comparative Attribute Relationships

48. Classifiers • Boosted Decision Trees • From [Hoiem et al., IJCV 2007] • Scene • 20 Trees, 8 Nodes • Binary Attribute • 40 Trees, 8 Nodes • Comparative Attribute • 20 Trees, 4 Nodes • Differential features as in [Gupta and Davis, ECCV 2008]

49. Supervision Active Learning Supervised [Torralba et al., PAMI 2008]

50. Supervision Semi-Supervised Active Learning Unsupervised Supervised Graph-based Methods Bus Train