Why Categorize in Computer Vision?

1. Why Categorize in Computer Vision?

2. Why Use Categories? People love categories!

3. Why Use Categories? What if we didn’t have categories? Humuhumunukunukuapua'a – “fish that grunts like a pig”

4. Why Use Categories? Our minds work very intimately with categories • Every common noun in English is a category • Proper nouns name object instances • “this,” “that,” “the,” “my,” “yours,” etc. refer to object instances anonymously

5. The Categorization Problem

6. The Categorization Problem Categorization/Classification: Given a set of pre-defined categories, “bin” this image Does not necessarily require object detection Vertical Dimension: • General: “Animal” • Basic: “Bird” • Specific: “Robin”

7. The Categorization Problem What kinds of categorization are computers good at? • Basic -- especially when using context clues • Specific -- due to low intra-class variation

8. The Categorization Problem Bad at? • General, due to high intra-class variation and a lack of visual cues

9. The Categorization Problem Bad at? • Categories defined by non-visual characteristics (like chairs)

10. Summary • Semantic categories allow humans to convey a large amount of information concisely • We want computers to be able to do the same • What work has been done on this problem? Has it been successful?

11. Uses of Categorization

12. Two Examples • Using Context in Categorization • Fine-Grain Object Classification

13. Caltech 101 (2003) • Dataset for basic-level categorization • Objects from 101 classes • Famously difficult

14. Categorization with Context Goal: Resolve ambiguity between similar-looking objects of different classes using the semantic context of an object Rabinovich et al. (UC San Diego): Objects in Context First paper to attempt to use context at the object level PASCAL 2007 dataset

15. Categorization with Context

16. Categorization with Context Approach • Segment image to preserve some spatial data • Perform Bag-of-Features to give an initial ranked list of labels for each segment • Use a Conditional Random Field (CRF) framework to find agreement between segment labels

17. Categorization with Context

18. Bag-of-Features with Segmentation Labeling Segments: Confidence:

19. Conditional Random Field Way to assign joint probabilities to elements without considering every possible combination in the training set

20. Conditional Random Field Idea • Given set of segments S, set of labels C • Want to find p(C | S) without knowing p(S) • Associate a special graph with C that obeys the “Markov Property” (uses S) • The ordered pair (S, C) is a CRF conditioned on S

21. Conditional Random Field

22. Results

23. Results False correction

24. Fine-Grain Classification

25. Fine-Grain Image Categorization Challenge: need good classifiers that capture detail well

26. Fine-Grain Image Categorization Yao et al. (Stanford): Combining Randomization and Discrimination for Fine-Grained Image Categorization Approach Random forest with discriminative classifiers This is a kind of machine learning framework that allows us to handle the fine detail in this problem.

27. Fine-Grain Image Categorization

28. Random Discriminative Tree Approach • For each tree node, train an SVM classifier for a randomly sampled image region • At each node, make a yes-or-no decision • Uses grayscale SIFT descriptors

29. Random Discriminative Tree

30. Results

31. Conclusion • Semantic categories allow humans to convey a large amount of information concisely • Categorization has been used for basic-level object detection and scene recognition • Fine-grain categorization can provide us with expert-level classification of objects • Not all categories are defined by visual characteristics!

32. Questions?