Building text features for object image classification

1. Building text features for object image classification Group 1: Eddie Sun, Youngbum Kim, Yulong Wang

2. Which object is presented？

3. Why we need text features?

4. Main idea & Insights • Main idea • Determine which objects are present in an image based on the text that surrounds similar images. • Insights • First, it is often easier to determine the image content using surrounding text than with currently available image features. • Given a large enough dataset, we are bound to find very similar images to an input image, even when matching with simple image features.

5. Illustration for building text features Internet Images with text Text Features

6. Framework of the approach Texts of These Similar Images Training Process K Most Similar Images Visual Features: SIFT, Gist, Color, Gradient and Unified of all previous one

7. Experiment • Dataset • The PASCAL Visual Object Classes Challenge

8. Experiment • Features • SIFT • Gist • an abstract representation of the scene that spontaneously activates memory representations of scene categories (a city, a mountain, etc.) • Color • Color Features in the RGB space • Gradient • Unified • a concatenation of the above four features

9. Experiment

10. Experiment

11. Experiment

13. Experiment

14. Experiment

15. Summary How it works Results

16. How it works? Return most similar images with their labels Internet images dataset with text • SIFT • Gist • Color • Gradient • Unified Get similar images based on visual features Construct text features from labels Extract visual features Cute, puppy, canine Dog cool dogs, boxer Input Image 1. Training images 2. Test images Visual features Dog Visual Classifier Puppy Dog, pet, animal Text features Text Classifier Learn parameters on training images Merge • Notes • Unified Feature – weighted average of the above 4 features • Text features – normalized histogram of tags counts Fusion Classifier Dog Final Output

17. Results • Text features are built from visual features. Better visual features -> better text features • Combining visual and text classifiers Visual and text classifiers correct each other • Number of training images Small number of training images -> text classifiers outperform visual classifiers Combine -> always better • Number of Internet images in dataset 200,000 -> 600,000 : Big improvement 600,000 -> 1 million : very small improvement

18. Questions?

19. Thank you!