Attribute-augmented Semantic Hierarchy

1. Towards Bridging Semantic Gap and Intention Gap in Image Retrieval Attribute-augmented Semantic Hierarchy Hanwang Zhang1, Zheng-Jun Zha2, Yang Yang1, Shuicheng Yan1, Yue Gao1, Tat-Seng Chua1 1: National University of Singapore 2: Institute of Intelligent Machines, Chinese Academy of Sciences

2. What happened? Data Search Engine Query User Large-scale Unstructured

3. What happened? SemanticGap Data Search Engine Query User IntentionGap

4. Bridging Semantic Gap High-level Semantic SemanticGapBridged? ontological No! semantic Low-level Visual Feature

5. Bridging Intention Gap User Intention IntentionGapBridged? No! Low-level Visual Feature

6. Challenges Semantics Search Intent Low-level Feature 5/33

7. Solution: Attributes Semantics Search Intent Attributes Low-level Feature 6/33

8. Attributes Component snout, ear, etc Appearance furry, brown, etc cat or dog?etc Discriminability

9. Solution: Attribute-augmented Semantic Hierarchy (A2SH) 1 Root • Semantichierarchy • Poolofattributes • Conceptclassifiers • Attributeclassifiers Animal Vehicle 2 metal head • Hierarchical Semantics • Hierarchical Semantic • Similarity Cat Dog Root Animal Dog Pug wheel wet leg glass furry shiny brown Corgi Pug General framework for Content-based Image Retrieval

10. A Prototype ofA2SH ILSVRC2012ImageNet Concepts: 1322 (958 leaves) Depth: 3 ~ 11 Images: 1.23 million 50% training 50% testing Tail Leg • 95,800imagesaremanually labeledwith33attributes • Automaticallydiscovered2-26attributesforeachconceptnode • 15 ~ 58 attributes per concept

11. Why A2SH? • Attributes bridge the semantic gap 1 concept Smaller Variance attribute 2 glass Descriptive,Transferrable wing wheel

12. Why A2SH? • A2SH well defines attributes more informative Which “Wing”?

13. Why A2SH? • A2SHbridges the intention gap 1 Intention as attributes throughattributeandimagefeedbacks LegSkin Attribute Feedback Image Feedback 2 Feedbacksareautomaticallydigested into multiple levels Leg Tail

14. Demo on A2SH

15. How A2SH: System Overview

16. How A2SH: Off-line

17. Concept Classifiers predicts whether an image belongs to concept c C

18. Concept Classifiers predicts whether an image belongs to concept c _ hierarchicalonev.s.all + _ c + • Exploit hierarchical relation • Alleviate error propagation + +

19. Attribute Classifiers predicts the presence of an attribute a of concept c • Nameable attributes: • human nameable, hierarchical supervised learning • Unnameable attributes: • human unnameable, hierarchicalunsupervisedlearning • They together offer a comprehensive description of the multiple facets of a concept

20. Unnameable Attribute Classifiers • Nameable attributes are not discriminative enough. • Discover new attributes for concepts that share many nameable attributes. • 2-26 for each concept. Ear Snout Eye Furry D. Parikh, K. Graman. “Interactively Building a Discriminative Vocabulary of Nameable Attributes”, CVPR 2011.

21. What we have now? • Concept classifiers • Semantic path prediction • Attribute classifiers • Imagerepresentation along the semantic path Hierarchical Semantic Representation 20/33

22. Hierarchical Semantic Similarity Images are represented by attributes in the context of concepts Hierarchical semantic similarity

23. LocalSemantic Metric Same concept  close, different concepts  far

24. What we have now? • Concept classifiers • Semantic path prediction • Attribute classifiers • Imagerepresentation along the semantic path • Hierarchical Semantic Similarity Function • Semantic similarity between images Hierarchical Semantic Representation 23/33

25. How A2SH: On-line

26. Automatic Retrieval Hierarchical semantic similarity Ic c child(c) Candidate images are retrieved by semantic indexing Lowcomplexity! Efficient! candidate images

27. Evaluation • A2SH: our method • hBilinear: retrieves images by bilinear semantic metric (Deng et al. 2011 CVPR) • hPath: length (confidence) of the common semantic path of an image and the query • hVisual: hPath+visual similarity • fSemantic: flat semantic feature similarity • fVisual: visual feature similarity Training: 50%, Gallery: 50% (95, 800 queries)

28. Evaluation: AutomaticRetrieval Effective! Efficient!

29. Case Study AutomaticRetrieval matched semantically similar fVisual hBilinear A2SH

30. Interactive Retrieval • Image-level Feedback Query

31. Interactive Retrieval • Attribute-level Feedback Query Leg Cloth Zhang et al. “Attribute Feedback”, MM 2012

32. Evaluation: InteractiveRetrieval 2-min fixed time

33. Case Study InteractiveRetrieval initial matched semantically similar QPM HF A2SH

34. Summary Attribute-augmented Semantic Hierarchy A2SH SH with Attributes Framework for CBIR Effectiveness Verified Gaps bridging 1.23 M Images 33/33

35. & Q A ? !

36. Nameable Attribute Classifiers selected base

37. Unnameable Attribute Classifiers confusion matrix

38. Unnameable Attribute Classifiers confusion matrix

39. Data Set • Only leaves have images and each concept’s images are merged bottom-top • 50% to 50% training and testing (gallery) • 100 random images per leaf from testing are used as queries • 100 random images from each leaf’s training images are annotated with attributes • Color, texture, edge and multi-scale dense SIFT. LLC with max-pooling, 2-level spatial pyramid. 35,903-d feature vector

40. 0.93 Concept Classifiers

41. Nameable Attributes Classifiers 0.92 0.77

42. Nameable Attributes Classifiers

43. Unnameable Attributes Classifiers

44. Unnameable Attributes Classifiers

45. Local Metric Learning

46. Automatic Retrieval

47. Interactive