Symbol Grounding for Semantic Image Interpretation: From Image Data to Semantics

1. Symbol Grounding for Semantic Image Interpretation:From Image Data to Semantics Céline Hudelot, Monique Thonnat and Nicolas Maillot INRIA Sophia Antipolis, FRANCE Workshop on Semantic Knowledge in Computer Vision, ICCV 2005

2. Outline • Introduction • Symbol Grounding Problem • Ontology-based Communication • Learning Approach • Knowledge-based Approach • A Symbol Grounding Engine • Conclusion SKCV

3. Introduction Problem: • What does it means to perform semantic image interpretation ? • What does it means to associate semantics to a particular image ? SKCV

4. Introduction Different interpretations are possible • Image semantics is not inside the image • Image interpretation depends on a priori knowledge and on the context • A white object on a green background • An insect • An infection of white flies on a rose leaf SKCV

5. Semantic level Visual level A circular shape, orange hue and regular granulated texture PLATE_OF_FRUITS Image level Composition link Region 1: Area : 105 compactness :0.9 Circularity : 0.85 HSV (0.05,0.2, 0.6) ... FRUIT Specialization link APPLE PEAR ORANGE PEACH STEM Introduction • Three abstraction levels of data • Vision [Marr,82], Cognitive Science [Gardenfors,2000] SKCV

6. Region 1: Area : 105 compactness :0.9 Circularity : 0.85 HSV (0.05,0.2, 0.6) ... FEATURE EXTRACTION SEGMENTATION Introduction • Three sub-problems: • Image processing : extraction of numerical image data • Symbol grounding : mapping between image data and high level representations of semantic concepts Region 1: Area : 105 compactness :0.9 Circularity : 0.85 HSV (0.05,0.2, 0.6) ... Orange Fruit : Has for shape : circular Has for hue: orange Has for texture : granulated Symbol grounding • Semantic interpretation : reasoning at the high level SKCV

7. Area : 105 compactness :0.9 Circularity : 0.85 HSV (0.05,0.2, 0.6) ... The Symbol Grounding Problem • Definition: • Problem of the mapping between image data and semantic data • Objective • Propose generic tools to solve the symbol grounding problem as a problem as such The Orange Fruit SKCV

8. The Symbol Grounding Problem • Proposed Approach • An independent intermediate level called visual level between the semantic level and the image level • Two ontologies to make easier the communication between the different levels • Visual concept ontology • Image processing ontology • A cognitive vision approach involving a priori knowledge and machine learning SKCV

9. Semantic level Orange Visual concept ontology Visual level A circular shape, orange hue and regular granulated texture Image processing ontology Image level Region 1: Area : 105 compactness :0.9 Circularity : 0.85 HSV (0.05,0.2, 0.6) ... The Symbol Grounding Problem • Proposed Approach Symbol grounding problem : matching image data with combination of visual concepts SKCV

10. The Symbol Grounding Problem • Proposed Approach • Build the correspondence links between images features and visual concepts • Learning approach : the correspondence links are learned from images samples • A priori knowledge based approach: links are built explicitly and stored in a knowledge base • A symbol grounding engine uses these links to perform the matching SKCV

11. Ontology Based Communication • A visual concept ontology [Maillot et al. 03] • Experts often use and share a generic visual vocabulary to describe their domain • A hierarchy of three kinds of 2D visual concepts • Spatial Concepts (64 concepts) • Shape, Size: circular, elongated,… • Spatial Structure : network of, ring of,… • Binary spatial relations : near of, connected to, left of • Color Concepts (37 concepts) : red, light, vivid (ISCC-NBS lexicon) • Texture Concepts (14 concepts) : granulated, regular (cognitive studies [Bhushan,97]) • Application independent • A basis for further extensions SKCV

12. Knowledge Base Domain Expert Knowledge Acquisition Visual Concept Ontology Manually Segmented and Annotated Images Images Samples Ontology Based Communication • Why a Visual Concept Ontology ? • To guide and constrain the semantic knowledge acquisition • Reduce the semantic gap : a shared representation of image content at an intermediate level • Communication between the semantic level and the visual level SKCV

13. Ontology Based Communication • An Image Processing Ontology • Domain of discourse of image processing: set of generic terms to describe images and image processing results • “Images have an ontological description of their own” • Hierarchical set of : • Image entity concepts : region, edge, graph …(11 concepts) • Image feature concepts : eccentricity, RGB values, … (167 concepts) • Image processing functionalities :object_extraction, feature_extraction,… (5 generic functionalities) • Communication level between the image level and the visual level • Not complete, a basis for further extensions SKCV

14. Visual Concept Detectors Positive and negative samples of each visual concept Feature Extraction Feature Selection Training Supervised Learning Approach • Goal: Training a set of detectors (e.g. Multi Layer perceptrons, SVM) to the detection of visual concepts • Each visual concept C is associated to a set of image features FC • Only visual concepts used during the semantic knowledge acquisition phase are learned SKCV

15. Gabor Filter LDA NN circular shape orange hue granulated texture Granulated Texture Detector Positive and negative samples of visual concept  Granulated Texture  Manually segmented and annotated images Supervised Learning Approach • Example : learning of the visual concept granulated texture SKCV

16. Supervised Learning Approach • Reduce the learning problem by addressing it at an intermediate level of semantics • Automatic building of the symbol grounding link between visual concepts C and image features F • Does not learn spatial structure and spatial relations • Dependent on the learning base : a large amount of image samples is needed SKCV

17. A Priori Knowledge Base Approach • Explicit representation with frames: • Visual concepts (symbolic data): description of visual concepts C and of their grounding link with image features F • Image data concepts (image data): primitives (ridge, region, edge), features (area, eccentricity) organized in feature sets • Spatial relations : topology (RCC8), distance and orientation • Explicit representation with rules: • Object extraction criteria: to constrain image processing • Spatial deduction criteria: to infer spatial relations SKCV

18. A Priori Knowledge Based Approach • Visual concept : simple examples VisualConcept{ name Circular_Surface Super Concept Elliptical_Surface Grounding Link Symbolnameeccentricity Commentratio of the length of the longest chord to the longest chord perpendicular to it Linguistic-values[ high very_high] FuzzySet Fhigh ={0.57, 0.62, 0.76, 0.84} Fvery_high ={0.76, 0.84, 1, 1} Domain[0 1] Symbolnamecompactness Linguistic-values[ high very_high] …} VisualConcept{ name Orange Super Concept Hue Grounding Link FloatnameH_value Domain[0.0 0.1] FloatnameL_value Domain[0.5 1.0] } SKCV

19. A Priori Knowledge Base Approach • Explicit representation of spatial relations [Le Ber, 98] : distance, orientation, topology (Binary, 2D) Spatial Relation{ name Externally_Connected Super Relation Discrete Inverse Externally_Connected Complement None Symmetry true Conditions Intersection(Interior(O1), Interior(O2))=Ø Intersection(Boundary(O1),Boundary(O2))!=Ø Objects_In_Relation VisualObject name O1 VisualObject name O2} Spatial Relation{ name Near_of Super Relation DistanceRelation Inverse Near_Of Complement Far_From Symmetry true Float name distance_threshold Conditions Distance(O1,O2) < distance_threshold Objects_In_Relation VisualObject name O1 VisualObject name O2} SKCV

20. O2 O3 O1 A Priori Knowledge Base Approach Object extraction criteria: how to constrain image processing(using visual concepts and spatial relations) Example Spatial deduction criteria: how to infer spatial relations from other ones Example: Rule { Let c a visual content context and O a visual object If O.geometry is a Open Curve and O.width is {Thin, Very Thin} then c.ImageEntityType:=Curvilinear Structure } Rule { Let O1, O2, O3 three visual objects If NTTP(O1, O2) is true and Left_Of(O2,O3) is true then Left_Of(O1,O3) is true} SKCV

21. A Priori Knowledge Base Approach • Reduce the learning problem by addressing it at an intermediate level of semantics • No need of image samples • Spatial relations are explicit • Manual building of the symbol grounding links between visual concepts C and image features F • Difficult to express some criteria for texture SKCV

22. Symbol Grounding Engine • Symbol Grounding (Symbols, Image) • Image processing request building using object extraction criteria • Primitive selection (region, ridge,…) • Feature extraction • Matchingbetween image processing results (image features F) and symbolic data (visual concepts C) • Fuzzy Matching using explicit knowledge (Frames) • OR, Matching using the detectors obtained during the learning • Spatial Reasoning formultiple objects management using spatial deduction criteria and spatial relations SKCV

23. Conclusion • The two methods have been tested on real world applications • A priori knowledge based approach : • Automatic early diagnosis of rose disease [Hudelot et al 03] • Supervised learning approach : • Application on aircraft/cars retrieval [Maillot et al 05] • Two complementary methods • The symbol grounding link is difficult to build explicitly by a human expert in vision (e.g. texture concepts) • A large amount of data (image examples) is not available for all the applications SKCV

24. Conclusion • Original Symbol Grounding Approach: • Ontology-based Approach • Visual concept ontology and Image processing ontology • Independence between application domain semantics and image processing library • Symbol grounding link • Either learned from samples or a priori knowledge • Future works • Learning for spatial relations • Extension of the visual concept ontology • Temporal concepts SKCV

25. Symbol Grounding for Semantic Image Interpretation: From Image Data to Semantics Any Questions?? SKCV

26. The Symbol Grounding Problem • Related Works • Knowledge based Vision: • Not often considered as a problem as such • Encapsulated in the semantic level • Intermediate Symbolic Representation [Brolio,89] • VISIONS system [Hanson,78] • Database management technology • Conceptual Spaces [Chella, 1997] • Conceptual space = metric space which dimensions are entity qualities • Natural concepts = convex regions in the conceptual space SKCV

27. The Symbol Grounding Problem • Related Works • Artificial intelligence : the Symbol grounding problem [Harnad, 90] • Robotics community: the Anchoring problem • « Problem of connecting, inside an artificial system, symbols and sensor data that refer to the same physical objects in the external world » [coradeschi99] • Image retrieval community : the semantic gap • Use of ontological engineering: object ontology [Mezaris, 04] , visual ontology [Mao,98],ontology for language based querying [Town, 04] SKCV

28. Ontology Based Communication • Ontology : set of concepts and relations useful to describe a domain • “A formal, explicit specification of a shared conceptualization” [Gruber, 93] • Conceptualization : abstract relevant model of a phenomenon • Explicit: the meaning of the concepts is defined explicitly • Formal: machine readable • Shared: consensual knowledge accepted by a group SKCV