1 / 29

A Cognitive Vision Platform for Semantic Image Understanding

A Cognitive Vision Platform for Semantic Image Understanding. Monique THONNAT and Celine HUDELOT Orion team INRIA Sophia Antipolis FRANCE. Overview. Introduction Cognitive Vision Platform Application to Plant Disease Recognition Conclusion. Problem:

nile
Download Presentation

A Cognitive Vision Platform for Semantic Image Understanding

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Cognitive Vision Platformfor Semantic Image Understanding Monique THONNAT and Celine HUDELOT Orion team INRIA Sophia Antipolis FRANCE

  2. Overview • Introduction • Cognitive Vision Platform • Application to Plant Disease Recognition • Conclusion M. Thonnat

  3. Problem: • What does it mean to perform image understanding? • semantic image understanding (e.g. object classification) • What does it mean to associate semantics to a particular image ? Introduction: the Problem M. Thonnat

  4. Introduction • Different interpretations of this image are possible: • A light object on a dark background • An astronomical object • NGC4473 galaxy Image semantics is not inside the image Image interpretation depends on a priori knowledge M. Thonnat

  5. Introduction • Focus: • complex natural objects with existing taxonomy • Proposed approach: Knowledge-based Vision • formalize the a priori knowledge for image interpretation in knowledge bases • explicit the reasoning (how to use a priori knowledge) for each subtask of image interpretation • propose a platform reusable for different applications M. Thonnat

  6. Cognitive Vision Platform • A platform performing 3 subtasks • Semantic data interpretation • Application expert knowledge (domain taxonomy and terminology) • Visual data management • Matching between numerical image data and symbols • Scene analysis using spatial reasoning • Image processing • numerical object description • program supervision techniques : to automate the management of an image processing library M. Thonnat

  7. Cognitive Vision Platform • For each task: • An application independent engine • A conceptual model for the knowledge • Two ontologies for the interoperability between the different components: • Visual Concept Ontology: spatial, color and texture concepts • Image Processing Ontology: image data and image processing functionality concepts M. Thonnat

  8. Cognitive Vision Platform M. Thonnat

  9. Cognitive Vision: Semantic Interpretation • Goal: • Find the semantic class of physical objects or situations observed on images • How: • Perform the interpretation in the same way experts do: • Use a priori knowledge of application domain terminology and taxonomy • Top down strategy M. Thonnat

  10. Cognitive Vision: Interpretation • Knowledge Formalization: • Declarative knowledge: • Domain class: application concept (plantleaf, pollen grain) described by visual concepts(green color and oval shape or pink and circular)and subparts • Domain class tree : hierarchy of domain classes • Context: explicit representation of current domain context and acquisition context • Domain request: request of an end user • Representation by frames with slots M. Thonnat

  11. Cognitive Vision: Interpretation • Knowledge Formalization Domain Class name White_Fly SuperClass Insect SubPart Description Domain Class Fly_Body Domain Class 2 Fly_Antenna Sub-part Domain Class name Fly_Antenna Visual Description ST_VisualConceptShape[line] Thickness [thin] Color_VisualConceptHue [white] Spatial_RelationConnected [Fly_Body] Spatial_RelationRight_of [Fly_Body] Domain Class name Fly_Body Visual Description ST_VisualConceptShape[oval] Elongation [important] Color_VisualConceptHue [white] M. Thonnat

  12. Cognitive Vision: Interpretation • Knowledge Formalization: • Inferential knowledge: • Context criteria: describe decisions during the semantic interpretation • Initialization interpretation criteria: information on how to initialize the problem using the context • Post interpretation criteria: information to refine the interpretation results according to the context • Implemented by rules • Exemple of post interpretation criteria If Powdery Mildew detected and temperature < 25 C and Humidity > 80% then Alert “treatment is needed” M. Thonnat

  13. Cognitive Vision: Interpretation • Reasoning • Depth-first domain class tree traversal • Visual object hypothesis propagation by building visual data management requests (visual object instance finding) • Matching between visual object instances and predefined domain classes • Classification refinement M. Thonnat

  14. Cognitive Vision: Visual Data Management • Goal: • Matchingbetween symbols and sensor data • How: • Data management, spatial reasoning, top down and bottom up strategies • Symbol grounding or Anchoring: • Anchoring = « Problem of connecting, inside an artificial system, symbols and sensor data that refer to the same physical objects in the external world » [coradeschi99] M. Thonnat

  15. Cognitive Vision: Visual Data Management • Knowledge Formalization: • Declarative knowledge: • Visual concepts(symbolic data): description of visual concepts and of their grounding relation with image descriptors • Image data concepts (sensor data): primitives (ridge, region, edge), descriptors (area, eccentricity) • Spatial relations : topology (RCC8), distance and orientation • Visual data management requests : express the visual data management problem • Represented by frames with slots M. Thonnat

  16. Cognitive Vision: Data Management • Knowledge Formalization: • Inferential knowledge: • Object extraction criteria: how to constrain image processing requests (using visual concepts and spatial relations) • Spatial deduction criteria: how to infer spatial relations from another ones • to diagnose the image processing results • Visual evaluation criteria: how to diagnose image processing results • Implemented by rules M. Thonnat

  17. Cognitive Vision: Data Management • Knowledge Formalization: • Example of object extraction criteria Let c a visual content context and O a visual object If O.geometry is an Open Curve and O.thickness is {Thin, Very Thin} then c.ImageEntityType:=Curvilinear Structure “Ridge or Valley” • Example of spatial deduction criteria Let O1, O2, O3 three visual objects If NTTP(O1, O2) is true and Left_Of(O2,O3) is true then Left_Of(O1,O3) is true • Example of visual evaluation criteria If mode is interactive then assess_data_by _user [correct under_segmentation over_segmentation noisy] M. Thonnat

  18. Cognitive Vision: Visual Data Management • Reasoning • Image processing request building according to visual object hypotheses (Object extraction criteria) • Matchingbetween image processing results and symbolic data • Instantiation and sending of visual objects to the Interpretation task • Spatial Reasoning: multiple objects (spatial deduction criteria) M. Thonnat

  19. Cognitive Vision: Image Processing Goal : Objectextraction and numerical description How: Use ofprogram supervision techniques: • Dynamic configuration and execution of a library of image processing programs (versus fixed procedure) • Explicit formalization of expertise on how to use programs M. Thonnat

  20. Cognitive Vision: Image Processing • Knowledge formalization: • Declarative knowledge: • Goals: image processing functionality (thresholding, edge extraction,…) • Operators: knowledge to solve a given problem: • primitive: particular program • composite: particular combination of programs • Program supervision requests: instantiations of goals on particular data, under particular constraints M. Thonnat

  21. Cognitive Vision: Image Processing • Knowledge Formalization Primitive Operator Name Recursive_Gaussian_Derivation Input data Image name input Output data Image name mfxx Image name mfyy Image name mfxy Parameters sigma default 1.0 Preconditions valid input Postconditions valid mfxx, valid mfxy Initialization Criteria Rule init-sigma Let c a visual content context If true Then sigma := c.objectwidth/sqrt(3) Calling syntax: Gaussian -sigma input mfxx mfyy mfxy Composite Operator Name Ridge_Extraction Functionality Object Extraction Input data image input_image Output data image segmented_image Preconditions valid input Postconditions valid mfxx Body “sequential decomposition” Recursive_Gaussian_derivation – Steger_Detector-Ridge-Filtering Distribution Ridge_Extraction.input_image / Recursive_Gaussian_Derivation.input … Flow Recursive_Gaussian_Derivation.mfxx / Steger_detector.mfxx M. Thonnat

  22. Cognitive Vision: Image Processing • Reasoning: Planning techniques (HTN) • Program selection in a library of programs • Selected programs execution • Evaluation and adjustment if needed Request + data Program Supervision Engine Program Utilisation KB 7 2 1 Planning Execution plan (part of) results correct 4 6 5 3 Library of programs Repair Evaluation Actions to correct judgements incorrect M. Thonnat

  23. Cognitive Vision Platform M. Thonnat

  24. Application on plant disease diagnosis • Why Image Understanding ? • Plant disease diagnosis = visual observation which aims at inferringdisease presence by the observation of signs and symptoms TO BE ABLE TO REASON :signs and symptoms interpretation in terms of diseases TO BE ABLE TO SEE :Focusing on relevant criteria Star shape network of white and thin filaments (5-10 μ) Presence of elliptical white blobs in the centre of the network Climatic Context: High humidity, Temperature : 25 °C Early powdery mildew infection in propitious conditions Early diagnosis: Microscopic image (x64) of rose leaf part M. Thonnat

  25. Powdery mildew : State of infection : early Vegetal support : red leaf Powdery mildew : State of infection : very early Vegetal support : green leaf Two white flies close to their eggs Complexity and variability of object appearance Variability of contexts Multiple objects and various object types Scene knowledge and spatial reasoning Intelligent management of image processing programs Need of domain knowledge Application on Plant Disease Diagnosis: Rose Diseases M. Thonnat

  26. Healthy White fly • Mycelium: • Part of : Fungi • network of at least 2 connected Hyphae • nb_hyphae = {unknown} • Hyphae: • Part of : Mycelium • Geometry: line • Thickness: thin, very thin • Straightness:=almost straight • Luminosity=bright • ... Subclass Leaf Insects Acarid Ungerminated Non Healthy Aphid Virus Subpart Germinated tubes red Powdery mildew Vegetal tissue Fungi Filamentous green Veins Penicillium Pellets Application on plant disease diagnosis • Domain knowledge base : the class tree M. Thonnat

  27. Input : User Request Fungi infection? Variety : Leonidas Leaf : young Season: summer Temp: 24° C Humidity: 80 ... Interpretation Domain concept tree traversal to build visual object hypotheses Interpretation Domain concept tree traversal to build visual object hypotheses White fly VegetalPart 2 1 1 Image + Context Subpart Acarid Insects Very Early Visual Object Hypothesis Aphid Subclass Leaf Scene Virus Group of : Geometry: star shape network of { Geometry: line Thickness : thin width [7..10 m] very thin width [5..7 m] Straightness : almost straight Lightness: bright} Spatial Relation: Connected} Powdery mildew Dispersed Disease Fungi Clump Penicillium Pellet 3 Data Management Symbolic request toimage processing request 4 Image Processing Request • Goal: segmentation • Contraints:Image entity = ridge • Object.width = [1..3] • Object.intensity > 150 • Input Data: Image : input image • Mask : area of interest 5 Ridge 1 numerical descriptors Image Data Ridge 2 + Numerical Descriptors Ridge 3 + Numerical descriptors Image Processing: request solving by program supervision techniques 6 Application : early detection of plant diseases M. Thonnat

  28. Interpretation : diagnosis Post classification rules activation Recognised domain concept Diagnosis Early powdery mildew infection on young leaf Freely dispersed mycelium 11 12 10 Interpretation : classification Matching between visual object instances and domain concepts 9 8 Network of lines Visual Object Instance Data Management Visual object hypothesis verification and instantiation Line 2 Line 3 Line line1 Thickness:=thin (0.8) Straightness:= straight (0.5) Lightness:=bright (0.7) Connected (line2) Connected (line4) + link to image data EC Ridge 1 numerical descriptors Image Data EC Ridge 2 + Numerical Descriptors Ridge 3 + Numerical descriptors Line 1 EC Line 5 Line 4 EC 7 Application : early detection of plant diseases M. Thonnat

  29. Conclusion • A platform for automatic recognition of natural objects • Ontology-based formalism for knowledge acquisition • 3 dedicated reusable engines for • semantic interpretation • visual data management • program supervision for image processing • Future works • integrate results on : • machine learning for visual concept detection (Nicolas Maillot) • machine learning for image segmentation (Vincent Martin) M. Thonnat

More Related