GNORASI workshop

1. Reasoning and semantic interpretation of visual data in GNORASI GNORASI workshop Knowledge and processing algorithms for remote sensing data Charalampos Doulaverakis CERTH/ITI

2. Goals of semantic interpretation • Reasoning and representation of knowledge for semantic-enabled image analysis • Expert knowledge and visual information processing data are represented through ontologies • Development of a reasoning process for the knowledge assisted interpretation of images • Reasoning methods • Fuzzy inference support

3. Ontologies and reasoning Definition on ontology Representation and querying Land use/Land cover ontologies

4. Ontology definition • An ontology defines a set of representational primitives with which to model a domain of knowledge or discourse (Gruber, 1995) • Classes:which represent a set of objects • Properties: which express attributes and relations between classes and objects • Constraints: for expressing logical consistency • Individuals: atoms (objects) which are members of a class • Abstraction level of data models • Analogous to hierarchical and relational models • Ontologies, through inference, can provide us with implicitly defined information • Reasoning engines Thomas R. Gruber. Toward Principles for the Design of Ontologies Used for Knowledge Sharing. International Journal of Human-Computer Studies, vol. 43, no. 5-6, pages 907-928, 1995

5. Ontology languages • Several languages have been proposed • RDF(S) • RDF: Uses URI for expressing relationships between objects (triples) • RDF: Allows structured and semi-structured data to be mixed, exposed, and shared across different applications. • RDFS: Allows the definition of classes, the relations between them and semantic constraints on RDF • OWL • Based on Description Logics. • Designed to represent rich and complex knowledge • 3 types of increasing expressivity • Lite, DL, Full • OWL2 • Logical extension of OWL • Deals with weaknesses in OWL expression and integrates features requested by users • 3 variations • EL, QL, RL

6. Rule languages • They add expressive extensions to ontology languages • E.g: SandArea(?x), SeaArea(?y), isAdjacent(?x,?y)-> Beach(?x) • Standard languages have been defined • RuleML, SWRL • Other rule languages are offered by reasoning engines • Such as Jena, OWLIM, Pellet, Hermit, …

7. Query languages • Query languages have been proposed for retrieving information from ontology repositories • SPARQL, SeRQL, RDQL • Most have similarities with SQL • SELECT * WHERE {?X rdf:typegn:LandCover} • Retrieve all instances of class gn:LandCover(SPARQL) • Query languages can be used for deriving new facts • CONSTRUCT {?region gn:depictsgn:Vegetation} WHERE { ?region gn:hasNDVI ?value . FILTER (?value > 0.5) } • A type of rule expression • Such approaches are proposed e.g. in SPIN (SPARQL Inference Notation)

8. Land Use/Land Cover systems • Available land classifications correspond to specific applications, e.g. crop or vegetation characterization • As such, they cannot be used for generic telesensing applications • Most important of them are • CORINE • Organized in a 3 level hierarchy: 5 categories of the 1st level are broken down to 15 categories on the 2nd level which in turn are broken down to 44 3rd level categories • Land Cover Classification System (LCCS) • It doesn’t specify predefined land cover categories, it rather defines general classification criteria for characterizing land covers

9. GNORASI classification Challenges Solutions Usage examples Development details

10. Challenges • Object-based image analysis produces a large number of objects (thousands) • Probabilistic inference for class membership • Classification is based on user-defined rules • Feature-based • Geospatial restrictions • Classes can appear as premises in rules

11. Solutions • Numerical computations are executed outside the ontological framework • Fuzzy membership values are computed using membership functions • Ontological inference is used for the assignment of objects to classes • According to user rules • SPARQL Update and GeoSPARQL are used to define the rules in ontological terms • Development decision • The ontological classification is implemented as an external web service

12. Rule definition UI Class ruleset Hierarchy Rule definition

13. Fuzzy values • The outcome of the rule definition processor are the fuzzy values of all objects for the features present in the rules • These arithmetic values have to be assigned to semantic entities, i.e. the defined classes

14. Ontology data • The fuzzy values along with the class hierarchy and rule definitions are sent to the ontology classification service • The following are performed by the service • The class hierarchy is added to the core ontology • The fuzzy values are transformed to ontological data properties • Rules are translated to SPARQL Update queries • Rules are iteratively executed

15. Example SPARQL Update • Example queries for assigning an object to classes Sidewalk and Vegetation with confidences ?conf1 and ?conf2 Sidewalk INSERT {?object gn:depicts gn:depiction1. gn:depiction1 gn:depictsClassgn:Sidewalk. gn:depiction1 gn:withConfidence?conf. } WHERE {?object rdf:typegn:Object. ?object gn:fuzzyNDVIMean?conf. } Vegetation INSERT {?object gn:depicts gn:depiction2. gn:depiction2 gn:depictsClassgn:Vegetation. gn:depiction2 gn:withConfidence?conf2. } WHERE {?object rdf:typegn:Object. ?object gn:fuzzyNDVIMean?conf2. } INSERT {?object gn:depicts gn:depiction1. gn:depiction1 gn:depictsClassgn:Sidewalk. gn:depiction1 gn:withConfidence?conf. } WHERE {?object rdf:typegn:Object . ?object gn:fuzzyNDVIMean?conf. ?filterObjectrdf:typegn:Object. ?filterObjectgn:depicts ?gn:filterDepiction. ?filterDepictiongn:depictsClassgn:Road FILTER (geof:sfTouches(?object, ?filterObject)) } Sidewalk rule with geospatial restriction (adjacent to Road)

16. Classification • In the example, Sidewalk depends on Road definition. Objects assigned to Road must exist • Iterative rule execution until convergence (no changes in the repository) • In the end, every object will be assigned to classes with different membership values INSERT {?object gn:depicts gn:depiction1. gn:depiction1 gn:depictsClassgn:Sidewalk. gn:depiction1 gn:withConfidence ?conf. } WHERE { ?object rdf:typegn:Object. ?object gn:fuzzyNDVIMean ?conf. ?filterObjectrdf:typegn:Object. ?filterObjectgn:depicts ?gn:filterDepiction. ?filterDepictiongn:depictsClassgn:Road FILTER (geof:touches(?object, ?filterObject)) }

17. Classification • In a hierarchy, objects will try to match the deepest classes

18. Classification • Object are assigned to classes with the highest membership value • A minimum threshold is applied • The service returns a list [<object id> <class id> <confidence>]* • This is the classification result

19. Development • Java based • REST communication • Server • Grizzly project1 • Reasoner employed • OpenRDF Sesame2 backend, OWLIM-lite3 reasoner • Geospatial library • uSeekM IndexingSail4 1Grizzy, https://grizzly.java.net/ 2OpenRDF Sesame, http://www.openrdf.org 3 OWLIM Lite, http://www.ontotext.com/owlim 4IndexingSail, https://dev.opensahara.com/projects/useekm/wiki/IndexingSail

20. Concluding • Efficient ontology-based classification • Employ both feature-based classification and geospatial restrictions • Handling of fuzzy membership values • Built as a web service

21. Thank you! Questions?