Drago ş Br ă t ăş anu *, Corina V ă duva **, Inge G ă vat**, Mihai Datcu***

1. Latent Knowledge Discovery in Satellite Images Dragoş Brătăşanu*, Corina Văduva**, Inge Găvat**, Mihai Datcu*** * Romanian Space Agency, ROSA ** University Politehnica of Bucharest, UPB *** German Aerospace Center, DLR ESA-EUSC-JRC 2011 31 March 2011, Ispra – Varese, Italy Image Information Mining: Geospatial Intelligence from Earth Observation

2. Earth Observation for Cartography State of the art method Urban Atlas Satellite image Data Mining Algorithms Multi-temporal satellite image GIS data , maps Knowledge

3. Earth Observation for Cartography Image classification Satellite image Data Mining Algorithms Topology and geometry Topological and geometrical knowledge

4. Application for Very High Resolution Images GeoEye image, New Town, Tasmania, Australia White building Industrial area White building Sports center

5. Outline • Hierarchical Representation of Image Content: • hierarchical structuring for image content; • hierarchical analyzing for the image content. • Latent Dirichlet Allocation Model • generative probabilistic model; • bag of words assumption; • text – image analogy. • Pixel & Patch Level Analysis – Semantic Rules Discovery: • methodology for feature extraction; • case study – experimental results. • Scene Level Analysis – Semantic Rules Extraction Based on Spatial Positioning of Objects: • methodology for feature extraction; • case study – experimental results. • Conclusions

6. Hierarchical Representation of Image Content • Access to geospatial information should support a wide variety of EO data users • => necessity of a common language – latent semantics hidden in the image. • Add ontology to the satellite data and bridge the semantic gap between science and politics. Hierarchical structuring for image content Hierarchical analyzing for the image content

7. Latent Dirichlet Allocation Bag of words: • Text retrieval methods use the statistics of words in documents; • word, document, corpus • De Finetti’s theorem leads to the “bag of words” model; • Words are exchangeable (conditionally independent) => which takes to a generative process; • Only histograms of words are used; • A image – text analogy has to be made : • word • document • corpus Titanic Water Authorities Ship Water Titanic Ship Authorities corpus word document Pixels’ grey levels Pairs of objects

8. Latent Dirichlet Allocation • 1. LDA for patch level analysis – Semantic rules discovery: • Algorithm for extracting words – pixels defined by spectral indexes. Training set • Documents are assigned to topics with αi,j probability. • Words are assigned to topics with βi,j probability. αj = p(zi|D) βi,j = p(wi|zi) p(w|W) Maximum likelihood wj = zi i=argmaxiβi,j i = no. of topics j = no. of topics

9. Latent Dirichlet Allocation 2.LDA for scene level analysis – Semantic rules discovery based on spatial interaction: • Algorithm for extracting words – topological signatures Compute histogram of forces for θ=[θp : θp+3600] Normalization to [0;1] Generate reference punctiform object in the centroid Compute principal axis θp Spatial signature Groups of objects Object2 Object1 Training set Spatial class Word LDA + maximum likelihood wj = zi p(w|W) Semantic class KNN classifier wj = zi,k j = 1 … N, no. of words i =1 … n, no. of spatial topics (LDA) k = 1 … K, no. of spectral classes combination Combining objects 2 by 2 K=C(C+1)/2 possible types of combinations C spectral classes of objects

10. Latent Dirichlet Allocation • LDA generative process: • Choose a K-dimensional Dirichlet random variable θ~Dir(α), • where K = the number of topics in the collection ; • For each of words position nє{1, …, N}: • choose a topic zn~Multinomial(θ); • choose a word wn from p(wn |zn , β) β z w α θ N M • Document probability density function under the assumption the document and word belong to a topic with probabilities α, β.

11. Latent Dirichlet Allocation Documents = mixtures of proportions of topics Corpus visual words/blobs Corpus real words • Latent Dirichlet Allocation – Concept: • generative probabilistic model for collections of discrete data (corpus); • three-level hierarchical model, in which documents of a corpus are represented as • random mixtures over latent topics; • each topic is in turn, characterized by a distribution over words. • the order of words in a document is ignored in the LDA model (bag-of-words • assumption), so each document of the corpus is represented as a sequence of N words. Define vocabulary LDA Reduced number of words Vocabulary is reduced to topics Bag-of-words each visual word in a document is modeled as a sample from a topic classification D. Blei, A. Yang and M. Jordan, “Latent Dirichlet Allocation”, Journal of Machine Learning Research 3. 2003, pp. 993-1022

12. WORKFLOW Pixel & Patch Level Analysis – Semantic Rules Discovery Knowledge discovery * Input image – low-level features * Visual vocabulary – intermediate-level semantic concepts attached to the spectral map (Soilmapper) * Interactive training CLC-like concepts * LDA classifies each pixel to one of the CLC-like classes with high-level semantic meaning * The semantic rules bridge the gap between human-centered concepts and low-level machine features

13. WORKFLOW Pixel & Patch Level Analysis – Semantic Rules Discovery Interactive training stage • Quickbird image, 1000x1000 pixels • Training Documents - the link between the image features and semantics; • 3 - 5 documents / topic required; • LDA generates p(w|z) – the topics Water Bodies, River Urban Areas

14. Pixel & Patch Level Analysis – Basic Semantic Rules Discovery • Automatic mapping – pixel level • each pixel is classified to the • topic with max likelihood p(w|z) Vegetation Topic 55% forest type 1 40% grassland 5% barren soil Water Topic 50% deep water 40% shallow water 10% wetlands Urban Areas Topic 30% industrial buildings 50% civilian houses 20% roads

15. Pixel & Patch Level Analysis – Basic Semantic Rules Discovery • Automatic mapping • document level • each pixel is classified to the • topic with max likelihood p(D|z) • the size of the document is • variable according to the user’s • requirements • the legend is automatically • generated from the training • document Urban Areas

16. Pixel & Patch Level Analysis – Basic Semantic Rules Discovery Knowledge extraction • Landsat image • Vocabulary of Worlds (Soilmapper) • Automatic Map – pixel level • Automatic Map – document level • Semantic Rules bridging the gap between low level features (Landsat image) and high-level semantic classes CLC 2000

17. Scene Level Analysis – Semantic Rules Discovery Italy, QuickBirdimage, 2000x2000 pixels • Data: • 129 objects of interest; • 5 spectral classes; • 766 pairs of objects; • 25 tiles; • Training: • ~30% for training; • Results: • 7 spatial topics; • 15 spectral combinations; • 105 semantic classes. houses, open lands, water, large buildings, refugee camps

18. Scene Level Analysis – Semantic Rules Discovery Case study - Experimental results Object distribution over the 7 spatial topics Pairs of objects distribution over the 15 spatial-spectral classes

19. Scene Level Analysis – Semantic Rules Discovery Example of semantic classes • pairs of houses and large buildings with the • spatial arrangement in topic 5: • pairs of open lands and large buildings • with the spatial arrangement in topic 2: 2 1 2 1 1 2

20. Scene Level Analysis – Semantic Rules Discovery Example of semantic classes • pairs of refugee camps and open lands with the spatial arrangement in topic2: • pairs of houses and refugee camps with the spatial arrangement in topic 5:

21. Scene Level Analysis – Semantic Rules Discovery Example of semantic classes • pairs of refugee camps and large buildings with the spatial arrangement in topic 1: 1 1 2 2 1 2

22. Conclusions • The hierarchical structuring of image content enables an efficient contextual analysis for a better scene understanding. • A flexible generative probabilistic model for collections of discrete data, the LDA is a generative model for really efficient recognition, if the training data set is representative with respect to the type of information that is requested. • Experiments highlights good performances with small amount of training data. Acknowledgement • The research presented in this paper is in the frame of ESA-PECS Romanian Knowledge Based Earth Observation (RoKEO) project.

23. Thank you for your attention ! Questions ? corina.vaduva@gmail.com