Fuzzy based Contextual Cueing for Region Level Annotation

1. Fuzzy based Contextual Cueing for Region Level Annotation Sheng-huaZhong, Yan Liu, Yang Liu, Fu-lai Chung Department of Computing The Hong Kong Polytechnic University

2. Outline • Introduction to region level annotation • Definition and motivation • Representative work and their limitations • Contextual cueing for image understanding • Contextual cueing and spatial invariants • Spatial invariants for image understanding • Fuzzy theory for contextual modeling • Motivation of utilizing fuzzy theory in contextual cueing modeling • Fuzzy representation and reasoning for spatial invariants • Label propagation using fuzzy based contextual cueing • Experiments • Conclusion and future work 2

3. Outline • Introduction to region level annotation • Definition and motivation • Representative work and their limitations • Contextual cueing for image understanding • Contextual cueing and spatial invariants • Spatial invariants for image understanding • Fuzzy theory for contextual modeling • Motivation of utilizing fuzzy theory in contextual cueing modeling • Fuzzy representation and reasoning for spatial invariants • Label propagation using fuzzy based contextual cueing • Experiments • Conclusion and future work 3

4. Introduction to Region Level Annotation Annotation Water Cow Grass (a) An image with given image-level annotations (b) An image with automatic label to region assignments • Definition of region level annotation • Segment the image to semantic regions • Assign the given image-level annotations to precise regions • Motivation of automatic region level annotation • Helpful to achieve reliable content-based image retrieval [Liu, ACM MM 09’] • Substitute tedious manually region-level annotation 4

5. Representative Work of Region Level Annotation • Early work on region level annotation • Known as simultaneous object recognition and image segmentation • Unsupervised learning • Handle images with single major object or with clean background[Cao, ICCV 07’] • Supervised learning • Focus on special object recognition or special domain [Li, CVPR 09’] • Latest work for real-world applications • Label propagation by bi-layer sparse coding [Liu, ACM MM 09’] • Common annotations are more likely to have similar visual features in the corresponding regions • Show impressive results on nature images

6. Limitation of Visual Similarity in Region Level Annotation

7. Outline • Introduction to region level annotation • Definition and motivation • Representative work and their limitations • Contextual cueing for image understanding • Contextual cueing and spatial invariants • Spatial invariants for image understanding • Fuzzy theory for contextual modeling • Motivation of utilizing fuzzy theory in contextual cueing modeling • Fuzzy representation and reasoning for spatial invariants • Label propagation using fuzzy based contextual cueing • Experiments • Conclusion and future work 7

8. Contextual Cueing • The aim of the proposed work • Provide more semantic understanding of the image with the aid of contextual cueing • Definition of contextual cueing [Chun, CP 98’] • The manner in which the human brain gathers information from visual elements and their surroundings • Contextual cueing acquirement • Incidentally learn form past experiences of regularities • Contextual cueing representation • The knowledge about spatial invariants [Biederman CoP 82’] • Probability: likelihood that object will be present • Co-occurrence: likelihood that objects will be present together • Size: familiar relative size of objects • Position: typical positions of some objects • Topological relations: left of, above, surround, inside, and etc

9. Spatial Invariants for Image Understanding • Spatial invariants in human visual processing [Chun, TCS 00’] • Guide the visual attention • Speed the visual elements search • Help object recognition • Spatial invariants for image understanding • The performance can be improved obviously even one or two kinds of spatial invariants are used simply [Rabinovich, ViSU 09’] [Galleguillos, CVPR 08’] (a) An ambiguous object (b) A hat on the head (c) A cup surrounding dishware

10. Outline • Introduction to region level annotation • Definition and motivation • Representative work and their limitations • Contextual cueing for image understanding • Contextual cueing and spatial invariants • Spatial invariants for image understanding • Fuzzy theory for contextual modeling • Motivation of utilizing fuzzy theory in contextual cueing modeling • Fuzzy representation and reasoning for spatial invariants • Label propagation using fuzzy based contextual cueing • Experiments • Conclusion and future work 10

11. Contextual Cueing Modeling by Fuzzy Theory • The difficulty of modeling contextual cueing • Classical bivalent set theory causes serious semantic loss • Example of imprecise position and ambiguous topological relationship • Fuzzy theory • Measure the degree of the truth • Fuzzy membership to quantize the degree of truth • Fuzzy logic allows decision making using imprecise information (b) Example of topological relationship for object recognition (a) Example of ambiguous topological relationship

12. Fuzzy Membership for Spatial Invariants of Position • Vertical location is very important for image understanding according to the conclusion in [Torralba, PSR06’] • Top, middle, and bottom are used to characterize the position : the number of representative point. (a) Fuzzy membership for position

13. Fuzzy Membership for Spatial Invariants of Topological relationship 13

14. Outline • Introduction to region level annotation • Definition and motivation • Representative work and their limitations • Contextual cueing for image understanding • Contextual cueing and spatial invariants • Spatial invariants for image understanding • Fuzzy theory for contextual modeling • Motivation of utilizing fuzzy theory in contextual cueing modeling • Fuzzy representation and reasoning for spatial invariants • Label propagation using fuzzy based contextual cueing • Experiments • Conclusion and future work 14

15. Flowchart of Fuzzy based Contextual Cueing Label Propagation

16. Algorithm of Label Propagation by Fuzzy based Contextual Cueing

17. Detailed Calculation of Contextual Cueing using Fuzzy Membership • Whether the difference of the fuzzy membership in adjacent iterations is larger than threshold • Calculate the similarity of fuzzy position between every annotation and concept • Calculate the similarity of spatial topological relationship between annotation and concept • Update the fuzzy membership of every patch • Normalization • The membership of this iteration is set as the final membership 17

18. Examples of Spatial Invariants for Some Objects (b) Fuzzy membership of topological relationship (a) Fuzzy membership of position

19. Illustration of Fuzzy Based Contextual Cueing Label Propagation Annotation Sky Water Beach Boat (a) Original image with given image-level annotations (a) Over segmentation result (c) Label propagation inter images (d) Label propagation using fuzzy based contextual cueing

20. Outline • Introduction to region level annotation • Definition and motivation • Representative work and their limitations • Contextual cueing for image understanding • Contextual cueing and spatial invariants • Spatial invariants for image understanding • Fuzzy theory for contextual modeling • Motivation of utilizing fuzzy theory in contextual cueing modeling • Fuzzy representation and reasoning for spatial invariants • Label propagation using fuzzy based contextual cueing • Experiments • Conclusion and future work 20

21. Experiment on MSRC Dataset • MSRC Dataset • 380 images with 18 categories • Including building, grass, tree, cow, boat, sheep, sky, mountain, aeroplane, water, bird, book, road, car, flower, cat, sign, and dog • Comparison methods • Four baseline methods implemented by binary SVM with different values of maximal patch size • SVM1: 150 pixels, SVM2: 200 pixels, SVM3: 400 pixels, and SVM4: 600 pixels • Two latest LRA techniques [Liu , ACM MM09’] • Label propagation with one-layer sparse coding • Label propagation with bi-layer sparse coding • Experimental result Table 1. Label-to-region assignment accuracy comparison.

22. Experiment Analysis on MSRC Dataset Annotation Sky Building Tree Road (a) An image with annotations (b) Bi-layer result (c) FCLP result Annotation Sky Building Tree Road Car (d) An image with annotations (e) Bi-layer result (f) FCLP result

23. Experiment on COREL Dataset • COREL Dataset • 150 images with 8 categories • Experimental result Table 2. Label-to-region assignment accuracy comparison.

24. Outline • Introduction to region level annotation • Definition and motivation • Representative work and their limitations • Contextual cueing for image understanding • Contextual cueing and spatial invariants • Spatial invariants for image understanding • Fuzzy theory for contextual modeling • Motivation of utilizing fuzzy theory in contextual cueing modeling • Fuzzy representation and reasoning for spatial invariants • Label propagation using fuzzy based contextual cueing • Experiments • Conclusion and future work 24

25. Conclusion and Future Work • Region level annotation is an important task in multimedia content analysis • Targets the ultimate problem of image understanding • Makes the commercial multimedia search engine possible • Contextual cueing can effectively improve the performance of region level annotation • Works well on natural image datasets • Demonstrates impressive performance when objects have similar visual information • Fuzzy theory is a good model to describe contextual cueing • Especially for imprecise position information and ambiguous topological relationship • A good try to fill the semantic gap between human judgment and computable features using mathematical model • Future works • Extend fuzzy based label propagation to image level annotation • Model contextual cueing in large scale dataset

26. Reference [1] Biederman, R. Mezzanotte, and J. Rabinowitz, “Scene perception: detecting and judging objects undergoing relational violations”, In Cognitve Psychology, vol. 14(2), pp. 143–77, 1982. [2] K. Miyajima and A. Ralescu, “Spatial organization in 2D images”, In IEEE International Conference on Fuzzy Systems, pp. 100–105, 1994. [3] Chun, M. M. & Jiang, Y.. , “Contextual cueing: implicit learning and memory of visual context guides spatial attention”, In Cognit. Psychol., vol. 36, pp. 28-71, 1998. [4] Chun, M. M., “Contextual cueing of visual attention”, Trends in Cognitive Sciences, vol. 4, pp.170-178, 2000. [5] A. Torralba, A. Oliva, M.S. Castelhano and J.M. Henderson., “Contextual guidance of eye movements and attention in real world scenes: The role of global features in object search”, In Psychological Review., pp. 766-786, 2006. [6] L. Cao and F. Li. Spatially coherent latent topic model for concurrent object segmentation and classification. In ICCV, pp. 1–8, 2007. [7] C. Galleguillos, A. Rabinovich and S. Belongie, “Object categorization using co-occurrence, location and appearance”, In CVPR, June. 2008. [8] J. Li, R. Socher, and L. Fei-Fei, “Towards total scene understanding: classification, annotation and segmentation in an automatic framework”, In CVPR, 2009. [9] Rabinovich, A., and Belongie, S. “Scenes vs. objects: acomparative study of two approaches to context based recognition,” In ViSU, 2009. [10] Xiaobai Liu, Bin Cheng, Shuicheng Yan, Jinhui Tang, Tat Seng Chua, Hai Jin, “Label to region by Bi-Layer sparsely priors”, In Proceedings of ACM Multimedia, pp. 115-124, Oct. 2009. 26

27. Q & A Thank You ! 27 27