Irwin King and Zhong Jin Nov. 2001

1. Relevance Feedback Content-Based Image Retrieval Using Query Distribution Estimation Based on Maximum Entropy Principle Irwin King and Zhong Jin Nov. 2001

2. 1. Introduction Content-Based Image retrieval (CBIR) : The selection of images from a collection via primitive visual features representing color, shape, and texture extracted from images themselves. Successful CBIR systems require the integration of various techniques in the fields of : • Pattern Recognition (PR) • Digital Image Processing (DIP) • Information Retrieval (IR)

3. 1. Introduction (Cont ’d) Relevance feedback (RF) : an iterative and interactive process for query reformulation based on user's feedback. RF techniques include mainly: • query moving technique • similarity function re-weighting technique

4. 1. Introduction (Cont ’d) Problems: Since the retrievals under the commonly used nearest-neighbor rule cannot reflect the query distribution function properly , most of relevance feedback techniques may fail under the following assumption: • The number of relevant retrievals is small • The number of iterations is required to be small

5. 2. Background Review RF technique in CBIR system can be regarded as a form of two-stage automatic learning for the unknown query distribution function: • Estimate the query distribution function by using the Expectation-Maximization (EM) algorithm or by the classical statistics. • Generate the inquiries to be returned to the user, where the nearest-neighbor rule is commonly used.

6. 2. Background Review (Cont ’d) • Limitation of estimation theories: • EM has its limitations in CBIR Because of a small number of labeled data in RF. • Only relevant informaton can be utilized by classical statistical theory.

7. 2. Background Review (Cont ’d) Limitation of the nearest-neighbor rule: • the retrievals generated cannot wholly reflect the Query Distribution Function (QDF) because the underlying QDF may not be isotropic in nature. Note: QDF is the statistical distribution function deformed by all the images similar to the given query image in high dimensional feature space.

8. 2. Background Review (Cont ’d) Maximum Entropy Principle (MEP): To obtain estimations by determining a probability distribution associated with a random variable over a discrete space which has the greatest entropy subject to constraints on the expectations of a given set of functions of the variable. Shannon’s Entropy The Maximum Entropy (MAXENT) solution with no bias (or constraints) is

9. 2. Background Review (Cont ’d) Some work on IR by MEP: In the early 80's, Cooper et al. made a strong case for applying the maximum entropy approach to the problems of information retrieval. Kantor extended the analysis of the MEP in the context of information retrieval. Recently, Greiff and Ponte took a fresh look at modeling approaches to information retrieval and analyzed classical probabilistic IR models in light of the MEP.

10. 3. Proposed Framework Our novel framework for image retrieval includes the following stages: • Estimation stage -- Estimate the query mean and the query covariance matrix by using accumulative relevance retrieval information and irrelevant retrieval information. • Generation stage -- Generate a set of inquiries for relevance selection based on MEP.

11. 3. Proposed Framework (Cont ’d) Estimation stage When the number T of relevant retrieval is less than the dimension M of the feature space, it is assumed that When T=1, an estimation can be given by an equal-probability constrain

12. 3. Proposed Framework (Cont ’d) • For K number of retrievals, K+1 points can be determined according to the following equal-probability conditions: Generation stage • all similar images in the database can be divided in the following K subsets: where

13. 4. Experiments and Analysis Database There are 1,400 trademark images with 128*128. Here are ten samples:

14. 4. Experiments and Analysis (Cont ’d) Here are 10 deformation transformations:

15. 4. Experiments and Analysis (Cont ’d) 100 Test Images:

16. 4. Experiments and Analysis (Cont ’d) Feature Extraction:7 dimensional invariant moment

17. 4. Experiments and Analysis (Cont ’d) The retrieval performance is measured using the following Average Retrieval Precision (ARP): Experimental Aim: to evaluate the efficiency of the proposed generation stage, Generation MAXENT. where K=10

18. 4. Experiments and Analysis (Cont ’d) Step 1: For a query image, return K retrievals by Euclidean distance. Step 2: Perform the estimation stage and return retrievals by Euclidean distance, Mahalanobis distance, and Generation MAXENT respectively. Step 3: Perform the estimation stage and return K retrievals by using the Mahalanobis distance; In order to compare Generation MAXENT with Euclidean distance and Mahalanobis distance, a set of three-step experiments are designed as follows:

19. 4. Experiments and Analysis (Cont ’d) Experimental results: Http://www.cse.cuhk.edu.hk/~miplab/MAXENT

20. 4. Experiments and Analysis (Cont ’d) Result analysis: According to the ARP's in Step 3 in Table 1, the proposed generation stage Generation MAXENT outperforms the commonly used Euclidean distance and Mahalanobis distance The proposed generation stage Generation Based On MAXENT aims to retrieve image samples which can reflect the query distribution function. This is the reason why the ARP of MAXENT in Step 2 in Table 1 is lower than those of Euclidean distance and Mahalanobis distance.

21. 5. Conclusion Novel two-stage relevance feedback framework for content-based image retrieval based on query estimation and the Maximum Entropy Principle is shown to be succeful in improving accuracy and speed on a trademark image database. Future work: to overcome the difficulty in image retrieval for high-dimensional features in large image databases

22. Acknowledgment This paper is supported in part by an Earmarked Grant from the Hong Kong Research Grants Council #CUHK4407/99E.