1 / 61

Jianping Fan

Relevance Feedback for Image Retrieval. Jianping Fan. Department of Computer Science. University of North Carolina at Charlotte. Charlotte, NC 28223. Content-Based Image Retrieval (CBIR). Image Database. Query Image. CBIR over Internet. Online Image Database.

chesterk
Download Presentation

Jianping Fan

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Relevance Feedback for Image Retrieval Jianping Fan Department of Computer Science University of North Carolina at Charlotte Charlotte, NC 28223

  2. Content-Based Image Retrieval (CBIR) Image Database Query Image

  3. CBIR over Internet Online Image Database

  4. Flowchart of CBIR System Large-Scale Image Database Feature Extraction for Representation Database Indexing offline online Query Intention Estimation Feature Extraction Query Image Database Indexing Query Results

  5. Query-By-Example Query Example Feature Extraction Distance Function Within-Node Nearest Neighbor Search Top-K Results

  6. Somethings could be wrong • Query term is far from user’s expectation or query intention estimation is incorrect • Similarity function is incorrect • Database indexing is incorrect Relevance Feedback?

  7. Flowchart of CBIR System Large-Scale Image Database Feature Extraction for Representation Database Indexing Users are not involved! offline online Query Intention Estimation Feature Extraction Query Image Database Indexing Users are highly involved! Query Results

  8. Large-Scale Image Collections Feature Extraction for Image Representation Users’ Preferences could be different! Database Indexing Construction

  9. 2. What’s Relevance Feedback? • User sends his/her request to the database system; • The database system sends him/her some ranked answers; • The user can exchange his/her judgment with the system. no relevant? no

  10. Query Point Movement: Query is incorrect

  11. Query Point Movement: Query is incorrect

  12. Query Point Movement: Query is incorrect • Query Updating Vectors for Negatives New Query Vector Previous Query Vector Vectors for Positive Images

  13. Query Point Movement: Query is incorrect • Query Updating Far away from Negatives New Query Vector Previous Query Vector Closer to Positive Images

  14. Query Point Movement: Query is incorrect

  15. Query Point Movement: Query is incorrect • Why it may work? • Neighborhood is changed, we may touch new images in database even our similarity function and image database keep unchanged! • When it may not work? Semantic Gap

  16. Query Point Movement: Query is incorrect • Two Approaches for Query Point Movement • Query point movement control • Informative sampling

  17. Query Point Movement: Query is incorrect Target Image • Query Point Movement Control Where to go? No Convergence Best Search Road Initial Query Point Potential Convergence Search Road Different moving paths have different neighborhoods

  18. Query Point Movement: Query is incorrect • Close to positive ones • Far away from negative ones ? Informative Image Sampling

  19. Query Point Movement: Query is incorrect a. Initialize the query

  20. Query Point Movement: Query is incorrect b. Send the query to system

  21. Query Point Movement: Query is incorrect c. Client mark the relevant examples

  22. Query Point Movement: Query is incorrect d. System Evaluation according to client feedback

  23. Query Point Movement: Query is incorrect e. Second client feedback

  24. Query Point Movement: Query is incorrect f. Second System Evaluation

  25. Use new similarity function • Distance Weighting Approach: Database Indexing Query Example Feature Extraction

  26. Use new similarity function Query Example Feature Extraction Distance Function Within-Node Nearest Neighbor Search Top-K Results

  27. Use new similarity function • Effectiveness of Feature Weighting: Original Feature Space Weighted Feature Space

  28. Use new similarity function • Why it works? • It may have different neighborhoods when the number of returned images is fixed

  29. Use new similarity function • Problem for Feature Weighting Approach • Cost-Sensitive: It is very expensive to update the feature weights on real time! b.Semantic Gap: The distance functions may not be able to characterize the underlying image similarity effectively! c.Visualization: The underlying image display tools may separate similar images in different places, it is hard for users to evaluate the visual similarity (relevance) between the images!

  30. Use new similarity function • Challenging Issues a. Convergence: It is very important to guarantee the algorithm for kernel updating is converged! b. Cost Reduction: It is very important to reduce the cost for kernel updating!

  31. Integrating Relevance Feedback with Database Indexing • Query Point Movement • User Different Similarity Functions

  32. Query is initialized by keyword Kernel-Based Clustering of Google Search Results Similarity-Based Image Projection and Visualization Intention capturing and Kernel Selection for Junk Image Filtering 4. Relevance Feedback for Query by Keywords Relevance is user-dependent!

  33. Requirements for such new search engine: 4. Relevance Feedback for Query by Keywords • Fast algorithm for feature extraction; • Multiple kernels for diverse image similarity characterization; • Implicit query intention capturing and real-time kernel updating

  34. 4. Relevance Feedback for Query by Keywords Keyword-Based Google Images Search Fast Feature Extraction & Basic Kernels Mixture-of-Kernels & Image Clustering Hyperbolic Image Visualization Accept? No Query Intention Expression & Hypothesis Making Increment Kernel Learning Through incremental learning, we can consider multiple competing hypotheses for the same task!

  35. 4. Relevance Feedback for Query by Keywords Fast Feature Extraction

  36. 4. Relevance Feedback for Query by Keywords Image Representation & Similarity • Color histogram for whole image • 10 color histograms for different patterns • Wavelet transformation Time-Constrainted Image Analysis!

  37. 4. Relevance Feedback for Query by Keywords points in HD Space images They are invisible for human eye!

  38. 4. Relevance Feedback for Query by Keywords • Color Histogram Kernel • Wavelet Filter Bank Kernel • Sub-Image Color Histogram Kernel Basic kernels for image similarity characterization:

  39. 4. Relevance Feedback for Query by Keywords Mixture-of-kernels for diverse similarity characterization: • It could be expensive for learning a good combination! • The similarity between the images depends on the given kernel function!

  40. 4. Relevance Feedback for Query by Keywords Hypothesis Making & Initial Analysis R Majority Outliers subject to: Decision function:

  41. 4. Relevance Feedback for Query by Keywords Similarity-Preserving Image Projection Transform large amount of images (represented by high-dimensional visual features) into their similarity contexts for enabling better visualization!

  42. 4. Relevance Feedback for Query by Keywords Hyperbolic Image Visualization & Hypothesis Assessment projection Invisible HD Space Visible 2D Disk Unit

  43. 4. Relevance Feedback for Query by Keywords Mountain

  44. 4. Relevance Feedback for Query by Keywords Ocean

  45. 4. Relevance Feedback for Query by Keywords Sunrise

  46. 4. Relevance Feedback for Query by Keywords Grass

  47. User-System Interaction for Making New Hypothesis 4. Relevance Feedback for Query by Keywords Hypothesis-Driven Image Re-Clustering

  48. 4. Relevance Feedback for Query by Keywords Hypothesis-Driven Data Analysis: • Updating decision function: margin between relevant images and irrelevant images! b. Updating the combination of feature subsets! c. Updating image projection optimization criteria to obtain more accurate projection! d. Updating image representation!

  49. 4. Relevance Feedback for Query by Keywords Incremental Learning: Update decision function • Dual Problem Subject to:

More Related