470 likes | 748 Views
Learning in Large Scale Image Retrieval Systems. Under the guidance of: Dr. C. V. Jawahar & Dr. Vikram Pudi. by Pradhee Tandon Roll No. 200607020. Image Retrieval. Explosive growth in images Easy access to most of these on the web Contemporary systems used tags
E N D
Learning in Large Scale Image Retrieval Systems Under the guidance of: Dr. C. V. Jawahar & Dr. Vikram Pudi by Pradhee Tandon Roll No. 200607020
Image Retrieval Explosive growth in images Easy access to most of these on the web Contemporary systems used tags The best commercial systems are still tag based Inadequate and unreliable Manual tagging is infeasible Content based retrieval is the best option
Content Based Image Retrieval Query Feature Extraction Comparison Module Results Image Feature Database Query Results
Content Based Image Retrieval Query Feature Extraction Comparison Module Results Image Feature Database Feature Index
Content Based Image Retrieval Query Feature Extraction Comparison Module Results Rf Relevance Learning Feature Index Memory & Logs
Content Based Image Retrieval Query Feature Extraction Comparison Module Results Rf Relevance Learning Learning Memory Feature Index
Features Color Histograms Texture Filters Shape Context SIFT GLOH Spatial indexing methods Kd – trees R-tree Distance Metrics Euclidean Mahalanobis KL Divergence Relevance Feedback Short term learning Long term learning Content Free Retrieval Active Learning Diversity Retrieval Scope of work
Requirements • Efficient Image Retrieval • Learning relevant features in images • Learning image-image relationships • Diversity in retrieval for improved learning
Requirements • Efficient Image Retrieval • Learning relevant features in images • Learning image-image relationships • Diversity in retrieval for improved learning
Implementation of FISH • Image Representation in FISH • MPEG-7 Colour Structure Descriptor • Maximum Response Filters for Textures Developed on the LAMP stack, using C/C++, Perl, PHP, HTML, MySQL and Apache TPIE toolkit from Duke University for B+ tree implementation
Indexing Scheme Interactive response over large databases (less than a second) Efficient scalable index (dynamic with data) Similarity indexing scheme (r-tree, kd-tree, ss-tree) Support for changing similarity metrics (metric changes with learning) B+ tree based index Nataraj et. al, MMM 2007, Efficient Search with Changing Similarity Measures on Large Multimedia Datasets
The Retrieval Algorithm Query Feature Extraction Comparison Module Results Rf Relevance Learning Feature Index Learning Memory Retrieval in FISH
Retrieval Performance Retrieval times with increasing DB sizein (secs) & #dimensions fixed at 10 Retrieval times with increasing #Dimensionsin (secs) & DB size fixed at 1 lakh
Requirements • Efficient Image Retrieval • Learning relevant features in images • Learning image-image relationships • Diversity in retrieval for improved learning
Content Based Image Retrieval Query Feature Extraction Comparison Module Results Rf Relevance Learning Feature Index Memory & Logs
Learning - expectations Effective – capture user intent correctly Efficient – interactive retrieval response Scalable – limited computational overhead for large collections Adaptive – caters to individual user’s subjectivities • Intra-query or short term learning (STL) Evolving – incrementally improves across users and queries • Inter-query or long term learning (LTL) Dynamic – seamlessly absorbs changes in the collection
Learning - Method • Relative relevance of features using feedback • Numerous methods can be used • Discriminative variance is as - • Weights are incrementally learnt over iterations using – • At the end of the session long term learning is updated for the relevant images using – • Image to image dissimilarity is computed using – • Weighted Mahalanobis
Improved accuracy Precision across sessions using LTL Rank Convergence of top N relevant samples Sum of ranks of Top 10 relevant images converges close zero (downshifted) over multiple sessions with long term learning
Improved retrieval System learns the rock and sky pattern over sessions System learns the yellow flower in the hedge over sessions Top 9 results for queries across 3 different sessions (left-most are queries too)
Content from Learning • Long term memory allows learning of relevant image features • Converges to popular content over sessions • For example, • Assume, features are associated with individual pixels, colors • Consider a gray image, pixels for more relevant features are colored brighter Actual image Content image
Visual Content Extraction Over sessions After a large number of sessions
Requirements • Efficient Image Retrieval • Learning relevant features in images • Learning image-image relationships • Diversity in retrieval for improved learning
Content Based Image Retrieval Query Feature Extraction Comparison Module Results Rf Relevance Learning Feature Index Memory & Logs
Image-Image Relations Query Given a history of patterns in behavior and a current partial pattern, collaborative filtering predicts the next pattern for the latter Content Free Image Retrieval orCFIR, uses feedback logs to predict the next set of results for the current pattern
Hybrid Image Retrieval We integrate them in a Bayesian inference like framework, • The a priorirelationships from logs • Theevidence from visual similarity • Retrieval is an a posteriori estimation problem
Bayesian Image Retrieval System Architecture of the proposed Bayesian Image Retrieval System
Bayesian Image Retrieval... posterior = prior * evidence Efficient a priori updates • The prior probabilities are not stored, reduces updates • Co-relevance between images are stored in a matrix • The a prioriis estimated using the co-relevance values Evidence computation • Weights are learnt using discriminative variance method • WeightedMahalanobis for (dis)similarity
Concept Discovery a priorimatrix has embedded patterns of similar co-relevances Co-relevance patterns can be summarized into ‘k’ concepts • cluster the patterns into V concepts 1…k. • clustering is repetitive but offline • exhaustive comparisons are avoided
Accuracy with Bayesian Gain in precision with Bayesian Gain in precision across sessions Using real human user feedback logs Using annotation based feedback logs
Accuracy with Bayesian CBIR results and Bayesian results
Requirements • Efficient Image Retrieval • Learning relevant features in images • Learning image-image relationships • Diversity in retrieval for improved learning
Diversity in Image Retrieval Query Query
Skylines – the natural solution Results should be similar in a variety of different ways Skylines return non-dominated samples Non-dominated samples are closer to the query than all the others, in at-least one way (attribute)
Skyline Extraction Architecture of the proposed skyline based similarity retrieval system
Efficient Skylines Synthetic data with 10 dimensions and 10000 and 15000 data points Real image data with 12 and 9 dimensions with 11901 real images
Preferential Skylines Relevance feedback represents user’s preference Weights learned using feature relevance Skylines are then computed in user space
Contributions Designed and implemented a web-based image retrieval system, called FISH Proposed an efficient feature relevance learning algorithm Integration of complimentary CFIR and CBIR a Bayesian inference framework Skylines to retrieve diversely similar samples for a given query
Future directions Videos are richer and the next step Efficient higher level concept discovery is needed Skylines with preference should be explored further
Publications Pradhee Tandon, Piyush Nigam, Vikram Pudi, C. V. Jawahar, “FISH: A Practical System for Fast Interactive Image Search in Huge Databases”, in Proceedings of the 7th ACM International Conference on Image and Video Retrieval (CIVR ’08), July 6-8, 2008, Niagara Falls, Canada. Pradhee Tandon, C. V. Jawahar, “Long Term Learning for Content Extraction in Image Retrieval”, in Proceedings of the 15th National Conference on Communications (NCC ’09), January 16-18, 2009, Guwahati, India. Pradhee Tandon, C. V. Jawahar, “Bayesian Image Retrieval” submitted to 3rd International Conference on Pattern Recognition and Machine Intelligence (PReMI ’09), December 16-20, 2009, New Delhi, India.
The Retrieval Algorithm *Learning discussed in detail later
Bayesian Image Retrieval • The a priori probability of retrieving image ‘a’with query ‘q’ is P(R) = n(q,a)/n(a) • where n(a) denotes relevant retrievals for ‘a’ • The evidence from visual similarity is computed as p(S|R) = f(w,q,a) • where weights ‘w’ are refined using relevance feedback • The posterior probability of retrieval is computed as p(R|S) = p(S|R) P(R) • the denominator can be ignored • PicHunter is a hybrid but does no feature learning • Zhong et. al, use Bayes inference for a probabilistic decision only