Download
1 / 30
300 likes | 413 Views
Contextual Image Search. Wenhao Lu , Jingdong Wang , Xian- Sheng Hua , Shengjin Wang , Shipeng Li Tsinghua University, Beijing, P. R. China, Microsoft Research Asia, Beijing, P. R. China,. MM 2011. Outline. System overview Database construction
E N D
Contextual Image Search Wenhao Lu , Jingdong Wang , Xian-ShengHua, Shengjin Wang , Shipeng Li Tsinghua University, Beijing, P. R. China, Microsoft Research Asia, Beijing, P. R. China, MM 2011
Outline • System overview • Database construction • Contextual image search with text/image input • Experiment • Future Work MM 2011
System overview • Text input MM 2011
System overview • Image input MM 2011
Database construction MM 2011
Database construction 1. Feature extraction (MSER) extracts stable regions from the image by considering the change in area w.r.t the change in intensity of a connected component defined MM 2011
Database construction 2. SIFT descriptor MM 2011
Database construction 2. SIFT descriptor MM 2011
Contextual Image Search With Text Input 1. Context Capturing • textual contexts: page title / document title • local context • visual contexts: vision-based page segmentation algorithm • (VIPS) MM 2011
vision-based page segmentation Traditional DOM tree MM 2011
vision-based page segmentation VIPS MM 2011
vision-based page segmentation • DOM tree +Visual Info Tag cue:<HR> Color cue: background color Text cue Size cue MM 2011
Contextual Image Search With Text Input 2. Contextual Query Augmentation • Goal: remove possible ambiguities • Augmented query = query + textual context Candidate augmented query evaluate the relevance between the context and augmented query (Okapi BM25) MM 2011
Contextual Image Search With Text Input 2. Contextual Query Augmentation • Okapi BM25 : extended context (using synonyms, stemming, and so on) ~ k=2.0, b=0.75 MM 2011
Contextual Image Search With Text Input 2. Contextual Query Augmentation 3. Image Search by Text Rank score = : static score (ex. the Web page holding this image)
Contextual Reranking • textually contextual reranking : discarding the augmented query related words , • visually contextual reranking 1. Filter out images whose semantic contents may not be relevant to the query. (compute local textual context and query) MM 2011
Contextual Reranking • visually contextual reranking 2. Visual word weight: Find common pattern 3. Compute similarity :visual contexts :an image :histogram vector of i :histogram vector of k MM 2011
Overall Ranking = 0.2 = 0.2 =1 MM 2011
Contextual Image Search with Image Input • Search to annotation • discovers the candidate textual queries using the technique • “Annotating images by mining search result” (IEEE 2008) MM 2011
Contextual Image Search with Image Input • Search to annotation MM 2011
Contextual Image Search with Image Input • Search to annotation • First : find similar image • Second: surrounding texts of the obtained duplicated images • are mined to get a list of candidate textual queries visual features semantic features MM 2011
Contextual Image Search with Image Input • Search to annotation MM 2011
Contextual Image Search with Image Input 2. Contextual query identification ~ • calculate MM 2011
Experiment • 15,000,000 images and associated web pages • 5 users (level 0~level 3) MM 2011
Experiment 0.95 0.65 nDCG curves MM 2011
Experiment • Visual Result for Text Input MM 2011
Experiment • Visual Result for Text Input (Textual Reranking) MM 2011
Experiment • Visual Result for Text Input (Visual Reranking) MM 2011
Experiment • Visual Result for Image Input textual query “Van gogh” MM 2011
Future Work • More general contextual image search, including • mobile image search with wider contexts (e.g., • position, time, and history) • 2.Extend contextual image search to contextual • video search by applying the proposed methodology • and investigating extra video contexts MM 2011
