Image Matching and Retrieval by Repetitive Patterns

1. j i d d l k j j j i i i p = ( p ; p ) , p = ( p ; p ) k k ; 1 k ; 2 l l ; 1 l ; 2 Image Matching and Retrieval by Repetitive Patterns Petr Doubek, Jiri Matas, Michal Perdoch and Ondrej Chum Center for Machine Perception, Czech Technical University in Prague, Czech Republic Outline of the Algorithm References Motivation Conclusions Repeated Patterns Similarity Abstract Detection Detect repeated elements, find the lattice of the repeated pattern, rectify the lattice and calculate the mean tile Compute shift invariant tile descriptors Image Matching and Retrieval For each pair score with the most similar patterns • Image retrieval can benefit from repeated patterns if they are detected and handled properly • Proposed approach is able to detect 1D and 2D lattices under affine transformation • Shift invariant descriptors addresses tile ambiguity • We have shown retrieval based solely on repeated patterns, however it can be combined with standard bag-of-words retrieval approaches • T. Tuytelaars, A. Turina, and L. Van Gool, “Non-combinatorial detection of regular repetitions under perspective skew”, PAMI, vol.25, no.4, pp. 418-432, April 2003 • P. Doubek, J. Matas, M. Perdoch and O. Chum, “Detection of 2D lattice patterns of repetitive elements and their use for image retrieval”, technical report, CTU-CMP-2009-16, 2009 • T. K. Leung and J. Malik, “Detecting, localizing and grouping repeated scene elements from an image”, ECCV, 1996, pp. 546-555 • Multiple occurrences of a local patch pose a problem to one-to-one matching algorithms (local matches are ambiguous) • The presence of repeated local patches is in most cases non-accidental and therefore very distinctive • Regular and even non-regular repetitive patterns give a rise to geometric constraints Detection of repetitive patterns in images is a well-established computer vision problem. However, the detected patterns are rarely used in any application. A method for representing a lattice or line pattern by shift-invariant descriptor of the repeating tile is presented. The descriptor respects the inherent shift ambiguity of the tile definition and is robust to viewpoint change. Repetitive structure matching is demonstrated in a retrieval experiment where images of buildings are retrieved solely by repetitive patterns. • For a pair of images i,j with sets of detected repeated patterns Ci and Cj similarity sk,l of two patterns k,l is computed as • where and are shift invariant tile descriptions and • are pairs of peaks in RGB colour histograms. • Example: repeated patterns of two images and their similarity Image Retrieval by Repetitive Patterns • Experiment 1 • Performance of shift invariant representation • Ground truth for each query Gi = set of images of building i, was manually marked • Tested on two datasets with ~230 images in image retrieval of about 50 buildings Repetitive Pattern and Lattice Detection • Experiment 2 • Detection and matching of repeated patterns tested on image retrieval • Two publicly available datasets PSU-NRT(subset) and Pankrac+Marseille (http://cmp.felk.cvut.cz/data/repetitive) • Top three matches for some of the queries Detection of repeated elements. In our implementation affine covariant regions (MSERs and Hessian Affine) described by SIFT Agglomerative clustering of SIFTs. Each cluster hypothesise a repeated pattern For each element in the repeated pattern, find spatial nearest neighbours in each of the spatial sectors Find dominant vanishing points by Hough transform and form a 2D lattice Rectify lattice and divide pattern into tiles Dataset Query Top three best matches . A cluster of repeated elements Nearest neighbours and neighbourhood sectors A lattice from two vanishing points (corresponding to red and green directions) Shift-Invariant Tile Representation • Each repeated pattern is represented by an “average” appearance of a tile – a mean tile(a more complex representation possible) • After discovering lattice of a repeated pattern an intrinsic shift ambiguity remains • We propose two shift-invariant representations: • the magnitude of Fourier coefficients of a tile • “zero-phase” normalization: tile shifted so that phase of the first harmonic equals zero Original image Detected lattice Detected tiles Zero-phased tiles The authors were supported by Czech Science Foundation Project 102/07/1317 and by EC project FP7-ICT-247022 MASH. TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAA