490 likes | 636 Views
Efficient Search Methods. Coherency Sensitive Hashing. &. PatchMatch. Simon Korman and Shai Avidan. C. Barnes , E. Shechtman , A. Finkelstein, and D. B. Goldman. Presented by: Itamar Friedman. Nearest Neighbor Fields ( NNF ). For each patch – find nearest patch in other image
E N D
Efficient Search Methods Coherency Sensitive Hashing & PatchMatch Simon KormanandShaiAvidan C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman. Presented by: Itamar Friedman
Nearest Neighbor Fields (NNF) • For each patch – find nearest patch in other image • Dense : Number of patches ≈ image size • Exact NN is infeasible • Error is mean(patch-distances) A B NNF
Why NNF? Non-Local Means denoising Non Local Means. Baudes, Coll, Morel. CVPR 2005
Why NNF? Image Completion Object Removal by Examplar Based Inpainting. Criminisi et al’, CVPR 2003 Space-Time Completion. Wexler et al’, CVPR 2004
Appearance Space k-by-kpatch = k2 dimensional vector Ordinary Nearest Neighbor search • Two popular methods are: • KD-trees [Arya et al, 1998] • Locality Sensitive Hashing (LSH) [Indyk et al, 1998] • These methods • Can be very accurate • Are relatively slow
Image Space • Based on coherency
Each iteration, for each pixel: Compare two kinds of candidates (patches): 1. Randomly located patches (with probability that drops exponentially with distance from current match) Image Space - the PatchMatch algorithm[Barnes et al, SIGRAPH 2009] current match candidates
Each iteration, for each pixel: Compare two kinds of candidates (patches): 2. Propagated neighbor’s matches Image Space - the PatchMatch algorithm[Barnes et al, SIGRAPH 2009] left neighbor current match of left neighbor candidate
Each iteration, for each pixel: Compare two kinds of candidates (patches): 2. Propagated neighbor’s matches Image Space - the PatchMatch algorithm[Barnes et al, SIGRAPH 2009] current match of top neighbor top neighbor left neighbor current match of left neighbor candidates
An iterative randomized algorithm Simple and fast (runtime is in the seconds) Based on local propagation and mostly-local search Heavily relies on the fact that images are generally coherent Image Space - the PatchMatch algorithm[Barnes et al, SIGRAPH 2009]
Coherency Sensitive Hashing (CSH) Combine image space and appearance space • Find good matches • Propagate them across patches that either: • Are neighbors (image space) • Share hash values (appearance space)
Walsh Hadamard (WH) Basis 3 4 5 6 7 8 11 10 6 7 8 9 4 5 6 7 8 9 CSH - The LSH Framework • Project each point v on a random segmented line Hash function (code) Bin number LSH table Datar, Immorlica, Indyk, Mirrokni, 2004 The Gray-Code Filter Kernels, Hel-Or, Hel-Or, PAMI 2007
CSH indexing • Hash all patches of both images (with hash function g) Hash Table Image ‘A’ Image ‘B’ g g
CSH indexing • Hash all patches of both images (with hash function g) • Keep k (=2) representatives of each image in each entry Hash Table k k Image ‘A’ Image ‘B’ g g-1 g-1 g
CSH indexing - candidates Candidates = potential matching patches Image ‘A’ Image ‘B’
CSH indexing – candidate types Type I (appearance only) = Standard LSH Image ‘A’ Image ‘B’
CSH indexing – candidate types Type II (appearance and coherence) Image ‘A’ Image ‘B’ PM candidate
CSH indexing – candidate types Type III (appearance only) Image ‘A’ Image ‘B’
CSH indexing – candidate types Type 1 Type 2 Type 3 (> PM) (LSH)
CSH Algorithm – recap Preprocessing: • Project A’s and B’s patches on WH vectors Iterate (typically 1-3 times): • Hash patches into a table • For each patch in A • Create candidate patches in B • Pick the best one *further details in the paper A B
CSH Algorithm – recap Preprocessing: • Project A’s and B’s patches on WH vectors Iterate (typically 1-3 times): • Hash patches into a table • For each patch in A • Create candidate patches in B • Pick the best one *further details in the paper A B
CSH Algorithm – Hashing 1 6 5 2 4 3 3 4 2 1
CSH Algorithm – Hashing kernel (DC) Y channel – 32 bins kernel (DC) Cb channel – 4 bins kernel Y channel - 8 bins kernel Y channel - 2 bins
CSH Algorithm – Hashing • The reasoning being: • number of bins - relative to the dispersion of points along the projection line • Less bins the higher the kernel frequency • Less bins for color channels compared to Y • spacing of bins – attempting to capture a constant number of points in each bin • Use fixed percentiles • over the distribution
CSH Algorithm – recap Preprocessing: • Project A’s and B’s patches on WH vectors Iterate (typically 1-3 times): • Hash patches into a table • For each patch in A • Create candidate patches in B • Pick the best one *further details in the paper A B
Experiments - setup • Vid-Pairs data-set • 133 related pairs of ~2 MP images • Up to 30 frames apart in the video • Include both camera and object motion A B
Experiments – Image Reconstruction • Reconstruct image A, given: B and A->B • Each pixel is an avg. of corresponding pixels through the mapping • Main ingredient in many patch based algorithms (retargetting, inpainting, denoising, super-resolution, etc.) Image ‘A’ Image ‘B’ ? PM NNF CSH GT
Experiments – Image Reconstruction PM vs. GT CSH vs. GT
Experiments – Image Reconstruction Image ‘A’ Image ‘B’ ? NNF
Experiments – Image Reconstruction • Reconstruct image A, • given only image B and an NNF mapping A->B
Experiments – Image Reconstruction • Reconstruct image A, • given only image B and an NNF mapping A->B
Experiments – Image Reconstruction • Reconstruct image A, • given only image B and an NNF mapping A->B
Experiments – Image Reconstruction • Reconstruct image A, • given only image B and an NNF mapping A->B
Experiments – Image Reconstruction PM vs. GT CSH vs. GT
Experiments – Mapping (In)coherency Image ‘A’ Image ‘B’ CSH PM ‘x’-coordinate of PM mapping ‘x’-coordinate of CSH mapping
Reconstruction-Error vs. Incoherency Good (exact NN) bad
Summary - CSH • A new algorithm for NNF • combines appearance space and image space accuracy speed • other pleasing properties • Reconstruction, Incoherency, Texture • Limitations • Small data (images, patches) • Closely related image pairs • Additional memory for hash tables • Currently doesn’t support scale, rotation, illumination
Evaluation – CSH & PatchMatch • Well written. Self contained. • Novel. Technically convincing. • Well tested and evaluated. • Provide materials and code.
Future Work • The CSH shows inferior results with flat patches. Is it because of: (???) • The WH representation/functions • Noise in the flat patches • Distribution of selected/matching patches • Overcome limitations • “Smart” selection from a big set of data (images, patches) • Support scale, rotation, illumination by using a different representation such as {SIFT,DAISY,…}
Reference • PatchMatch: source code, data-set, more info, available at: • http://gfx.cs.princeton.edu/pubs/Barnes_2009_PAR/ • CSH: source code, data-set, more info, available at: • http://www.eng.tau.ac.il/~simonk/CSH • Huge gratitude to CSH team for their presentation published in their site.