1 / 48

Mixed-Resolution Patch-Matching (MRPM)

Mixed-Resolution Patch-Matching (MRPM). Harshit Sureka and P.J. Narayanan (ECCV 2012). Presentation by Yaniv Romano. Problem Definition. Patch Matching Nearest Neighbor Fields: F or every patch in image A, find the K most similar patches in image B. A. B. NNF.

ivie
Download Presentation

Mixed-Resolution Patch-Matching (MRPM)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Mixed-Resolution Patch-Matching (MRPM) HarshitSureka and P.J. Narayanan (ECCV 2012) Presentation by Yaniv Romano

  2. Problem Definition • Patch Matching \ Nearest Neighbor Fields: • For every patch in image A, find the K most similar patches in image B. A B NNF

  3. Motivation – BM3D Denoising[K. Dabovet al, IEEE-TIP 2007]

  4. Motivation – Super Resolution[M. Protter and M. Elad, IEEE-TIP 2009] Up-scaling by a factor of 2 - “known” pixel

  5. Motivation – Deinterlace[M. Protterand M. Elad, IEEE-TIP 2009]

  6. Motivation – Deinterlace[M. Protterand M. Elad, IEEE-TIP 2009]

  7. Motivation – Image Editing[Barnes et al, SIGRAPH 2009]

  8. Motivation – Image Editing[Barnes et al, SIGRAPH 2009]

  9. Motivation – Image Editing[Barnes et al, SIGRAPH 2009]

  10. Challenges ? *very-low extra-memory

  11. Today’s talk • Problem Definition. • Motivation. • Challenges. • Related works. • Pyramid Patch-Matching (PPM). • Mixed Resolution Pyramid Patch-Matching (MRPM). • Experimental Results. • Pros & Cons. • Future ideas.

  12. An iterative randomized algorithm. Based on local propagation and mostly-local search. Heavily relies on the fact that images are generally coherent. Image Space - the PatchMatch algorithm [Barnes et al, SIGRAPH 2009] left neighbor current match of left neighbor candidate

  13. An iterative randomized algorithm. Based on local propagation and mostly-local search. Heavily relies on the fact that images are generally coherent. Image Space - the PatchMatch algorithm [Barnes et al, SIGRAPH 2009] current match of top neighbor top neighbor left neighbor current match of left neighbor candidates

  14. Image Space - the PatchMatch algorithm [Barnes et al, SIGRAPH 2009] *very low extra-memory *Compared to the state-of-the-art (CSH)

  15. Image Space & Appearance Space CSH algorithm[Korman et al, ICCV 2011] • Hash patches of both images into a table. • Find matches, based on: • Appearance. • Coherence. Type 1 Type 2 Appearance & Coherence Appearance Appearance Type 3 PM

  16. Image Space & Appearance Space CSH algorithm[Korman et al, ICCV 2011] *medium-low extra-memory *Compared to previous methods.

  17. Pyramid Patch-Matching

  18. Image Pyramid • Down sampling the input image(s) • Gaussian averaging, area averaging, cubic interpolation, and others. …

  19. Pyramid Patch-Matching (PPM)

  20. Pyramid Patch-Matching (PPM)

  21. Pyramid Patch-Matching (PPM)

  22. Pyramid Patch-Matching (PPM)

  23. Pyramid Patch-Matching (PPM) Source Image Target Image

  24. Pyramid Patch-Matching (PPM) Source Image Exhaustive Search Target Image Upsampling Upsampling & Expanding Search 1 2 Find matches 3

  25. Pyramid Patch-Matching (PPM) Source Image Exhaustive Search Target Image Upsampling Upsampling & Expanding Search 1 2 Find matches 3 4

  26. Pyramid Patch-Matching (PPM) A , B = Input images Ap, Bp = level p of Pyramid(A) and Pyramid(B) matchessearch* at coarsest resolution level for p = P-1 to 0 • upsampling & expand matches from level p-1 • matchessearch* within this range Output: matches for each patch. *Possible to use other patch-matching and nearest-neighbor algorithms for search.

  27. PPM – Pros • Very simple. • Fast exhaustive search at the coarsest level. • Local searches at finer levels.

  28. PPM – Cons • Small regions can disappear. • The vicinity of a missed low resolution patch may not be searched in higher resolutions. • Lower resolution versions may lead to:  A smooth distance function.  Poor localization of matches.

  29. Mixed Resolution Pyramid Patch-Matching … … Upsampling Upsampling ?

  30. Mixed Resolution Pyramid Patch-Matching • The algorithm is the same as before with the following major difference: • UseMixed-Resolution • Vectors instead of the traditional patch vectors ? ?

  31. Pyramid Patch-Matching Exhaustive Search Upsampling Upsampling & Expanding Search 1 2 Find matches 3 4

  32. Mixed Resolution Pyramid Patch-Matching Exhaustive Search Upsampling Upsampling & Expanding Search 1 3 Find matches 4 2

  33. Mixed Resolution Pyramid Patch-Matching Exhaustive Search Upsampling Upsampling & Expanding Search 1 3 Find matches 4 2

  34. Search Range vs. Mixed Resolutions • Mixing resolutions is more effective than increasing the search range. Lower errorvalues are achieved with less cost of time by mixing resolutions *Error = Average Euclidean distance between source and matched patches

  35. GPU Implementation • Mixed resolution patch matching processes each pixel independently. • Perfect for the heavy parallel architecture of GPU.

  36. GPU Implementation Run time for K=5 NN, patch size = 7X7, coarsest resolution 32X32 :

  37. Experimental Results • Ground Truth - exhaustive search: • K=10 NN. • patch size = 8X8. • Error = Avg. Euclidian distance between the patches and their matches. • Mixed pixels from two resolution levels. • 3X3 search range.

  38. How many of the ground truth matches are found? • Higher % implies better accuracy: Advantages of the MRPM method are more pronounced when several nearest matches are needed

  39. Average Error • With a similar computation time to CSH they obtain lower average error:

  40. Image Reconstruction • Find a dense patch matching between A & B: • Reconstruct image A using image B. • Reconstruction Mean Square Error, lower is better: The proposed Image Prior is weaker than CSH (coherency)

  41. Bottom line *medium extra-memory *Compared to CSH

  42. Mixed Resolution Pyramid Patch-Matching • Pros: • The algorithm: • Faster than the state-of-the-art. • Fast exhaustive search at the coarsest level. • Local searches at finer levels. • More accurate than the state-of-the-art. • Take into consideration the fine details of the patches. • Very simple and easy to implement. • Parallel property, matching is independent. • The article: • Self-contained. • Provides code (both CPU and GPU implementations). • The comparisons to previous works are sufficient. • Builds the story very well.

  43. Mixed Resolution Pyramid Patch-Matching • Cons: • The algorithm: • Decisions made early cannot be reversed later. • Does not use image coherency prior explicitly. • If a NN found, they do not exploit this for its neighbors. • May compare the same information several times. • Redundant data: • image, 3 levels pyramid: • Does not invariant to illumination, rotation etc. • The article: • Cumbersome experiments. • Doesn’t provide a deep explanation why the CSH is better in the image reconstruction test. • Doesn’t show a visual comparison between these images.

  44. Future ideas • Future idea #1: • Instead of image pyramid…

  45. Future ideas • Future idea #1: • Use the mixed vectors with “Wavelets”. The mixed vectors contain the whole image details compared to image pyramid

  46. Future ideas Exhaustive Search • Future idea #1 – reduce data redundancy: • Use the mixed vectors with “Wavelets”. • Future idea #2 – strengthen the image prior: • Use image coherency. Upsampling Upsampling & Expanding Search 1 3 Find matches 4 2

  47. We Are Done ! Thank You

More Related