1 / 36

Application into Image Inpainting

Application into Image Inpainting. Criminisi, A.; Perez, P.; Toyama, K., "Region filling and object removal by exemplar-based image inpainting," IEEE Trans. on Image Proc. , vol.13, no.9, pp. 1200- 1212, Sept. 2004. At the Intersection. Error Concealment, Image Recovery … ….

lin
Download Presentation

Application into Image Inpainting

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Application into Image Inpainting Criminisi, A.; Perez, P.; Toyama, K., "Region filling and object removal by exemplar-based image inpainting," IEEE Trans. onImage Proc., vol.13, no.9, pp. 1200- 1212, Sept. 2004

  2. At the Intersection

  3. Error Concealment, Image Recovery … …

  4. Importance of Scanning Order

  5. Heuristic Rules Geometric cue: Convexity Photometric cue: gradient Green/red color: high (low) priority pixels

  6. Image Inpainting Specific Scanning order Repair by Nonparametric sampling Repaired image Image to repaired

  7. Synthetic Image Example Initial Intermediate Final Kanisza Triangle

  8. Real-world Image Example

  9. Computer Assignment 5 • Implement Efros-Leung’s scheme • Test it on image inpainting (you might want to try different textures) • When does it work and when does it fail? • Computational complexity reduction

  10. Open Questions • Where to inpaint? • Sequential vs. parallel inpainting • Performance evaluation • Seeing is not always believing • Failure examples • Few researchers report them • Comparison with other approaches • Why does A work better than B?

  11. Error Concealment in Image Communication image video Sequential Parallel

  12. Why Parallel? • Avoid choosing scanning order and its associated error propagation problem • Modeling multimedia signal at a patch (e.g., speech frame, image block) level instead of data (e.g., speech sample, image pixel) level • Parallelism has to work with distributed processing together (redundancy is exploited to fight against uncertainty)

  13. RPP B1 B3 B2 B0 B4 Why Nonlocal? Local neighbors in range (photometric similarity) nonlocal patches in domain (geometric proximity)

  14. Algorithm Flowchart Projection onto observation constraint Projection onto prior constraint Data clustering in Rp Coring algorithm Outer loop Inner loop Better estimate of local neighborhood in the patch space Better estimate of missing data in the time domain

  15. B0 B1 Bk … Pack into 3D Array D 3D-FFT Thresholding ^ ^ ^ B0 B1 Bk … Pack into 3D Array D 3D-IFFT Sparsity Enforcement Projection onto prior constraint

  16. Image Examples exemplar based Guleryuz’s scheme our work exemplar based Guleryuz’s scheme our work original damaged original damaged

  17. Parallel and Distributed Processing (PDP) • Dominant form of Connectionism (models mental or behavioral phenomena as the emergent processes of interconnected networks of simple units.) • More widely known as neural networks since 1980s

  18. ? + = Image Quiltingfor Texture Synthesis & Transfer Alexei Efros and Bill Freeman (SIGGRAPH’2001)

  19. p Pixel-level Nonparametric Sampling non-parametric sampling • Assuming Markov property, compute P(p|N(p)) • Building explicit probability tables infeasible • Instead, let’s search the input image for all similar neighborhoods — that’s our histogram for p • To synthesize p, just pick one match at random Input image Synthesizing a pixel

  20. Block-level Chaos Mosaic • Process: 1) tile input image; 2) pick random blocks and place them in random locations 3) Smooth edges input idea result

  21. Comparison with Periodic Mosaic chaotic periodic

  22. Doesn’t work for structured textures input result

  23. B p Synthesizing a block • Idea: unit of synthesis = block • Exactly the same but now we want P(B|N(B)) • Much faster: synthesize all pixels in a block at once • Not the same as multi-scale! Image Quilting=Nonparametric Sampling+Chaos Mosaic non-parametric sampling • Observation: neighbour pixels are highly correlated Input image

  24. B1 B1 B2 B2 Neighboring blocks constrained by overlap Minimal error boundary cut block Input texture B1 B2 Random placement of blocks

  25. 2 _ = overlap error min. error boundary Minimal error boundary overlapping blocks vertical boundary

  26. Search for the Shortest Path end start

  27. An Elegant Solution • Exhaustive search is computationally infeasible • Fortunately, there exists a fast and optimum solution known as Dynamic Programming (DP) in Operations Research and Viterbi Algorithm (VA) in communication

  28. DP/VA Example start end Basic Idea: Only keep the best Q survivors in sequential decision

  29. DP/VA Step-by-Step Result

  30. Image Quilting Algorithm • Pick size of block and size of overlap • Synthesize blocks in raster order • Search input texture for block that satisfies minimum error constraints (above and left) • Paste new block into resulting texture

  31. + =

  32. parmesan + = rice + =

  33. Summary for Texture Synthesis • Parametric approach • Histogram/high order statistics matching in the wavelet/steerable pyramid space • Nonparametric approach • Conceptually simple and effective • Interesting application into image inpainting & quilting • Produce visually convincing results • Parametric vs. nonparametric: which is the winner? • It depends on the application • How to combine them together?

More Related