1 / 33

Image Completion using Global Optimization

Image Completion using Global Optimization. Presented by Tingfan Wu. The Image Inpainting Problem. Outline. Introduction History of Inpainting Camps – Greedy & Global Opt. Model and Algorithm Markov Random Fields (MRF) & Inpainting Belief Propagation (BP) Priority BP Results

Antony
Download Presentation

Image Completion using Global Optimization

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Image Completion using Global Optimization Presented by Tingfan Wu

  2. The Image Inpainting Problem

  3. Outline • Introduction • History of Inpainting • Camps – Greedy & Global Opt. • Model and Algorithm • Markov Random Fields (MRF) & Inpainting • Belief Propagation (BP) • Priority BP • Results • Structural Propagation

  4. Method Type PriorityTexture Synth. Need User Guidance

  5. Exampled Based Method—Jigsaw Puzzle PatchesNot Available

  6. Method Type PriorityTexture Synth. Need User Guidance

  7. Ooops Greedy v.s Global Optmization Greedy Method Global Optimization Refine Globally  Cannot go back 

  8. Outline • Introduction • History of Inpainting • Camps – Greedy & Global Opt. • Model and Algorithm • Markov Random Fields (MRF) & Inpainting • Belief Propagation (BP) • Priority BP • Results • Structural Propagation

  9. Random Fields / Belief Network Random Variable(Observation) • RF：Random Variables on Graph • Node : Random Var. (Hidden State) • Belief : from Neighbors, and Observation Good Project Writer?(High Project grade) Smart Student?(High GPA) Good Test Taker?(High test score) Good Employee (No Observation yet) Edge: Dependency

  10. Story about MRF • (Bayesian) Belief Network (DAG) • Markov Random Fields (Undirected, Loopy) • Special Case: • 1D - Hidden Markov Model (HMM) Hidden Markov Model (HMM) Office Helper Wizard

  11. Inpainting as MRF optimization • Node : Grid on target region, overlapped patches • Edge : A node depends only on its neighbors • Optimal labeling (hidden state) that minimizing mismatch energy

  12. MRF Potential Functions Mismatch (Energy) between .. • Vp (Xp ) : Source Image vs. New Label • Vpq(Xp, Xq) : Adjacent Labels • Sum of Square Distances (SSD) in Overlapping Region

  13. Global Optimizatoin min

  14. Outline • Introduction • History of Inpainting • Camps – Greedy & Global Opt. • Model and Algorithm • Markov Random Fields (MRF) & Inpainting • Belief Propagation (BP) • Priority BP • Results • Structural Propagation

  15. Belief Propagation(1/3) Good Project Writer?(High Project grade) Smart Student?(High GPA) Good Test Taker?(High test score) Good Employee (No Observation yet) • Undirected and Loopy • Propagate forward and backward

  16. X X q p Belief Propagation(2/3) • Message Forwarding • Iterative algorithm until converge O(|Candidate|2) Candidates at Node Q Candidates at Node P Neighbors (P)

  17. Belief Propagation(3/3)

  18. Priority BP • BP too slow: • Huge #candidates  Timemsg = O(|Candidates|2) • Huge #Pairs Cannot cache pairwise SSDs. • Observations • Non-Informative messages in unfilled regions • Solution to some nodes is obvious (fewer candidates.)

  19. Human Wisdom Candidates Start from non-ambiguous part And Search for Brown feather+green grass Nobody start from here

  20. Priority BP • Observations • needless messages in unfilled regions • Solution to some nodes is obvious (fewer candidates.) • Solution: Enhanced BP: • Easy nodes goes first (priority message scheduling) • Keep only highly possible candidates (maintain a Active Set)

  21. ? ? ? ? ? ? ? ? Priority & Pruning Discard Blue Points High Priorityprune a lot Low Priority Candidates sorted by relative belief Pruning may miss correct label

  22. #Candidates after Pruning Active Set (Darker means smaller) Histogram of #candidates Similar candidates

  23. A closer look at Priority BP • Priority Calculation • Priority : 1/(#significant candidate) • Pruning (on the fly ) • Discard Low Confidence Candidates • Similar patches  One representative (by clustering) • Result • More Confident More Pruning • Confident node helps increase neighbor’s confidence. • Warning: • PBP and Pruning must be used together

  24. Outline • Introduction • History of Inpainting • Camps – Greedy & Global Opt. • Model and Algorithm • Markov Random Fields (MRF) & Inpainting • Belief Propagation (BP) • Priority BP • Results • Conclusion • Structural Propagation

  25. Results-Inpainting(1/3) Darker pixels  higher priority Automatically start from salient parts.

  26. Results-Inpainting(2/3)

  27. Results-Inpainting(3/3) • Up to 2minutes / image (256x170) on P4-2.4G

  28. More : Texture Synthesis • Interpolation as well as extrapolation

  29. Conclusion • Priority BP • {Confident node first} + {candidate pruning} • Generic – applicable to other MRF problems. • Speed up • MRF for Inpainting • Global optimization • avoid visually inconsistence by greedy • Priority BP for Inpainting • Automatically start from salient point.

  30. Sometimes … • Image contains hard high-level structure • Hard for computers • Interactive completion guided by human.

  31. Potential Func. For Structural Propagation • User input a guideline by human region. • Potential Function respect distance between lines Jian Sun et al, SIGGRAPH 2005

  32. Video • Link:Microsoft Research

More Related