Filling Algorithms

1. Filling Algorithms Pixelwise MRFs Chaos Mosaics • Patch segments are pasted, overlapping, across the image. • Then either: • Ambiguities are removed by smoothing (Chaos Mosaics-MSR). • Or a least cost path through the (chosen) overlapping images are found. Efros’01 uses dynamic programming, while Graphcut textures’03 uses… graphcut. Pixel distributions are determined by comparision with those with similar neighbourhoods. These distributions are sampled from or heuristics are performed on them to determine how to fill them.

2. Taken from Efros’ original presentation Synthesizing One Pixel • Assuming Markov property, what is conditional probability distribution of p, given the neighbourhood window? • Instead of constructing a model, let’s directly search the input image for all such neighbourhoods to produce a histogram for p • To synthesize p, just pick one match at random SAMPLE p Infinite sample image Generated image

3. Really Synthesizing One Pixel Taken from Efros’ original presentation SAMPLE p finite sample image Generated image • However, since our sample image is finite, an exact neighbourhood match might not be present • So we find the best match using SSD error (weighted by a Gaussian to emphasize local structure), and take all samples within some distance from that match

4. The pixel metric doesn’t matter One of the surprising things about this is that the choice pixel metric doesn’t seem to matter. Efros uses || . ||22 on RGB space, I’ve been using || . ||1, while Criminisi uses a metric based on the CIELab colour space. It doesn’t seem to matter which you use, presumably because we are only concerned with nearest neighbours, and they are all topologically equivalent.

5. Chaotic Mosaics B1 B1 B2 B2 Neighboring blocks constrained by overlap Minimal error boundary cut Dynamic Programming solution block

6. Chaotic Mosaics Graphcuts solution Takes full advantage of the power of graphcuts method, it treats the whole image as one patch and finds optimal joins along it. Pros: Finds optimal (and often seamless) matches Cons: Doesn’t find anything else, the recycling the optimal matches still leaves you with tiling artefacts.

7. Chaotic Mosaics Graphcuts solution Image Quilting Graphcuts

8. Pixel choice and Filling Algorithms • Onion skin or Outside in • The first. • The simplest? • Works with single textures or simple convex filling regions • Just picks away at the image one layer at a time

9. Pixel choice and Filling Algorithms • Linear structure propagation • Onion skin +pushing in on linear textures • Better than Onion skin for multi textural environments • When all you have is hammer, everything starts to look like a nail. ~ Artefacts from trying too hard. Missing Data Correction in Still Images and Image Sequences, Bornard et al. 2002

10. Pixel choice and Filling Algorithms • Linear structure propagation • Onion skin +pushing in on linear textures • Better than Onion skin for multi textural environments • When all you have is hammer, everything starts to look like a nail. ~ Artefacts from trying too hard. Missing Data Correction in Still Images and Image Sequences, Bornard et al. 2002

11. Filling Algorithms Onion Peel Vs. Linear propagation Now Onion Peel Push In

12. Pixel choice and Filling Algorithms • Max. entropy fill • Consistent with MRF assumptions. • Locally convex with a minimum of occlusions at point of fill. • Spirals in on simple shapes.

13. The same but quicker... Why Coarse to Fine? Standard efros 21 pixels Standard Efros 15 pixels Efros uniform pixel weighting 15 Course to fine 15 pixel nhood C2f uniform pixel weighting New metric Efros 15 pixels

14. Why Coarse to Fine? Structures...

15. Structures As Textures Why Coarse to Fine?

16. Structures As Textures Why Coarse to Fine?

17. Why Coarse to Fine? Texture as structure? ? Strong linear propagation comes from efros style fills naturally.

18. Not readily apparent due to the onion skin fill Efros Linear propagation The Efros Algorithm with my pixel choice

19. No guarantee it’s any better than linear propagation The algorithm often spots at the coarser levels that it has insufficient data to complete

20. No guarantee it’s any better than linear propagation Annoyingly, this problem is not solvable by more data, in the sense of higher resolution images. Information about how to propagate edges at the higher levels is still needed.

21. No guarantee it’s any better than linear propagation Annoyingly, this problem is not solvable by more data, in the sense of higher resolution images. Information about how to propagate edges at the higher levels is still needed.

22. No guarantee it’s any better than linear propagation Annoyingly, this problem is not solvable by more data, in the sense of higher resolution images. Information about how to propagate edges at the lowest resolution is still needed.

23. No guarantee it’s any better than linear propagation Annoyingly, this problem is not solvable by more data, in the sense of higher resolution images. Information about how to propagate edges at the lowest resolution is still needed.

24. No guarantee it’s any better than linear propagation Annoyingly this problem is not solvable by more data, in the sense of higher resolution images. Information about how to propagate edges at the lowest resolution is still needed.

25. No guarantee it’s any better than linear propagation Annoyingly, this problem is not solvable by more data, in the sense of higher resolution images. Information about how to propagate edges at the lowest resolution is still needed.

26. No guarantee it’s any better than linear propagation Compare it with a smaller fill

27. Why Coarse to Fine? Is manifold learning possible? Up to a point, we don’t care what colour the books are when propagating the shelf. Similar structural edge patterns are apparent everywhere. ?

28. Why Coarse to Fine?

29. Problems • Speed • Massively slower(hours rather than seconds) than patch based synthesis even with coarse to fine reducing neighbourhood size. • Can we reduce the search space via image segmentation? • Alternatively turn our soft coarse to fine constraints into something harder, by only testing pixels from the neighbourhoods of the k-closest fits at a coarser level.

30. Problems Surprisingly, this could even increase robustness by preventing the growth of miss-fittings