Objective:

2. Objective: • Find correspondence between two image • Why? • Motion analysis • Tracking • Stereo • 3D reconstruction • Object detection • Image enhancement • Video editing • … • … Reference Source Correspondence

3. Why else? Restoration • Huge digital photo database availableon line. • A lot of redundancy in personalphoto collections. 3D Deblurring

4. Correspondence-Land: Assumptions dViewpoint = Small dTime = Small Commoncontent Non-rigid transform dViewpoint = Small Mostly rigid objects Result: Dense: Optical Flow Stereo reconstruction Sparse: SIFT Dense

5. Main contestants – Sparse SIFT: • Find Feature points + descriptors. • Use Scale to get region. • Dense result! • Properties: • Robust and accurate! • Limitations: • Smooth regions undetected • Fixed scale = bad matches.

6. Main contestants – SIFT Flow: Optical Flow Energy: • Dense  • Handles large geometrical • distortions poorly  SIFTFlow Energy:

7. Main algorithm - abstract: For scale = coarse to fine do end

8. Nearest-neighbor search: Objective: For each 8x8 patch Find Where Features per pixel: Final wanted result: For example:

9. PatchMatchCorrespondence Algorithm: • Iterative two stage operation: • 1. Random Search • 2. Propagation Barnes, Connelly, et al. "PatchMatch: a randomized correspondence algorithm for structural image editing." ACM Transactions on Graphics-TOG 28.3 (2009): 24.

10. Generalized PatchMatch: • PatchMatchGeneralizedPatchMatch • Searching for : • Candidates: • Example: Barnes, Connelly, et al. "The generalized patchmatch correspondence algorithm.“ Computer Vision–ECCV 2010. Springer Berlin Heidelberg, 2010. 29-43.

11. Nearest-neighbor search – cont.: Objective: For each 8x8 patch Find Where - Extracted with PatchMatch. Bias and Gain:

12. Aggregating consistent regions: Objective: Find reliable matches, remove outliers. Define a consistency error measure:

13. Aggregating consistent regions: Nearest neighbor search result:

14. Aggregating consistent regions: 1.2 1.7 3.6 0.5 Apply consistency threshold:

15. Aggregating consistent regions: Remove incoherentregions:

16. Progress so far: After NN search After region aggregation Source

17. Global color mapping: • Why? • Improve performance of correspondencealgorithm. • Produce color matched reference image. • Color Model: • Piecewise cubic spline with 7 breaks per RGB channel. Red Channel 80 50

18. Search constraints: • Our transformation contains 12 DOF Match ambiguity increased. • Solution: Limit search range for . • Constrained Locally, inside reliable regions. • of LabG Constrained Globally, usingreliable regions.

19. Evaluation - Correspondence: • Compared to: • GPM – Their setup, same features ( ) • But without: Multi-scale, region aggregation, search narrowing. • Sparse SIFT – 15 pixels radios around the descriptor centers. • SIFT-Flow – [Liu et al. 2008a] Viewpoint Display Metric: Correct if Zoom+rotation

20. Evaluation - Correspondence: Viewpoint Zoom + rotation

21. Evaluation - Correspondence:

22. Evaluation - Correspondence: Correct if Mikolajczyk et al. Dataset

23. Evaluation – Global color transfer: SIFT, Main Idea: Global color transfer, SIFT scale. Pitie, Main Idea: 1D pdf matching + Gradient Adj.

24. Evaluation – Global color transfer:

25. Applications – Local color transfer: • Global– transform using original Global transform. • Local- 1. Apply local adjustments inside reliable regions (Histogram • matching). • 2. Propagate to other pixels using Poisson blending.

26. Applications – Mask transfer: • Find correspondence of Background and Foreground patches. • Use result as initialization for Grab-Cut [Rother et al. 2004]

27. Applications – Deblurring: • Kernel Estimation: using modified Cho and Lee [2009] with a validity mask. • Deconvolution: using modified Levin et al. [2007]. • Source is color matched.

28. For Conclusion:: • Pros: • Authors present a novel useful correspondence method. • Consistent regions aggregation – Great Idea! • Cons: • Experimental evaluation – Lacks large scale comparison, Focused only on superior cases. • Did not prove superiority over Co-Recognition*

29. Future directions: • Aid dense 3D reconstruction? • Color transfer between different images (same composition)? • Extract geometric properties of objects in the scene. Thank you!