1 / 38

Introduction

Image Congealing ( batch/multiple) image (alignment/registration ) Advanced Topics in Computer Vision (048921) Boris Kimelman. Introduction. Dramatic increase in popularity of image and video sharing sites Hard to measure image similarity: Illumination Occlusion Misalignment.

barth
Download Presentation

Introduction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Image Congealing(batch/multiple) image (alignment/registration)Advanced Topics in Computer Vision (048921)Boris Kimelman

  2. Introduction • Dramatic increase in popularity of image and video sharing sites • Hard to measure image similarity: • Illumination • Occlusion • Misalignment

  3. Problem Definition • Align many images of an object to a fixed canonical template • Two main approaches: • Feature based • Similarity based • Mathematically: given images , compute transformation such that the images are aligned

  4. Applications • Batch image alignment (congealing) • Identification pre-processing • Video stabilization • Background segmentation • Facial contour detection • Inpainting

  5. Congealing example Input images Input images realigned using the transformations computed by RASL

  6. Unsupervised Joint Alignment of Complex ImagesGary B Huang, Vidit Jain, Erik Learned-Miller ICCV 2007

  7. Basic assumptions • Input images have similar structure and shape • Thus, low variability of pixel values at specific location • Distribution Field: empiricaldensity function at each pixel Pixel stack

  8. Basic algorithm Input: Images Iterate: • Compute empirical distribution for image set • Find a transformation that reduces entropy of distribution field Output: aligned images, distribution fields Each stage increases image likelihood

  9. Funneling: new image alignment • Add to training set and re-run • Instead, save sequence of distribution fields and increase likelihood of new image at each iteration Image Funnel New Image Aligned Image

  10. Congealing Color Images • Attempt 1: choose to be color values • Attempt 2: choose as indicator of edge presence • Attempt 3: choose as SIFT descriptor

  11. Congealing with SIFT descriptor (1) • Cluster SIFT descriptors using k-means • Congealing on hard assignments forces pixels to take relatively small number of values • Use soft assignment of pixels to clusters (GMM EM) • Analogy with grayscaleusing binary alphabet

  12. Congealing with SIFT descriptor (2) Window around pixel SIFT vector and clusters Posterior distribution

  13. Mathematical formulation

  14. Labeled Faces in the Wild database • 13233 images • Size: 250X250X16MB • 5749 people • 1680 people with two or more images

  15. Results on faces

  16. Align for identification Hyper feature based identifier

  17. Results on cars

  18. Evaluation • LFW database contribution • Novel:  Information theory point of view  Funneling process  Demo code available • Results:  No measure of alignment accuracy  Comparison only against face alignment algorithm • Writing level:  convincing  illustrations would help

  19. RASL: Robust Alignment by Sparse and Low-Rank Decomposition for Linearly Correlated Images YigangPeng, ArvindBalasubramanian, John Wright, Ma YiCVPR 2011

  20. How to measure image similarity? • Learned-Miller: Minimize sum of entropies of pixel stacks • Least-Squares congealing: minimize norm between images • If the criterion is satisfied exactly the matrix rank is 1 Learned-Miller Generalize: lower rank as much as possible Least Squares

  21. Basic Assumptions • Input images exhibit high linear correlation • If are n aligned images then is low rank: • A practical assumption: • Errors are large in magnitude but sparse

  22. Mathematical Formulation • The model is: • is a low rank matrix that models image batch linear structure • is a matrix of large but sparse errors that models: corruption, occlusion, shadows

  23. Graphical Explanation Matrix of corrupted observations Underlying low-rank matrix Sparse error matrix

  24. Modeling Misalignment • Above model depends on the assumption that images are aligned • Instead of observing , we observe Problem: given observations recover images and transformations

  25. Optimization Formulation (1) • Find a low-rank matrix and a sparse matrix such that • In Lagrangianform:

  26. Optimization Formulation (2) • Cost function is highly non-convex and discontinuous • Replace the cost function with its convex surrogate:

  27. Nuclear norm • is the nuclear norm: • are the matrix singular values • Resembles -norm replacement by -norm

  28. Constraint Linearization • Linearize non-linear constraint and iterate: , is standard basis Problem to solve:

  29. RASL Algorithm • Input: images • Iterate: • Compute jacobian: • Warp: • Solve: • Update: • Output:

  30. Region of Attraction • Perturb each image by a transformation: Alignment is successful if maximum difference of the eye corners across all pairs of images is less than one pixel in the canonical frame

  31. Results on controlled data set

  32. Results on LFW

  33. Stabilizing video frames

  34. Aligning planar surfaces despite occlusions

  35. Evaluation • Novel:  Unifying framework for image congealing  Rank minimization as image similarity  Code available • Results:  Comprehensive algorithm assessment Compare only against one algorithm  Extensive site about rank minimization • Writing level:  Convincing Advanced mathematics required (optimization)

  36. Comparison of Papers

  37. Future issues • Multi sensor congealing: complex relationship between corresponding pixels • Learned Miller – occlusion removal by interspace alignment • RASL • mix between image spaces • funneling

  38. Thank You

More Related