Download
1 / 38
380 likes | 509 Views
Image Congealing ( batch/multiple) image (alignment/registration ) Advanced Topics in Computer Vision (048921) Boris Kimelman. Introduction. Dramatic increase in popularity of image and video sharing sites Hard to measure image similarity: Illumination Occlusion Misalignment.
E N D
Image Congealing(batch/multiple) image (alignment/registration)Advanced Topics in Computer Vision (048921)Boris Kimelman
Introduction • Dramatic increase in popularity of image and video sharing sites • Hard to measure image similarity: • Illumination • Occlusion • Misalignment
Problem Definition • Align many images of an object to a fixed canonical template • Two main approaches: • Feature based • Similarity based • Mathematically: given images , compute transformation such that the images are aligned
Applications • Batch image alignment (congealing) • Identification pre-processing • Video stabilization • Background segmentation • Facial contour detection • Inpainting
Congealing example Input images Input images realigned using the transformations computed by RASL
Unsupervised Joint Alignment of Complex ImagesGary B Huang, Vidit Jain, Erik Learned-Miller ICCV 2007
Basic assumptions • Input images have similar structure and shape • Thus, low variability of pixel values at specific location • Distribution Field: empiricaldensity function at each pixel Pixel stack
Basic algorithm Input: Images Iterate: • Compute empirical distribution for image set • Find a transformation that reduces entropy of distribution field Output: aligned images, distribution fields Each stage increases image likelihood
Funneling: new image alignment • Add to training set and re-run • Instead, save sequence of distribution fields and increase likelihood of new image at each iteration Image Funnel New Image Aligned Image
Congealing Color Images • Attempt 1: choose to be color values • Attempt 2: choose as indicator of edge presence • Attempt 3: choose as SIFT descriptor
Congealing with SIFT descriptor (1) • Cluster SIFT descriptors using k-means • Congealing on hard assignments forces pixels to take relatively small number of values • Use soft assignment of pixels to clusters (GMM EM) • Analogy with grayscaleusing binary alphabet
Congealing with SIFT descriptor (2) Window around pixel SIFT vector and clusters Posterior distribution
Labeled Faces in the Wild database • 13233 images • Size: 250X250X16MB • 5749 people • 1680 people with two or more images
Align for identification Hyper feature based identifier
Evaluation • LFW database contribution • Novel: Information theory point of view Funneling process Demo code available • Results: No measure of alignment accuracy Comparison only against face alignment algorithm • Writing level: convincing illustrations would help
RASL: Robust Alignment by Sparse and Low-Rank Decomposition for Linearly Correlated Images YigangPeng, ArvindBalasubramanian, John Wright, Ma YiCVPR 2011
How to measure image similarity? • Learned-Miller: Minimize sum of entropies of pixel stacks • Least-Squares congealing: minimize norm between images • If the criterion is satisfied exactly the matrix rank is 1 Learned-Miller Generalize: lower rank as much as possible Least Squares
Basic Assumptions • Input images exhibit high linear correlation • If are n aligned images then is low rank: • A practical assumption: • Errors are large in magnitude but sparse
Mathematical Formulation • The model is: • is a low rank matrix that models image batch linear structure • is a matrix of large but sparse errors that models: corruption, occlusion, shadows
Graphical Explanation Matrix of corrupted observations Underlying low-rank matrix Sparse error matrix
Modeling Misalignment • Above model depends on the assumption that images are aligned • Instead of observing , we observe Problem: given observations recover images and transformations
Optimization Formulation (1) • Find a low-rank matrix and a sparse matrix such that • In Lagrangianform:
Optimization Formulation (2) • Cost function is highly non-convex and discontinuous • Replace the cost function with its convex surrogate:
Nuclear norm • is the nuclear norm: • are the matrix singular values • Resembles -norm replacement by -norm
Constraint Linearization • Linearize non-linear constraint and iterate: , is standard basis Problem to solve:
RASL Algorithm • Input: images • Iterate: • Compute jacobian: • Warp: • Solve: • Update: • Output:
Region of Attraction • Perturb each image by a transformation: Alignment is successful if maximum difference of the eye corners across all pairs of images is less than one pixel in the canonical frame
Evaluation • Novel: Unifying framework for image congealing Rank minimization as image similarity Code available • Results: Comprehensive algorithm assessment Compare only against one algorithm Extensive site about rank minimization • Writing level: Convincing Advanced mathematics required (optimization)
Future issues • Multi sensor congealing: complex relationship between corresponding pixels • Learned Miller – occlusion removal by interspace alignment • RASL • mix between image spaces • funneling
