680 likes | 825 Views
Energy Minimization with Label Costs and Model Fitting. presented by Yuri Boykov co-authors: Andrew Delong Anton Osokin Hossam Isack. Overview. Standard models in vision (focus on discrete case) MRF/CRF, weak-membrane, discontinuity-preserving...
E N D
Energy Minimization with Label Costsand Model Fitting presented by Yuri Boykov co-authors: Andrew Delong Anton OsokinHossamIsack
Overview • Standard models in vision (focus on discrete case) • MRF/CRF, weak-membrane, discontinuity-preserving... • Information-based: MDL (Zhu&Yuille’96) , AIC/BIC (Li’07) • Label costs and their optimization • LP-relaxations, heuristics, α-expansion++ • Model Fitting • dealing with infinite number of labels ( PEARL ) • Applications • unsupervised image segmentation • geometric model fitting (lines, circles, planes, homographies, ...) • rigid motion estimation • extensions…
Reconstruction in Vision: (a basic example) I L observed noisy image I image labeling L (restored intensities) I= { I1, I2 , ... , In } L= { L1, L2 , ... , Ln} How to compute L from I ?
Energy minimization(discrete approach) • MRF framework • weak membrane model(Geman&Geman’84, Blake&Zisserman’83,87) data fidelity spatial regularization discontinuity preserving potentials Blake&Zisserman’83,87
Optimization • Convex regularization • gradient descent works • exact polynomial algorithms • TV regularization • a bit harder (non-differentiable) • global minima algorithms (Ishikawa, Hochbaum, Nikolova et al.) • Robust regularization • NP-hard, many local minima • good approximations (message passing, a-expansion)
Potts model(piece-wise constant labeling) maxflow/mincut combinatorial algorithms • Robust regularization • NP-hard, many local minima • provably good approximations (a-expansion)
Potts model(piece-wise constant labeling) depth layers Right eye image Left eye image maxflow/mincut combinatorial algorithms • Robust regularization • NP-hard, many local minima • provably good approximations (a-expansion)
Potts model(piece-wise constant labeling) C maxflow/mincut combinatorial algorithms • Robust regularization • NP-hard, many local minima • provably good approximations (a-expansion)
Adding label costs - set of labels allowed at each point p • Lippert[PAMI 89] • MDL framework, annealing • Zhu and Yuille[PAMI 96] • continuous formulation (gradient des cent ) • H. Li [CVPR 2007] • AIC/BIC framework, only 1st and 3rd terms • LP relaxation (no guarantees approximation) • Our new work [CVPR 2010] , extended a-expansion • all 3 terms, 3rd term is represented as some high-order clique), optimality bound • very fast heuristics for 1st & 3rd term (facility location problem, 60-es)
The rest of the talk… Why label costs?
Model fitting y=ax+b SSD
many outliers quadratic errors fail use more robust error measures, e.g. gives “MEDIAN” line - more expensive computations (non-differentiable) - still fails if outliers exceed 50% RANSAC
many outliers sample randomly two points, get a line RANSAC
many outliers sample randomly two points, get a line 2. count inliers for threshold T 10 inliers RANSAC
many outliers sample randomly two points, get a line 2. count inliers for threshold T 30 inliers 3. repeat N times and select model with most inliers RANSAC
Multiple models and many outliers Why not RANSAC again?
Multiple models and many outliers Higher noise Why not RANSAC again? In general, maximization of inliers does not work for outliers + multiple models
Energy-based approach energy-based interpretation of RANSACcriteria for single model fitting: - find optimal labelL for one very specific error measure
Energy-based approach If multiple models - assign different models (labels Lp) to every point p - find optimal labeling L = { L1, L2 , ... , Ln} Need regularization!
Energy-based approach If multiple models - assign different models (labels Lp) to every point p - find optimal labeling L = { L1, L2 , ... , Ln}
Energy-based approach If multiple models - assign different models (labels Lp) to every point p - find optimal labeling L = { L1, L2 , ... , Ln} - set of labels allowed at each point p
Energy-based approach If multiple models - assign different models (labels Lp) to every point p - find optimal labeling L = { L1, L2 , ... , Ln} Practical problem: number of potential labels (models) is huge, how are we going to use a-expansion?
PEARL • Propose • Expand • And • Reestimate • Labels data points
PEARL • Propose • Expand • And • Reestimate • Labels sample data to generate a finite set of initial labels data points + randomly sampled models
PEARL • Propose • Expand • And • Reestimate • Labels a-expansion: minimize E(L) segmentation for fixed set of labels models and inliers (labeling L)
PEARL • Propose • Expand • And • Reestimate • Labels reestimating labels in for given inliers minimizes first term of energy E(L) models and inliers (labeling L)
PEARL • Propose • Expand • And • Reestimate • Labels a-expansion: minimize E(L) segmentation for fixed set of labels models and inliers (labeling L)
PEARL • Propose • Expand • And • Reestimate • Labels after 5 iterations iterate until convergence
PEARL can significantlyimprove initial models single line fitting with 80% outliers deviation (from ground truth ) number of initial samples
Comparison formulti-model fitting Low noise original data points
Comparison formulti-model fitting Low noise some generalization of RANSAC
Comparison formulti-model fitting Low noise PEARL
Comparison formulti-model fitting High noise original data points
Comparison formulti-model fitting High noise Some generalization of RANSAC (Multi-RANSAC, Zuliani et al. ICIP’05)
Comparison formulti-model fitting High noise Other generalization of RANSAC (J-linkage, Toldo & Fusiello, ECCV’08)
Comparison formulti-model fitting High noise Hough transform Finding modes in Hough-space, e.g. via mean-shift (also maximizes the number of inliers)
Comparison formulti-model fitting High noise PEARL
Fitting circles regularization with label costs only Here spatial regularization does not work well
Fitting planes (homographies) Original image (one of 2 views)
Fitting planes (homographies) (a) Label costs only
Fitting planes (homographies) (b) Spatial regularity only
Fitting planes (homographies) (c) Spatial regularity + label costs
(unsupervised) Image Segmentation Original image
(unsupervised) Image Segmentation (a) Label costs only [Li, CVPR 2007]
(unsupervised) Image Segmentation (b) Spatial regularity only [Zabih&Kolmogorov CVPR 04]
(unsupervised) Image Segmentation Zhu and Yuille 96 used continuous variational formulation (gradient discent) (c) Spatial regularity + label costs
(unsupervised) Image Segmentation (c) Spatial regularity + label costs
(unsupervised) Image Segmentation Spatial regularity + label costs
(unsupervised) Image Segmentation Spatial regularity + label costs