580 likes | 835 Views
P 3 & Beyond Solving Energies with Higher Order Cliques. Pushmeet Kohli Pawan Kumar Philip H. S. Torr Oxford Brookes University CVPR 2007. Energy Functions. MAP Inference. Observed Variables. Hidden Variables. Energy Functions. MAP Inference. Energy Minimization.
E N D
P3 & BeyondSolving Energies with Higher Order Cliques Pushmeet Kohli Pawan Kumar Philip H. S. Torr Oxford Brookes University CVPR 2007
Energy Functions MAP Inference Observed Variables Hidden Variables
Energy Functions MAP Inference Energy Minimization
Energy Functions • Pairwise Energy Functions Unary Pairwise
Energy Functions • Pairwise Energy Functions Unary Pairwise
Energy Functions • Pairwise Energy Functions Unary Pairwise • Efficient Algorithms for Minimization • Message Passing • (BP, TRW) • Move Making • (Expansion/Swap)
Energy Functions • Pairwise Energy Functions Unary Pairwise • Efficient Algorithms for Minimization • Message Passing • (BP, TRW) • Move Making • (Expansion/Swap) Restricted Expressive Power!
Energy Functions • Higher Order Energy Functions Unary Pairwise Higher order More expressive than pairwise FOE: Field of Experts (Roth & Black CVPR05)
Energy Functions • Higher Order Energy Functions Unary Pairwise Higher order • Computationally expensive to minimize! • Exponential Complexity O(LN) • L = Number of Labels • N = Size of Clique
Minimizing Higher Order Energies Efficient BP in Higher Order MRFs (Lan, Roth, Huttenlocher & Black, ECCV 06) • 2x2 clique potentials for Image Denoising • Searched a restricted state space • 16 minutes per iteration Pairwise MRF Noisy Image Higher order MRF
Energy Functions • Higher Order Energy Functions Unary Pairwise Higher order • Our Method • Move making algorithm • Can handle cliques of thousand of variables • Extremely Efficient ( works in seconds)
Talk Outline • Move making Algorithms • Solvable Higher Order Potentials • Moves for the PN Potts Model • Application: Texture Segmentation
Move Making Algorithms Energy Solution Space
Move Making Algorithms Current Solution Search Neighbourhood Optimal Move Energy Solution Space
Computing the Optimal Move Current Solution Search Neighbourhood Optimal Move xc T Transformation function (t) E(x) Em Move Energy x T(xc, t) =xn = xc + t Em(t) = E(T(xc, t))
Computing the Optimal Move Current Solution Search Neighbourhood Optimal Move xc T Transformation function (t) E(x) Em Move Energy x T(xc, t) =xn = xc + t Em(t) = E(T(xc, t))
Computing the Optimal Move Current Solution Search Neighbourhood Optimal Move xc T Transformation function (t) E(x) Em Move Energy x T(xc, t) =xn = xc + t minimize t* Em(t) = E(T(xc, t)) Optimal Move
Computing the Optimal Move Current Solution Search Neighbourhood Optimal Move xc T Transformation function (t) E(x) Em Move Energy x Key Characteristic: Search Neighbourhood Bigger the better!
Moves using Graph Cuts Expansion and Swap Move Algorithm [Boykov, Veksler, Zabih] • Exponential Move Search Space (Good ) • Move encoded by binary vector t • Move Energy
Moves using Graph Cuts Expansion and Swap Move Algorithm [Boykov, Veksler, Zabih] • Exponential Move Search Space (Good ) • Move encoded by binary vector t • Move Energy • Optimal move t* in polynomial time Submodular
Expansion Move • a-Expansion Transformation • Variables take label aor retain current label • Optimal move can be computed for pairwise potentials which are metric. [Boykov, Veksler, Zabih]
Expansion Move Status: Tree Ground Initialize with Tree Expand Ground Expand House Expand Sky House Sky [Boykov, Veksler, Zabih]
Swap Move • Optimal move can be computed for pairwise potentials which are semi-metric. • ab - Swap Transformation • Variables labeled a, bcan swap their labels [Boykov, Veksler, Zabih]
Swap Move Tree Ground Swap Sky, House House Sky [Boykov, Veksler, Zabih]
Moves for Higher Order Potentials • Question you should be asking: Can my higher order potential be solved using α-expansions?
Moves for Higher Order Potentials • Question you should be asking: • Show that move energy is submodularfor all xc Can my higher order potential be solved using α-expansions?
Moves for Higher Order Potentials • Question you should be asking: • Show that move energy is submodularfor all xc Can my higher order potential be solved using α-expansions? Not an easy thing to do!
Moves for Higher Order Potentials • Form of the Higher Order Potentials Clique Inconsistency function: Pairwise potential: xj xi xk Sum Form c xm xl Max Form
Theoretical Results: Swap • Move energy is always submodular if non-decreasing concave. See paper for proofs
Theoretical Results: Expansion • Move energy is always submodular if increasing linear See paper for proofs
PN Potts Model c Cost : gred
PN Potts Model c Cost : gmax
Optimal moves for PN Potts • Computing the optimal swap move Label 1(a) Label 2 (b) Label 3 Label 4 c
Optimal moves for PN Potts • Computing the optimal swap move Label 1(a) Case 1 Not all variables assigned label 1 or 2 Label 2 (b) Label 3 Label 4 c
Optimal moves for PN Potts • Computing the optimal swap move Label 1(a) Case 1 Not all variables assigned label 1 or 2 Label 2 (b) Label 3 Label 4 c
Optimal moves for PN Potts • Computing the optimal swap move Label 1(a) Case 1 Not all variables assigned label 1 or 2 Label 2 (b) Label 3 Label 4 c
Optimal moves for PN Potts • Computing the optimal swap move Label 1(a) Case 1 Not all variables assigned label 1 or 2 Label 2 (b) Label 3 Label 4 Move Energy is independent of tc and can be ignored. c
Optimal moves for PN Potts • Computing the optimal swap move Label 1(a) Case 2 All variables assigned label 1 or 2 Label 2 (b) Label 3 Label 4 c
Optimal moves for PN Potts • Computing the optimal swap move Label 1(a) Case 2 All variables assigned label 1 or 2 Label 2 (b) Label 3 Label 4 Can be minimized by solving a st-mincut problem c
Solving the Move Energy Add a constant This transformation does not effect the solution
Solving the Move Energy • Computing the optimal swap move Source Ms v1 v2 vn Mt vi Source Set ti= 0 vj Sink Set tj= 1 Sink
Solving the Move Energy • Computing the optimal swap move Source Ms v1 v2 vn Case 1: all xi= a(vi Source) Mt Cost: Sink
Solving the Move Energy • Computing the optimal swap move Source Ms v1 v2 vn Case 2: all xi= b(vi Sink) Mt Cost: Sink
Solving the Move Energy • Computing the optimal swap move Source Ms v1 v2 vn Case 3: all xi= a,b(vi Source, Sink) Mt Cost: Sink
Optimal moves for PN Potts • The expansion move energy • Similar graph construction. See paper for details
Experimental Results • Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Original Image
Experimental Results • Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Colour Histogram Unary Cost: Tree
Experimental Results • Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Edge Sensitive Smoothness Cost