1 / 97

Probabilistic Inference Lecture 4 – Part 1

Probabilistic Inference Lecture 4 – Part 1. M. Pawan Kumar pawan.kumar@ecp.fr. Slides available online http:// cvc.centrale-ponts.fr /personnel/ pawan /. Recap. 0. 6. 2. 0. 4. Label l 1. 1. 2. 3. 1. Label l 0. 1. 5. 0. 3. 2. V b. V c. V a. d a. d b. d c.

tacy
Download Presentation

Probabilistic Inference Lecture 4 – Part 1

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Probabilistic InferenceLecture 4 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online http://cvc.centrale-ponts.fr/personnel/pawan/

  2. Recap 0 6 2 0 4 Label l1 1 2 3 1 Label l0 1 5 0 3 2 Vb Vc Va da db dc Labeling f : VL D : Observed data (image) V : Unobserved variables Va is assigned lf(a) L : Discrete, finite label set

  3. Recap 0 6 2 0 4 Label l1 1 2 3 1 Label l0 1 5 0 3 2 bc;f(b)f(c) Vb Vc Va a;f(a) da db dc Q(f; ) = ∑a a;f(a) + ∑(a,b) ab;f(a)f(b)

  4. Recap 0 6 2 0 4 Label l1 1 2 3 1 Label l0 1 5 0 3 2 Vb Vc Va da db dc f* = argminf Q(f; ) Q(f; ) = ∑a a;f(a) + ∑(a,b) ab;f(a)f(b)

  5. Recap 0 6 2 0 4 Label l1 1 2 3 1 Label l0 1 5 0 3 2 Vb Vc Va f* = argminf Q(f; ) Q(f; ) = ∑a a;f(a) + ∑(a,b) ab;f(a)f(b)

  6. Outline • Convex Optimization • Integer Programming Formulation • Convex Relaxations • Comparison • Generalization of Results

  7. Mathematical Optimization min g0(x) Objective function s.t. gi(x) ≤ 0 Inequality constraints hi(x) = 0 Equality constraints x is a feasible point gi(x) ≤ 0, hi(x) = 0 x is a strictlyfeasible point gi(x) < 0, hi(x) = 0 Feasible region - set of all feasible points

  8. Convex Optimization min g0(x) Objective function s.t. gi(x) ≤ 0 Inequality constraints hi(x) = 0 Equality constraints Feasible region is convex Objective function is convex Convex set? Convex function?

  9. Convex Set Line Segment x1 x2 c x1 + (1 - c) x2 c  [0,1] Endpoints

  10. Convex Set x1 x2 All points on the line segment lie within the set For all line segments with endpoints in the set

  11. Non-Convex Set x1 x2

  12. Examples of Convex Sets x1 x2 Line Segment

  13. Examples of Convex Sets x1 x2 Line

  14. Examples of Convex Sets Hyperplane aTx - b = 0

  15. Examples of Convex Sets Halfspace aTx - b ≤ 0

  16. Examples of Convex Sets t x2 x1 Second-order Cone ||x|| ≤ t

  17. Examples of Convex Sets Semidefinite Cone {X | X0} aTXa ≥ 0, for all a Rn All eigenvalues of X are non-negative aTX2a ≥ 0 aTX1a ≥ 0 aT(cX1 + (1-c)X2)a ≥ 0

  18. Operations that Preserve Convexity Intersection Polyhedron / Polytope

  19. Operations that Preserve Convexity Intersection

  20. Operations that Preserve Convexity Affine Transformation x  Ax + b

  21. Convex Function g(x) x1 x2 x Blue point always lies above red point

  22. Convex Function g(x) x1 x2 x g( c x1 + (1 - c) x2 ) ≤ c g(x1) + (1 - c) g(x2) Domain of g(.) has to be convex

  23. Convex Function g(x) x1 x2 x g( c x1 + (1 - c) x2 ) ≤ c g(x1) + (1 - c) g(x2) -g(.) is concave

  24. Convex Function Once-differentiable functions g(y) + g(y)T (x - y) ≤ g(x) g(x) (y,g(y)) g(y) + g(y)T (x - y) x Twice-differentiable functions 2g(x) 0

  25. Convex Function and Convex Sets g(x) x Epigraph of a convex function is a convex set

  26. Examples of Convex Functions Linear function aTx p-Norm functions (x1p + x2p + xnp)1/p, p ≥ 1 Quadratic functions xTQx Q 0

  27. Operations that Preserve Convexity Non-negative weighted sum g1(x) g2(x) + w2 + …. w1 x x xTQx + aTx + b Q 0

  28. Operations that Preserve Convexity Pointwise maximum g1(x) g2(x) , max x x Pointwise minimum of concave functions is concave

  29. Convex Optimization min g0(x) Objective function s.t. gi(x) ≤ 0 Inequality constraints hi(x) = 0 Equality constraints Feasible region is convex  Objective function is convex 

  30. Linear Programming min g0(x) Objective function s.t. gi(x) ≤ 0 Inequality constraints hi(x) = 0 Equality constraints min g0Tx Linear function s.t. giTx ≤ 0 Linear constraints hiTx = 0 Linear constraints

  31. Quadratic Programming min g0(x) Objective function s.t. gi(x) ≤ 0 Inequality constraints hi(x) = 0 Equality constraints min xTQx + aTx + b Quadratic function s.t. giTx ≤ 0 Linear constraints hiTx = 0 Linear constraints

  32. Second-Order Cone Programming min g0(x) Objective function s.t. gi(x) ≤ 0 Inequality constraints hi(x) = 0 Equality constraints min g0Tx Linear function Quadratic constraints s.t. xTQix + aiTx + bi ≤ 0 hiTx = 0 Linear constraints

  33. Semidefinite Programming min g0(x) Objective function s.t. gi(x) ≤ 0 Inequality constraints hi(x) = 0 Equality constraints min Q  X Linear function s.t. X 0 Semidefinite constraints Ai X = 0 Linear constraints

  34. Outline • Convex Optimization • Integer Programming Formulation • Convex Relaxations • Comparison • Generalization of Results

  35. Cost of V1 = 1 Cost of V1 = 0 Integer Programming Formulation 2 0 4 Unary Cost Label ‘1’ 1 3 Label ‘0’ 5 0 2 V2 V1 Labelling m = {1 , 0} ; 2 4 ] 2 Unary Cost Vector u = [ 5

  36. V1= 1 V1 0 Integer Programming Formulation 2 0 4 Unary Cost Label ‘1’ 1 3 Label ‘0’ 5 0 2 V2 V1 Labelling m = {1 , 0} ; 2 4 ]T 2 Unary Cost Vector u = [ 5 Label vector x = [ -1 1 ; 1 -1 ]T Recall that the aim is to find the optimal x

  37. Integer Programming Formulation 2 0 4 Unary Cost Label ‘1’ 1 3 Label ‘0’ 5 0 2 V2 V1 Labelling m = {1 , 0} ; 2 4 ]T 2 Unary Cost Vector u = [ 5 Label vector x = [ -1 1 ; 1 -1 ]T 1 Sum of Unary Costs = ∑iui (1 + xi) 2

  38. Pairwise Cost Matrix P Cost of V1 = 0 and V1 = 0 0 Cost of V1 = 0 and V2 = 0 0 0 1 0 Cost of V1 = 0 and V2 = 1 0 1 0 0 3 0 0 0 Integer Programming Formulation 2 0 4 Pairwise Cost Label ‘1’ 1 3 Label ‘0’ 5 0 2 V2 V1 Labelling m = {1 , 0} 0 3 0

  39. Pairwise Cost Matrix P 0 0 0 1 0 0 1 0 0 3 0 0 0 Integer Programming Formulation 2 0 4 Pairwise Cost Label ‘1’ 1 3 Label ‘0’ 5 0 2 V2 V1 Labelling m = {1 , 0} Sum of Pairwise Costs 1 ∑ijPij (1 + xi)(1+xj) 0 3 0 4

  40. Pairwise Cost Matrix P 0 0 0 1 0 1 = ∑ijPij (1 + xi + xj + Xij) 4 0 1 0 0 3 0 0 0 Integer Programming Formulation 2 0 4 Pairwise Cost Label ‘1’ 1 3 Label ‘0’ 5 0 2 V2 V1 Labelling m = {1 , 0} Sum of Pairwise Costs 1 ∑ijPij (1 + xi +xj + xixj) 0 3 0 4 X = x xT Xij = xi xj

  41. Uniqueness Constraint ∑ xi = 2 - |L| i  Va Integer Programming Formulation Constraints • Integer Constraints xi{-1,1} X = x xT

  42. ∑ xi = 2 - |L| i  Va Non-Convex Integer Programming Formulation 1 1 ∑ Pij (1 + xi + xj + Xij) x* = argmin + ∑ ui (1 + xi) 4 2 Convex xi{-1,1} X = x xT

  43. Outline • Convex Optimization • Integer Programming Formulation • Convex Relaxations • Linear Programming (LP-S) • Semidefinite Programming (SDP-L) • Second Order Cone Programming (SOCP-MS) • Comparison • Generalization of Results

  44. ∑ xi = 2 - |L| i  Va LP-S Schlesinger, 1976 Retain Convex Part 1 1 ∑ Pij (1 + xi + xj + Xij) x* = argmin + ∑ ui (1 + xi) 4 2 Relax Non-Convex Constraint xi{-1,1} X = x xT

  45. ∑ xi = 2 - |L| i  Va LP-S Schlesinger, 1976 Retain Convex Part 1 1 ∑ Pij (1 + xi + xj + Xij) x* = argmin + ∑ ui (1 + xi) 4 2 xi[-1,1] Relax Non-Convex Constraint X = x xT

  46. ∑ Xij = (2 - |L|) xi j  Vb LP-S Schlesinger, 1976 X = x xT Xij[-1,1] 1 + xi + xj + Xij≥ 0

  47. ∑ xi = 2 - |L| i  Va LP-S Schlesinger, 1976 Retain Convex Part 1 1 ∑ Pij (1 + xi + xj + Xij) x* = argmin + ∑ ui (1 + xi) 4 2 xi[-1,1] Relax Non-Convex Constraint X = x xT

  48. ∑ xi = 2 - |L| i  Va ∑ Xij = (2 - |L|) xi j  Vb LP-S Schlesinger, 1976 Retain Convex Part 1 1 ∑ Pij (1 + xi + xj + Xij) x* = argmin + ∑ ui (1 + xi) 4 2 xi[-1,1], Xij[-1,1] LP-S 1 + xi + xj + Xij≥ 0

  49. Outline • Convex Optimization • Integer Programming Formulation • Convex Relaxations • Linear Programming (LP-S) • Semidefinite Programming (SDP-L) • Second Order Cone Programming (SOCP-MS) • Comparison • Generalization of Results

  50. ∑ xi = 2 - |L| i  Va SDP-L Lasserre, 2000 Retain Convex Part 1 1 ∑ Pij (1 + xi + xj + Xij) x* = argmin + ∑ ui (1 + xi) 4 2 Relax Non-Convex Constraint xi{-1,1} X = x xT

More Related