Discrete Optimization Lecture 6 – Part I

1. Discrete Optimization Lecture 6 – Part I M. Pawan Kumar pawan.kumar@ecp.fr Slides available online http://cvn.ecp.fr/personnel/pawan

2. Programming Assignment I Nikos’ course website. Stereo correspondence. Images provided. Expansion algorithm. Max-flow code provided. 2 students per group. Deadline: 2 weeks.

3. Recap Label l1 Label l0 Vb Vc Va da db dc Labeling f : VL D : Observed data (image) V : Unobserved variables L : Discrete, finite label set

4. Recap Label l1 Label l0 Vb Vc Va da db dc Labeling f : VL D : Observed data (image) V : Unobserved variables Va is assigned lf(a) L : Discrete, finite label set

5. Recap 0 6 2 0 4 Label l1 1 2 3 1 Label l0 1 5 0 3 2 bc;f(b)f(c) Vb Vc Va a;f(a) da db dc Q(f; ) = ∑a a;f(a) + ∑(a,b) ab;f(a)f(b)

6. Recap 0 6 2 0 4 Label l1 1 2 3 1 Label l0 1 5 0 3 2 Vb Vc Va da db dc f* = argminf Q(f; ) Q(f; ) = ∑a a;f(a) + ∑(a,b) ab;f(a)f(b)

7. Recap 0 6 2 0 4 Label l1 1 2 3 1 Label l0 1 5 0 3 2 Vb Vc Va f* = argminf Q(f; ) Q(f; ) = ∑a a;f(a) + ∑(a,b) ab;f(a)f(b)

8. Outline • Convex Optimization • Integer Programming Formulation • Convex Relaxations • Comparison • Generalization of Results

9. Mathematical Optimization min g0(x) Objective function s.t. gi(x) ≤ 0 Inequality constraints hi(x) = 0 Equality constraints x is a feasible point gi(x) ≤ 0, hi(x) = 0 x is a strictlyfeasible point gi(x) < 0, hi(x) = 0 Feasible region - set of all feasible points

10. Convex Optimization min g0(x) Objective function s.t. gi(x) ≤ 0 Inequality constraints hi(x) = 0 Equality constraints Feasible region is convex Objective function is convex Convex set? Convex function?

11. Convex Set Line Segment x1 x2 c x1 + (1 - c) x2 c  [0,1] Endpoints

12. Convex Set x1 x2 All points on the line segment lie within the set For all line segments with endpoints in the set

13. Non-Convex Set x1 x2

14. Examples of Convex Sets x1 x2 Line Segment

15. Examples of Convex Sets x1 x2 Line

16. Examples of Convex Sets Hyperplane aTx - b = 0

17. Examples of Convex Sets Halfspace aTx - b ≤ 0

18. Examples of Convex Sets t x2 x1 Second-order Cone ||x|| ≤ t

19. Examples of Convex Sets Semidefinite Cone {X | X0} aTXa ≥ 0, for all a Rn All eigenvalues of X are non-negative aTX2a ≥ 0 aTX1a ≥ 0 aT(cX1 + (1-c)X2)a ≥ 0

20. Operations that Preserve Convexity Intersection Polyhedron / Polytope

21. Operations that Preserve Convexity Intersection

22. Operations that Preserve Convexity Affine Transformation x  Ax + b

23. Convex Function g(x) x1 x2 x Blue point always lies above red point

24. Convex Function g(x) x1 x2 x g( c x1 + (1 - c) x2 ) ≤ c g(x1) + (1 - c) g(x2) Domain of g(.) has to be convex

25. Convex Function g(x) x1 x2 x g( c x1 + (1 - c) x2 ) ≤ c g(x1) + (1 - c) g(x2) -g(.) is concave

26. Convex Function Once-differentiable functions g(y) + g(y)T (x - y) ≤ g(x) g(x) (y,g(y)) g(y) + g(y)T (x - y) x Twice-differentiable functions 2g(x) 0

27. Convex Function and Convex Sets g(x) x Epigraph of a convex function is a convex set

28. Examples of Convex Functions Linear function aTx p-Norm functions (x1p + x2p + xnp)1/p, p ≥ 1 Quadratic functions xTQx Q 0

29. Operations that Preserve Convexity Non-negative weighted sum g1(x) g2(x) + w2 + …. w1 x x xTQx + aTx + b Q 0

30. Operations that Preserve Convexity Pointwise maximum g1(x) g2(x) , max x x Pointwise minimum of concave functions is concave

31. Convex Optimization min g0(x) Objective function s.t. gi(x) ≤ 0 Inequality constraints hi(x) = 0 Equality constraints Feasible region is convex  Objective function is convex 

32. Linear Programming min g0(x) Objective function s.t. gi(x) ≤ 0 Inequality constraints hi(x) = 0 Equality constraints min g0Tx Linear function s.t. giTx ≤ 0 Linear constraints hiTx = 0 Linear constraints

33. Quadratic Programming min g0(x) Objective function s.t. gi(x) ≤ 0 Inequality constraints hi(x) = 0 Equality constraints min xTQx + aTx + b Quadratic function s.t. giTx ≤ 0 Linear constraints hiTx = 0 Linear constraints

34. Second-Order Cone Programming min g0(x) Objective function s.t. gi(x) ≤ 0 Inequality constraints hi(x) = 0 Equality constraints min g0Tx Linear function Quadratic constraints s.t. xTQix + aiTx + bi ≤ 0 hiTx = 0 Linear constraints

35. Semidefinite Programming min g0(x) Objective function s.t. gi(x) ≤ 0 Inequality constraints hi(x) = 0 Equality constraints min Q  X Linear function s.t. X 0 Semidefinite constraints Ai X = 0 Linear constraints

36. Outline • Convex Optimization • Integer Programming Formulation • Convex Relaxations • Comparison • Generalization of Results

37. Cost of V1 = 1 Cost of V1 = 0 Integer Programming Formulation 2 0 4 Unary Cost Label ‘1’ 1 3 Label ‘0’ 5 0 2 V2 V1 Labeling = {1 , 0} ; 2 4 ] 2 Unary Cost Vector u = [ 5

38. V1= 1 V1 0 Integer Programming Formulation 2 0 4 Unary Cost Label ‘1’ 1 3 Label ‘0’ 5 0 2 V2 V1 Labeling = {1 , 0} ; 2 4 ]T 2 Unary Cost Vector u = [ 5 Label vector x = [ -1 1 ; 1 -1 ]T Recall that the aim is to find the optimal x

39. Integer Programming Formulation 2 0 4 Unary Cost Label ‘1’ 1 3 Label ‘0’ 5 0 2 V2 V1 Labeling = {1 , 0} ; 2 4 ]T 2 Unary Cost Vector u = [ 5 Label vector x = [ -1 1 ; 1 -1 ]T 1 Sum of Unary Costs = ∑iui (1 + xi) 2

40. Pairwise Cost Matrix P Cost of V1 = 0 and V1 = 0 0 Cost of V1 = 0 and V2 = 0 0 0 1 0 Cost of V1 = 0 and V2 = 1 0 1 0 0 3 0 0 0 Integer Programming Formulation 2 0 4 Pairwise Cost Label ‘1’ 1 3 Label ‘0’ 5 0 2 V2 V1 Labeling = {1 , 0} 0 3 0

41. Pairwise Cost Matrix P 0 0 0 1 0 0 1 0 0 3 0 0 0 Integer Programming Formulation 2 0 4 Pairwise Cost Label ‘1’ 1 3 Label ‘0’ 5 0 2 V2 V1 Labeling = {1 , 0} Sum of Pairwise Costs 1 ∑ijPij (1 + xi)(1+xj) 0 3 0 4

42. Pairwise Cost Matrix P 0 0 0 1 0 1 = ∑ijPij (1 + xi + xj + Xij) 4 0 1 0 0 3 0 0 0 Integer Programming Formulation 2 0 4 Pairwise Cost Label ‘1’ 1 3 Label ‘0’ 5 0 2 V2 V1 Labeling = {1 , 0} Sum of Pairwise Costs 1 ∑ijPij (1 + xi +xj + xixj) 0 3 0 4 X = x xT Xij = xi xj

43. Uniqueness Constraint ∑ xi = 2 - |L| i  Va Integer Programming Formulation Constraints • Integer Constraints xi{-1,1} X = x xT

44. ∑ xi = 2 - |L| i  Va Non-Convex Integer Programming Formulation 1 1 ∑ Pij (1 + xi + xj + Xij) x* = argmin + ∑ ui (1 + xi) 4 2 Convex xi{-1,1} X = x xT

45. Outline • Convex Optimization • Integer Programming Formulation • Convex Relaxations • Linear Programming (LP-S) • Semidefinite Programming (SDP-L) • Second Order Cone Programming (SOCP-MS) • Comparison • Generalization of Results

46. ∑ xi = 2 - |L| i  Va LP-S Schlesinger, 1976 Retain Convex Part 1 1 ∑ Pij (1 + xi + xj + Xij) x* = argmin + ∑ ui (1 + xi) 4 2 Relax Non-Convex Constraint xi{-1,1} X = x xT

47. ∑ xi = 2 - |L| i  Va LP-S Schlesinger, 1976 Retain Convex Part 1 1 ∑ Pij (1 + xi + xj + Xij) x* = argmin + ∑ ui (1 + xi) 4 2 xi[-1,1] Relax Non-Convex Constraint X = x xT

48. ∑ Xij = (2 - |L|) xi j  Vb LP-S Schlesinger, 1976 X = x xT Xij[-1,1] 1 + xi + xj + Xij≥ 0

49. ∑ xi = 2 - |L| i  Va LP-S Schlesinger, 1976 Retain Convex Part 1 1 ∑ Pij (1 + xi + xj + Xij) x* = argmin + ∑ ui (1 + xi) 4 2 xi[-1,1] Relax Non-Convex Constraint X = x xT

50. ∑ xi = 2 - |L| i  Va ∑ Xij = (2 - |L|) xi j  Vb LP-S Schlesinger, 1976 Retain Convex Part 1 1 ∑ Pij (1 + xi + xj + Xij) x* = argmin + ∑ ui (1 + xi) 4 2 xi[-1,1], Xij[-1,1] LP-S 1 + xi + xj + Xij≥ 0