1 / 45

Constrained Optimization

Constrained Optimization. Rong Jin. Outline. Equality constraints Inequality constraints Linear Programming Quadratic Programming. Optimization Under Equality Constraints. Maximum Entropy Model English ‘in’  French { dans (1), en (2), à (3), au cours de (4), pendant (5)}.

raven
Download Presentation

Constrained Optimization

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Constrained Optimization Rong Jin

  2. Outline • Equality constraints • Inequality constraints • Linear Programming • Quadratic Programming

  3. Optimization Under Equality Constraints • Maximum Entropy Model • English ‘in’  French • {dans (1), en (2), à (3), au cours de (4), pendant (5)}

  4. Reducing variables • Representing variables using only p1 and p4 • Objective function is changed • Solution: p1= 0.2, p2 = 0.3, p3 =0.1, p4 = 0.2, p5 = 0.2

  5. Maximum Entropy Model for Classification • It is unlikely that we can use the previous simple approach to solve such a general • Solution: Lagrangian

  6. Equality Constraints: Lagrangian • Introduce a Lagrange multiplier  for the equality constraint • Construct the Lagrangian • Necessary condition • A optimal solution for the original optimization problem has to be one of the stationary point of the Lagrangian

  7. Example: • Introduce a Lagrange multiplier  for constraint • Construct the Lagrangian • Stationary points

  8. Lagrange Multipliers • Introducing a Lagrange multiplier for each constraint • Construct the Lagrangian for the original optimization problem

  9. Original Entropy Function Constraints Lagrange Multiplier • We have more variables • p1, p2, p3, p4, p5 and, 1, 2, 3 • Necessary condition (first order condition) • A local/global optimum point for the original constrained optimization problem  a stationary point of the corresponding Lagrangian

  10. Stationary Points for Lagrangian All probabilities p1, p2, p3, p4, p5 are expressed as functions of Lagrange multipliers s

  11. Dual Problem • p1, p2, p3, p4, p5 are expressed as functions of s • We can even remove the variable 3 • Put together necessary condition • Still difficult to solve

  12. Dual Problem • p1, p2, p3, p4, p5 are expressed as functions of s • We can even remove the variable 3 • Put together necessary condition • Still difficult to solve

  13. Dual Problem • Dual problem • Substitute the expression for ps into the Lagrangian • Find the s that MINIMIZE the substituted Lagrangian

  14. Expression for ps Substituted Lagrangian Dual Problem Original Lagrangian Finding s such that the above objective function is minimized

  15. Dual Problem Primal Problem Dual Problem • Using dual problem • Constrained optimization  unconstrained optimization • Need to change maximization to minimization • Only valid when the original optimization problem is convex/concave (strong duality) x*=* When convex/concave

  16. Maximum Entropy Model for Classification • Introduce a Lagrange multiplier for each linear constraint

  17. Original Entropy Function Consistency Constraint Normalization Constraint Maximum Entropy Model for Classification • Construct the Lagrangian for the original optimization problem

  18. Stationary points: first derivatives are zero Sum of conditional probabilities must be one Stationary Points Conditional Exponential Model !

  19. Dual Problem

  20. Dual Problem

  21. Dual Problem

  22. Dual Problem What is wrong?

  23. Dual Problem

  24. Dual Problem

  25. Dual Problem

  26. Dual Problem

  27. Dual Problem Minimizing L  maximizing the log-likelihood

  28. Support Vector Machine • Having many inequality constraints • Solving the above problem directly could be difficult • Many variables: w, b,  • Unable to use nonlinear kernel function

  29. Two cases: • g(x) = c, • g(x) > c   =0 Non-negative Lagrange Multiplier Inequality Constraints: Modified Lagrangian • Introduce a Lagrange multiplier  for the inequality constraint • Construct the Lagrangian • Karush-Kuhn-Tucker (KKT) condition • A optimal solution for the original optimization problem will satisfy the following conditions

  30. Example: • Introduce a Lagrange multiplier  for constraint • Construct the Lagrangian • KT conditions • Expressing objective function using  • Solution is =3

  31. Example: • Introduce a Lagrange multiplier  for constraint • Construct the Lagrangian • KT conditions • Expressing objective function using  • Solution is =3

  32. Expressing objective function using  • Solution is =3 Example: • Introduce a Lagrange multiplier  for constraint • Construct the Lagrangian • KKT conditions

  33. MinMax + SVM Model • Lagrange multipliers for inequality constraints

  34. SVM Model • Lagrangian for SVM model • Karush-Kuhn-Tucker condition

  35. SVM Model • Lagrangian for SVM model • Karush-Kuhn-Tucker condition

  36. Dual Problem for SVM • Express w, b,  using  and 

  37. Dual Problem for SVM • Express w, b,  using  and  • Finding solution satisfying KKT conditions is difficult

  38. Dual Problem for SVM • Rewrite the Lagrangian function using only and  • Simplify using KT conditions

  39. Maximize  Minimize Dual Problem for SVM • Final dual problem

  40. Quadratic Programming Find Subject to

  41. Find Subject to Linear Programming • Very very useful algorithm • 1300+ papers • 100+ books • 10+ courses • 100s of companies • Main methods • Simplex method • Interior point method Most important: how to convert a general problem into the above standard form

  42. Find Subject to Example • Need to change max to min

  43. Find Subject to Example • Need change  to 

  44. Find Subject to Example • Need to convert the inequality

  45. Find Subject to Example • Need change |x3|

More Related