Constrained Optimization

Constrained Optimization Rong Jin

Outline • Equality constraints • Inequality constraints • Linear Programming • Quadratic Programming

Optimization Under Equality Constraints • Maximum Entropy Model • English ‘in’  French • {dans (1), en (2), à (3), au cours de (4), pendant (5)}

Reducing variables • Representing variables using only p1 and p4 • Objective function is changed • Solution: p1= 0.2, p2 = 0.3, p3 =0.1, p4 = 0.2, p5 = 0.2

Maximum Entropy Model for Classification • It is unlikely that we can use the previous simple approach to solve such a general • Solution: Lagrangian

Equality Constraints: Lagrangian • Introduce a Lagrange multiplier  for the equality constraint • Construct the Lagrangian • Necessary condition • A optimal solution for the original optimization problem has to be one of the stationary point of the Lagrangian

Example: • Introduce a Lagrange multiplier  for constraint • Construct the Lagrangian • Stationary points

Lagrange Multipliers • Introducing a Lagrange multiplier for each constraint • Construct the Lagrangian for the original optimization problem

Original Entropy Function Constraints Lagrange Multiplier • We have more variables • p1, p2, p3, p4, p5 and, 1, 2, 3 • Necessary condition (first order condition) • A local/global optimum point for the original constrained optimization problem  a stationary point of the corresponding Lagrangian

Stationary Points for Lagrangian All probabilities p1, p2, p3, p4, p5 are expressed as functions of Lagrange multipliers s

Dual Problem • p1, p2, p3, p4, p5 are expressed as functions of s • We can even remove the variable 3 • Put together necessary condition • Still difficult to solve

Dual Problem • Dual problem • Substitute the expression for ps into the Lagrangian • Find the s that MINIMIZE the substituted Lagrangian

Expression for ps Substituted Lagrangian Dual Problem Original Lagrangian Finding s such that the above objective function is minimized

Dual Problem Primal Problem Dual Problem • Using dual problem • Constrained optimization  unconstrained optimization • Need to change maximization to minimization • Only valid when the original optimization problem is convex/concave (strong duality) x*=* When convex/concave

Maximum Entropy Model for Classification • Introduce a Lagrange multiplier for each linear constraint

Original Entropy Function Consistency Constraint Normalization Constraint Maximum Entropy Model for Classification • Construct the Lagrangian for the original optimization problem

Stationary points: first derivatives are zero Sum of conditional probabilities must be one Stationary Points Conditional Exponential Model !

Dual Problem

Dual Problem What is wrong?

Dual Problem

Dual Problem Minimizing L  maximizing the log-likelihood

Support Vector Machine • Having many inequality constraints • Solving the above problem directly could be difficult • Many variables: w, b,  • Unable to use nonlinear kernel function

Two cases: • g(x) = c, • g(x) > c   =0 Non-negative Lagrange Multiplier Inequality Constraints: Modified Lagrangian • Introduce a Lagrange multiplier  for the inequality constraint • Construct the Lagrangian • Karush-Kuhn-Tucker (KKT) condition • A optimal solution for the original optimization problem will satisfy the following conditions

Example: • Introduce a Lagrange multiplier  for constraint • Construct the Lagrangian • KT conditions • Expressing objective function using  • Solution is =3

Expressing objective function using  • Solution is =3 Example: • Introduce a Lagrange multiplier  for constraint • Construct the Lagrangian • KKT conditions

MinMax + SVM Model • Lagrange multipliers for inequality constraints

SVM Model • Lagrangian for SVM model • Karush-Kuhn-Tucker condition

Dual Problem for SVM • Express w, b,  using  and 

Dual Problem for SVM • Express w, b,  using  and  • Finding solution satisfying KKT conditions is difficult

Dual Problem for SVM • Rewrite the Lagrangian function using only and  • Simplify using KT conditions

Maximize  Minimize Dual Problem for SVM • Final dual problem

Quadratic Programming Find Subject to

Find Subject to Linear Programming • Very very useful algorithm • 1300+ papers • 100+ books • 10+ courses • 100s of companies • Main methods • Simplex method • Interior point method Most important: how to convert a general problem into the above standard form

Find Subject to Example • Need to change max to min

Find Subject to Example • Need change  to 

Find Subject to Example • Need to convert the inequality

Find Subject to Example • Need change |x3|

Constrained Optimization