Quadratic Programming and Duality

1. Quadratic Programming and Duality Sivaraman Balakrishnan

2. Outline • Quadratic Programs • General Lagrangian Duality • Lagrangian Duality in QPs

3. Norm approximation • Problem • Interpretation • Geometric – try to find projection of b into ran(A) • Statistical – try to find solution to b = Ax + v • v is a measurement noise (choose norm so that v is small in that norm) • Several others

4. Examples • -- Least Squares Regression • -- Chebyshev • -- Least Median Regression • More generally can use *any* convex penalty function

5. Picture from BV

6. Least norm • Perfect measurements  • Not enough of them  • Heart of something known as compressed sensing • Related to regularized regression in the noisy case

7. Smooth signal reconstruction • S(x) is a smoothness penalty • Least squares penalty • Smooths out noise and sharp transitions • Total variation (peak to valley intuition) • Smooths out noise but preserves sharp transitions

8. Euclidean Projection • Very fundamental idea in constrained minimization • Efficient algorithms to project onto many many convex sets (norm balls, special polyhedra etc) • More generally finding minimum distance between polyhedra is a QP

9. Quadratic Programming Duality

10. General recipe • Form Lagrangian • How to figure out signs?

11. Primal & Dual Functions • Primal • Dual

12. Primal & Dual Programs • Primal Programs • Constraints are now implicit in the primal • Dual Program

13. Lagrangian Properties • Can extract primal and dual problem • Dual problem is always concave • Proof • Dual problem is always a lower bound on primal • Proof • Strong duality gives complementary slackness • Proof

14. Some examples of QP duality • Consider the example from class • Lets try to derive dual using Lagrangian

15. General PSD QP • Primal • Dual

16. SVM – Lagrange Dual • Primal SVM • Dual SVM • Recovering Primal Variables and Complementary Slackness