Decomposition Algorithm

1. Decomposition Algorithm (Fixed the Working Set Size) • In chunking method, the working set might grow very fast • In decomposition algorithm, keep working set’s size fixed • When a new point is added to the working set, another point has to be removed • Dual objective is increased after updating • SMO is a special case of decomposition alg.

2. Sequential Minimal Optimization (SMO) • Deals with an equality constraint and a box constraints of dual problem • Works on the smallest working set (only 2) • Find the optimal solution by only changing value that is in the working set • The solution can be analytically defined • The best feature of SMO

3. Suppose that we change • In order to keep the equality constraint we have to change two value such that • The new value has to satisfy the box constraints Analytical Solution for Two Points • We have a more restriction on changing

4. Suppose that we change we can get • Once we have • A restrictive constraint: where A Restrictive Constraint on New if and if

5. and

6. and

7. and

8. and

9. We would like change to , so that • Once you the optimal solution, and the classifier is going to be: Don’t Forget What You Are Looking for! • You are solving 1-norm SVM in dual form • Have to keep equality & box constraints

10. Notations Used in Theorem 7.12

11. Two Important Parts of SMO (selection heuristics & stopping criterion) • A good selection of a pair of updating points will speed up the convergence • The selection heuristics maybe depend on the stopping criterion • Stopping criterion: duality gap => naturally to choose the points with the most violation of the KKT conditions (Too expensive)