Non-negative Matrix Factorization: Recent Algorithms, Extensions, and Software

1. Non-negative Matrix Factorization Recent algorithms, extensions and available software Atina Dunlap Brooks (adbrook2@stat.ncsu.edu) North Carolina State University

2. Recent Algorithms • Lee & Seung’s multiplicative updates are easy to understand and to implement • Can be very slow to converge • ALS can speed things up • Convergence theory is not particularly strong • Most NMF methods do not have robust convergence, but work well in practice

3. Projected GradientDescent Method • Chih-Jen Lin (2007) • Bound-constrained optimization • Projected Gradient

4. Projected GradientDescent Method • Can be applied to both the multiplicative updates and the ALS solution • Generally, greatest speed was achieved with the projected gradient combined with ALS

5. Fast Non-NegativeMatrix Approximation • Kim, Sra & Dhillon (2007) • Employs Newton-type methods to solve NMF • Uses curvature information vs. gradient descent approach • Provide an exact method (good accuracy, but still slow) and a very fast inexact method

6. References forAlgorithm Comparisons • Algorithms and Applications for Approximate Nonnegative Matrix Factorization by Berry, Browne, Langville, Pauca & Plemmons (2006) • Optimality, Computation, and Interpretations of Nonnegative Matrix Factorizations by Chu, Diele, Plemmons & Ragni (2004)

7. Extensions • Tri-Factorization • Semi-NMF • Convex-NMF • Non-negative Tensor Factorization • Inferential Robust Matrix Factorization

8. OrthogonalTri-factorization • Ding, Li, Peng & Park (2006) • Requiring orthogonality introduces uniqueness and improves clustering interpretations A = WSH, where WTW=I and HTH=I • W gives row clusters while H gives column clusters

9. Semi-NMF • Ding, Li & Jordan (2006) • Allows A and W to contain negative values, but H is restricted to non-negative • Provides more flexibility (negative entries) and a clustering which is usually better than k-means

10. Convex-NMF • Ding, Li & Jordan (2006) • Restricts W to be convex combinations of the columns of A • Ensures meaningful cluster centroids • W and H tend to be sparse

11. Non-NegativeTensor Factorization • Uses n-way arrays instead of the 2-dimensional arrays used by NMF • Presentations during the workshop by Michael Berry and Bob Plemmons

12. Inferential RobustMatrix Factorization • Fogel, Young, Hawkins & Ledirac (2007) • Uses the same method for robustness as Liu et al. (2003) for robust SVD • Paul Fogel will be presenting on an application

13. Software - Matlab • Matlab Code • Patrik Hoyer • http://www.cs.helsinki.fi/u/phoyer/ • Includes Lee & Seung’s multiplicative updates and Hoyer’s sparseness • Chih-Jen Lin • http://www.csie.ntu.edu.tw/~cjlin/nmf/ • Includes projected gradient descent applied to multiplicative updates and ALS

14. Software • C code – nnmf() • Simon Sheperd • http://www.simonshepherd.supanet.com/nnmf.htm • Very fast algorithm (as of 2004) • JMP script - irMF • Paul Fogel • http://www.niss.org/irMF/ • Inferential Robust Matrix Factorization

15. Thank You