Input Space Regularization Stabilizes Pre-Images For Kernel PCA De-Noising

1. Input Space Regularization Stabilizes Pre-Images For Kernel PCA De-Noising Trine Julie Abrahamsen and Lars Kai Hansen IEEE International Workshop on Machine Learning for Signal Processing, September 3, 2009

2. Outline • Introduction • Kernel PCA • The Pre-image Problem • Input Space Distance Regularization • Conclusions

3. Introduction Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions Definition of kernel function The Gaussian kernel • Extensive studies showed that current methods suffered from instabilities for very non-linear kernels • We therefore suggest to stabilize the pre-image estimate by including input space distance regularization Mika et al. 1999 Schölkopf et al. 2001

4. Kernel Principal Component Analysis Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions Linear PCA is performed in feature space. Thus, the first PC can be found as the normal direction, v1 , by All solutions must lie in the span of the training images, hence, The projection of onto the i’thPC can be found as While the projection onto the subspace spanned by the first q PCs is given by Schölkopf et al. 1998

5. The Pre-Image Problem Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions The pre-image problem = reconstruction of point in input space from feature space point Ill-posed due to properties of the -map. Relax search to find approximate pre-image Common methods seek to minimize the feature space distance where Mika et al. 1999 Schölkopf et al. 1999

6. Overview of Current Estimation Schemes Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions Mika et al. (1999) Kwok & Tsang (2004) Dambreville et al. (2006)

7. Input Space Distance Regularization Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions Which is equivalent to minimizing For RBF kernels the costfunction (whichshouldbemaximized) reduces to

8. Experiments on the USPS Data Set Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions USPS digitsGaussiannoiseadded Mika et al. Input spaceregularization Hull 1994

9. Experiments on the USPS Data Set Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions Evaluating the stability by confidence intervals on the mean squared error. Kwok & Tsang (2004) Mika et al. (1999) Dambreville et al. (2006) Input Space Reg.

10. Experiments on the USPS Data Set Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions Evaluating the sensitivity to initialization by the mean pairwise distance between de-noised images when the methods are initialized in random training points.

11. Conclusions Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions • Several of the current pre-image algorithms were shown to suffer from instabilities • Adding input space distance regularization stabilizes the pre-image with limited sacrifice in terms of de-noising efficiency Future Work • Derive guidelines for choosing λ • Introduce other types of regularization

12. References

13. Input Space Regularization Stabilizes Pre-Images For Kernel PCA De-Noising Trine Julie Abrahamsen and Lars Kai Hansen IEEE International Workshop on Machine Learning for Signal Processing, September 3, 2009

14. Distance distortions for non-linearkernels Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions

15. Many local minima with almost equal value Introduction - Kernel PCA - The Pre-image Problem – Input Space Regularization - Conclusions