Adaptive Offset Subspace Self-Organizing Map with an Application to Handwritten Digit Recognition

1. MDM/KDD’06 Adaptive Offset Subspace Self-Organizing Map with an Application to Handwritten Digit Recognition Huicheng Zheng, Pádraig Cunningham and Alexey Tsymbal August 20th, 2006 Thanks to the MUSCLE Internal Fellowship (http://www.muscle-noe.org).

2. Outline • Introduction • Learning self-organized subspaces • Adaptive Subspace Self-Organizing Map (ASSOM) • Learning self-organized linear manifolds • Adaptive Offset Subspace Self-Organizing Map (AOSSOM) • Experiments • Cluster separating • Handwritten digit recognition • Conclusions

3. Fundamental Requirements in MDM Among others: • Dimension reduction • Visualization The Self-Organizing Map (SOM) meets these requirements and finds its use in MDM [J.-C. Lamirel et al., 2000] [R. Honda et al., 2000]

4. rectangles circles triangles …… Learning Self-Organized Subspaces • Pattern invariance classes pattern subspaces • For example: Translation-invariant learning • Combining subspace learning and SOM [Kohonen. T., et al., 1997]: Adaptive Subspace Self-Organizing Map (ASSOM)

5. Applications of the ASSOM to MDM • Invariant-feature learning[Kohonen, T., et al., 1997] • Speech processing[Hase, H., et al., 1996] • Texture segmentation[Ruiz del Solar, J., 1998] • Image retrieval[De Ridder, D., et al., 2000] • Image classification[Zhang, B., et al., 1999]

6. Learning Linear Manifolds • The ASSOM does not learn linear manifolds adequately. • Faces viewed from different perspectives, variants of handwritten digits can be handled by linear manifolds [Liu, Z. Q., 2001] [Hinton, G. E., et al., 1997].

7. Learning Self-Organized Linear Manifolds • Principal Component Analysis Self-Organizing Map (PCASOM) [López-Rubio, E., et al., 2004] • Combining PCA with SOM • Computationally expensive • Adaptive Manifold Self-Organizing Map (AMSOM) [Liu, Z. Q., 2001] • Attaching mean vectors to the ASSOM • No given objective function • Adaptive Offset Subspace Self-Organizing Map (AOSSOM) [Proposed] • Attaching offset vectors to the ASSOM • Well defined objective function

8. Outline • Introduction • Learning self-organized subspaces • Adaptive Subspace Self-Organizing Map • Learning self-organized linear manifolds • Adaptive Offset Subspace Self-Organizing Map • Experiments • Cluster separating • Handwritten digit recognition • Conclusions

9. j A module representing the subspace L(j) Adaptive Subspace Self-Organizing Map Rectangular topology Hexagonal topology The module arrays in ASSOM

10. Competition and Adaptation • Repeatedly: • Competition: Winning module of • Adaptation: For the winner and the modules i in its neighborhood • Orthonormalize the basis vectors

11. Outline • Introduction • Learning self-organized subspaces • Adaptive Subspace Self-Organizing Map • Learning self-organized linear manifolds • Adaptive Offset Subspace Self-Organizing Map • Experiments • Cluster separating • Handwritten digit recognition • Conclusions

12. Adaptive Offset Subspace Self-Organizing Map (AOSSOM): Linear Manifold Learning • defines a set of linear manifold {Ai} • Learning of can be done by defining a criterion function: • defines an MSE criterion. • defines a spatial interaction between neighboring sites in the AOSSOM. • Using stochastic gradient descent:

13. Outline • Introduction • Learning self-organized subspaces • Adaptive Subspace Self-Organizing Map • Learning self-organized linear manifolds • Adaptive Offset Subspace Self-Organizing Map • Experiments • Cluster separating • Handwritten digit recognition • Conclusions

14. Cluster Separating by Learning Linear Manifolds (1) Nearly optimal parameters were used in all the experiments.

15. Cluster Separating by Learning Linear Manifolds (2) Instability was observed for the AMSOM when initial learning rate . A larger value has been used to alleviate the problem.

16. Cluster Separating by Learning Linear Manifolds (3) • Four random clusters with 2-D Gaussian distributions. • “AMSOM 1” uses the neighborhood-decreasing scheme proposed in [Liu, Z. Q., 2001]: A Gaussian function with the FWHM • “AMSOM 2” uses the same neighborhood function as the AOSSOM: A Gaussian function with the FWHM

17. Cluster Separating by Learning Linear Manifolds (3) Observation of projection error decreasing for D=1. Boldface texts emphasize the cases where the AOSSOM worked the best (8 out of 10 times).

18. Outline • Introduction • Learning self-organized subspaces • Adaptive Subspace Self-Organizing Map • Learning self-organized linear manifolds • Adaptive Offset Subspace Self-Organizing Map • Experiments • Cluster separating • Handwritten digit recognition • Conclusions

19. Application of the AOSSOM to Handwritten Digit Recognition • Large variety of handwriting styles • It is hard for hand-crafted feature extractor to cope with these variations. • Variations create embedded manifolds in the feature space. • Piece-wise linear approximation of embedded manifolds.

20. Handwritten Digit Image Database • MNIST database composed of 60,000 training digits and 10,000 test digits [LeCun, Y., et al., 1998]. • Some examples:

21. MNIST Learning with the AOSSOM • One AOSSOM trained for each digit class • For an input test digit image x, each AOSSOM Mk reconstructs an output .

22. MNIST Learning Performance Summary H: Linear manifold dimension W: No. of rows and columns of AOSSOM

23. Outline • Introduction • Learning self-organized subspaces • Adaptive Subspace Self-Organizing Map • Learning self-organized linear manifolds • Adaptive Offset Subspace Self-Organizing Map • Experiments • Cluster separating • Handwritten digit recognition • Conclusions

24. Conclusions • The AOSSOM permits to learn organized linear manifolds with a well-defined criterion function. • Higher robustness is observed in the experiments with the AOSSOM than with the AMSOM. • The AOSSOM is well suited for MDM tasks where data structures are best handled by piece-wise linear manifolds.

25. References • De Ridder, D., et al., 2000: The adaptive subspace map for image description and image database retrieval. SSPR&SPR. • Hase, H., et al., 1996: Speech signal processing using Adaptive Subspace SOM (ASSOM). Technical Report NC95-140, The Inst. of Electronics, Information and Communication Engineers, Tottori University, Koyama, Japan. • Hinton, G. E., et al., 1997: Modeling the manifolds of images of handwritten digits. IEEE Trans. Neural Networks, 8(1):65–74. • Honda, R., et al., 2000: Semantic indexing and temporal rule discovery for time-series satellite images. In S. J. Simoff and O. R. Zaïane, editors, Proc. First Int. Workshop on Multimedia Data Mining, in conjunction with Sixth ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, pp. 82–90, Boston, MA, USA. • Kohonen, T., et al., 1997: Self-Organized formation of various invariant-feature filters in the adaptive-subspace SOM. Neural Computation 9(6).

26. References • Lamirel, J.-C., et al., 2000: A Self Organizing Map (SOM) extended model for information discovery in a digital library context. In S. J. Simoff and O. R. Zaïane, editors, Proc. First Int. Workshop on Multimedia Data Mining, in conjunction withSixth ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, pp. 60–66, Boston, MA, USA. • LeCun, Y., et al., 1998: Gradient-based learning applied to document recognition. Proc. IEEE, 86(11):2278–2324. • Liu, Z. Q., 2001: Retrieving faces using Adaptive Subspace Self-Organising Map. In Proc. International Symposium on Intelligent Multimedia, Video and Speech Processing, pp. 377–380, Hong Kong. • López-Rubio, E., et al., 2004: A Principal Components Analysis Self-Organizing Map. Neural Networks, 17(2), pp. 261–270. • Ruiz del Solar, J., 1998: Texsom: texture segmentation using Self-Organizing Maps. Neurocomputing 21(1–3). • Zhang, B., et al., 1999: Handwritten digit recognition by adaptive-subspace self-organizing map (ASSOM). IEEE Trans. on Neural Networks 10(4).

27. Thanks and questions?