Enhancing Tensor Subspace Learning by Element Rearrangement

1. Enhancing Tensor Subspace Learning by Element Rearrangement Dong XU School of Computer Engineering Nanyang Technological University http://www.ntu.edu.sg/home/dongxu dongxu@ntu.edu.sg

2. Outline • Summary of our recent works on Tensor (or Bilinear) Subspace Learning • Element Rearrangement for Tensor (or Bilinear) Subspace Learning

3. What is Tensor? Tensors are arrays of numbers which transform in certain ways under coordinate transformations. Vector Matrix 3rd-order Tensor • Bilinear (or 2D) Subspace Learning: each image is represented as a 2nd-order tensor (i.e., a matrix) • Tensor Subspace Learning (more general case): each image is represented as a higher order tensor

4. Definition of Mode-k Product Original Tensor Projection: high-dimensional space ->low-dimensional space Reconstruction: low-dimensional space ->high-dimensional space Product for two Matrices Projection Matrix = New Tensor Original Matrix Projection Matrix New Matrix Notation:

5. Definition of Mode-k Flattening Matrix Tensor Potential Assumption in Previous Tensor-based Subspace Learning: Intra-tensor correlations: Correlations along column vectors of mode-k flattened matrices.

6. Data Representation in Dimensionality Reduction Vector Matrix 3rd-order Tensor Gray-level Image Filtered Image Video Sequence High Dimension Low Dimension PCA, LDA • Rank-1 • Decomposition, 2001 • Shashua • and A. Levin, Our Work Xu et al., 2005 Yan et al., 2005 … Examples Tensorface, 2002 M. Vasilescu and D. Terzopoulos,

7. What is Gabor Features? Gabor features can improve recognition performance in comparison to grayscale features. Chengjun Liu T-IP, 2002 Five Scales … Input: Grayscale Image Eight Orientations Output: 40 Gabor-filtered Images Gabor Wavelet Kernels

8. Why Represent Image Objects as Tensors instead of Vectors? • Natural Representation Gray-level Images (2D structure) Videos (3D structure) Gabor-filtered Images (3D structure) • Enhance Learnability in Real Application Curse of Dimensionality (Gabor-filtered image: 100*100*40->Vector: 400,000) Small sample size problem (less than 5,000 images in common face databases) • Reduce Computation Cost

9. Concurrent Subspace Analysis (CSA) as an Example(Criterion: Optimal Reconstruction) Dimensionality Reduction Reconstruction Input sample Sample in Low- dimensional space The reconstructed sample Objective Function: Projection Matrices? D. Xu, S. Yan, Lei Zhang, H. Zhang et al., CVPR 2005 and T-CSVT 2008

10. Tensorization - New Research Direction:Other Related Works • Discriminant Analysis with Tensor Representation (DATER): CVPR 2005 and T-IP 2007 • Coupled Kernel-based Subspace Learning (CKDA): CVPR 2005 • Rank-one Projections with Adaptive Margins(RPAM):CVPR 2006 and T-SMC-B 2007 • Enhancing Tensor Subspace Learning by Element Rearrangement:CVPR 2007 and T-PAMI 2009 • Discriminant Locally Linear Embedding with High Order Tensor Data(DLLE/T): T-SMC-B 2008 • Convergent 2D Subspace Learning with Null Space Analysis (NS2DLDA): T-CSVT 2008 • Semi-supervised Bilinear Subspace Learning:T-IP 2009 • Applications in Human Gait Recognition • CSA+DATER: T-CSVT 2006 • Tensor Marginal Fisher Analysis (TMFA): T-IP 2007 Note: Other researchers also published several papers along this direction!!!

11. Tensorization - New Research Direction Tensorization in Graph Embedding Framework Google Citation: 174 (until 15-Sep-2009) Direct Graph Embedding Linearization Kernelization Original PCA & LDA, ISOMAP, LLE, Laplacian Eigenmap PCA, LDA, LPP, MFA KPCA, KDA, KMFA Tensorization Type Formulation CSA, DATER, TMFA Example S. Yan, D. Xu, H. Zhang et al., CVPR, 2005 and T-PAMI,2007

12. Element Rearrangement: Motivations • The success of tensor-based subspace learning relies on the redundancy among the unfolded vector • However, such correlation/redundancy is usually not strong for real data • Our Solution: Element rearrangement is employed as a preprocessing step to increase the intra-tensor correlations for existing tensor subspace learning methods

13. Low correlation High correlation Element Rearrangement Sets of highly correlated pixels Columns of highly correlated pixels Motivations-Continued Intra-tensor correlations: Correlations among the features within certain tensor dimensions, such as rows, columns and Gabor features…

14. Problem Definition • The task of enhancing correlation/redundancy among 2nd–order tensor is to search for a pixel rearrangement operator R, such that 1. is the rearranged matrix from sample 2. The column numbers of U and V are predefined After the pixel rearrangement, we can use the rearranged tensors as input for tensor subspace learning algorithms!

15. Solution to Pixel Rearrangement Problem Initialize U0, V0 Compute reconstructed matrices n=n+1 Optimize U and V Optimize operator R

16. Step for Optimizing R • It is Integer Programming problem Sender Original matrix Receiver Reconstructed matrix • Linear programming problem in Earth Mover’s Distance (EMD) has integer solution. • We constrain the rearrangement within spatially local neighborhood or feature-based neighborhood for speedup.

17. Convergence Speed

18. Rearrangement Results

19. Reconstruction Visualization

20. Reconstruction Visualization

21. Classification Accuracy

22. Summary • . • Our papers published in CVPR 2005 are the first works to address dimensionality reduction with the image objects represented as high-order tensors of arbitrary order. • Our papers published in CVPR 2005 opens a new research direction. We also published a series of works along this direction. • Element arrangement can further improve data compression performance and classification accuracy.