Supervised by Prof. Xiaoou Tang & Prof. Jianzhuang Liu

1. Exploring Intrinsic Structuresfrom Samples:Supervised, Unsupervised, andSemisupervised Frameworks Supervised by Prof. Xiaoou Tang & Prof. Jianzhuang Liu

2. Outline • Trace Ratio Optimization • Notations & introductions Preserve sample feature structures Dimensionality reduction • Tensor Subspace Learning Explore the geometric structures and feature domain relations concurrently • Correspondence Propagation Outline

3. Concept. Tensor • Tensor: multi-dimensional (or multi-way) arrays of components Concept

4. Concept. Tensor • real-world data are affected by multifarious factors for the person identification, we may have facial images of different ► views and poses ► lightening conditions ► expressions ► image columns and rows • the observed data evolve differently along the variation of different factors Application

5. Concept. Tensor • it is desirable to dig through the intrinsic connections among different affection factors of the data. • Tensor provides a concise and effective representation. Images Image columns expression pose Image rows Illumination Application

6. Concept. Dimensionality Reduction • Preserve sample feature structures • Enhance classification capability • Reduce the computational complexity Introduction

7. Trace Ratio Optimization. Definition w.r.t. • Positive semidefinite • Orthoganality constraint • Homogeneous property: Optimization over the Grassman manifold • Special case, when W is a vector Generalized Rayleigh Quotient GEVD

8. Trace Ratio Formulation • Linear Discriminant Analysis Trace Ratio Formulation

9. Trace Ratio Formulation • Kernel Discriminant Analysis w.r.t. • Decompose w.r.t. • Let w.r.t. Trace Ratio Formulation

10. Trace Ratio Formulation • Marginal Fisher Analysis Inter-class graph (Penalty graph) Intra-class graph (Intrinsic graph) Trace Ratio Formulation

11. Trace Ratio Formulation • Kernel Marginal Fisher Analysis w.r.t. • Decompose w.r.t. • Let w.r.t. Trace Ratio Formulation

12. Trace Ratio Formulation • 2-D Linear Discriminant Analysis • Left Projection & Right Projection • Fix one projection matrix & optimize the other • Discriminant Analysis with Tensor Representation Concept

13. Trace Ratio Formulation • Tensor Subspace Analysis Trace Ratio Formulation

14. Trace Ratio Formulation Conventional Solution: GEVD Singularity problem of Nullspace LDA Dualspace LDA Trace Ratio Formulation

15. Preprocessing Remove the Null Space of with Principal Component Analysis. from Trace Ratio to Trace Difference

16. What will we do? from Trace Ratio to Trace Difference Objective: Trace Ratio Trace Difference Define Find So that Then from Trace Ratio to Trace Difference

17. What will we do? from Trace Ratio to Trace Difference Thus Constraint Let The Objective rises monotonously! Where are the leading eigen vectors of . We have from Trace Ratio to Trace Difference

18. Main Algorithm 1: Initialization. Initialize as arbitrary column orthogonal matrices. 2: Iterative optimization. For t=1, 2, . . . , Tmax, Do 1. Set. 2. Conduct Eigenvalue Decomposition: 3. Reshape the projection directions 4. 3: Output the projection matrices Main Algorithm Process

19. Traditional Tensor Discriminant algorithms • Two-dimensional Linear Discriminant Analysis Ye et.al • Discriminant Analysis with Tensor Representation Yan et.al • Tensor Subspace Analysis He et.al • project the tensor along different dimensions or ways • solve an trace ratio optimization problem • projection matrices for different dimensions are derived iteratively • DO NOT CONVERGE ! Tensor Subspace Learning algorithms

20. Discriminant Analysis Objective • No closed form solution Solve the projection matrices iteratively: leave one projection matrix as variable while keeping others as constant. Mode-k unfolding of the tensor

21. Discriminant Analysis Objective Trace Ratio: General Formulation for the objectives of the Discriminant Analysis based Algorithms. Between Class Scatter of the unfolded data Within Class Scatter of the unfolded data DATER: Constructed from Image Manifold TSA: Diagonal Matrix with weights Objective Deduction

22. Why do previous algorithms not converge? GEVD The conversion from Trace Ratio to Ratio Trace induces an inconsistency among the objectives of different dimensions! Disagreement between the Objective and the Optimization Process

23. What will we do? from Trace Ratio to Trace Difference Objective: Trace Ratio Trace Difference Define Find So that Then from Trace Ratio to Trace Difference

24. What will we do? from Trace Ratio to Trace Difference Thus Constraint Let The Objective rises monotonously! Where are the leading eigen vectors of . Projection matrices of different dimensions share the same objective We have from Trace Ratio to Trace Difference

25. Main Algorithm 1: Initialization. Initialize as arbitrary column orthogonal matrices. 2: Iterative optimization. For t=1, 2, . . . , Tmax, Do For k=1, 2, . . . , n, Do 1. Set. 2. Compute and . 3. Conduct Eigenvalue Decomposition: 4. Reshape the projection directions 5. 3: Output the projection matrices Main Algorithm Process

26. Highlights of our algorithm • The objective value is guaranteed to monotonously increase; and the multiple projection matrices are proved to converge. • Only eigenvalue decomposition method is applied for iterative optimization, which makes the algorithm extremely efficient. • Enhanced potential classification capability of the derived low-dimensional representation from the subspace learning algorithms. • The first work to give a convergent solution to the general tensor-based subspace learning. Hightlights of the Trace Ratio based algorithm

27. Experimental Results Visualization of the projection matrix W of PCA, ratio trace based LDA, and trace ratio based LDA (ITR) on the FERET database. Projection Visualization

28. Experimental Results Comparison: Trace Ratio Based LDA vs. the Ratio Trace based LDA (PCA+LDA) Comparison: Trace Ratio Based MFA vs. the Ratio Trace based MFA (PCA+MFA) Face Recognition Results.Linear

29. Experimental Results Trace Ratio Based KDA vs. the Ratio Trace based KDA Trace Ratio Based KMFA vs. the Ratio Trace based KMFA Face Recognition Results.Kernelization

30. Experimental Results Testing classification errors on three UCI databases for both linear and kernel-based algorithms. Results are obtained from 100 realizations of randomly generated 70/30 splits of data. Results on UCI Dataset

31. Experimental Results Monotony of the Objective & Projection Matrix Convergence

32. Experimental Results 1. TMFA TR mostly outperforms all the other methods concerned in this work, with only one exception for the case G5P5 on the CMU PIE database. 2. For vector-based algorithms, the trace ratio based formulation is consistently superior to the ratio trace based one for subspace learning. 3. Tensor representation has the potential to improve the classification performance for both trace ratio and ratio trace formulations of subspace learning. Face Recognition Results

33. Explore the geometric structures and feature domain consistency for object registration Geometric Structures & Feature Structures Correspondence Propagation

34. Aim • Exploit the geometric structures of sample features • Objects are represented as sets of feature points • Seek a mapping of features from sets of different cardinalities • Introduce human interaction for correspondence guidance Objective

35. Graph Construction Spatial Graph Similarity Graph

36. From Spatial Graph to Categorical Product Graph Assignment Neighborhood Definition Definition:Suppose and are the vertices of graph and respectively. Two assignments and are neighbors in are neighbors iff both pairs and respectively, namely, and and iff and are neighbors. means where

37. From Spatial Graph to Categorical Product Graph can be derived from: The adjacency matrix of where is the matrix Kronecker product operator. Smoothness along the spatial distribution:

38. Feature Domain Consistency & Soft Constraints Similarity Measure: returns the sum of all elements in T where is matrix Hardamard product and One-to-one correspondence penalty or where and

39. Assignment Labeling Inhomogeneous Pair Labeling Assign zeros to those pairs with extremely low similarity scores. Reliable Pair Labeling Assign ones to those reliable pairs Labeled assignments: Reliable correspondence & Inhomogeneous Pairs

40. Reliable Correspondence Propagation Arrangement: Assignment variables Coefficient matrices Spatial Adjacency matrices arrangement

41. Reliable Correspondence Propagation Objective: Feature domain agreement: Geometric smoothness regularization: One-to-one correspondence penalty: Objective

42. Reliable Correspondence Propagation Relax to real domain & Closed-form Solution: where and Solution

43. Rearrangement and Discretization Inverse process of the element arrangement: Reshape the assignment vector into matrix: Thresholding: Assignments larger than a threshold are regarded as correspondences. Eliciting: Sequentially pick up the assignments with largest assignment scores. Rearrangement & Discretizing

44. Semi-supervised & Unsupervised Frameworks Obscure correspondence guidance: Exact pairwise correspondence labeling: Rough correspondence of image parts Users give exact correspondence guidance Semisupervised & Automatic Systems

45. Experimental Results. Demonstration

46. Experiment. Dataset

47. Experimental Results. Details Automatic feature matching score on the Oxford real image transformation dataset. The transformations include viewpoint change ((a) Graffiti and (b) Wall sequence), image blur ((c) bikes and (d) trees sequence), zoom and rotation ((e) bark and (f) boat sequence), illumination variation ((g) leuven ) and JPEG compression ((h) UBC).

48. Future Works • From point-to-point correspondence to set-to-set correspondence. • Multi-scale correspondence searching. Summary

49. Future Works • From point-to-point correspondence to set-to-set correspondence. • Multi-scale correspondence searching. • Combine the object segmentation and registration. Summary

50. Publications: Publications: [1] Huan Wang, Shuicheng Yan, Thomas Huang and Xiaoou Tang, ‘A convergent solution to Tensor Subspace Learning’, International Joint Conferences on Artificial Intelligence (IJCAI 07 Regular paper) , Jan. 2007. [2] Huan Wang, Shuicheng Yan, Thomas Huang and Xiaoou Tang, ‘Trace Ratio vs. Ratio Trace for Dimensionality Reduction’, IEEE Conference on Computer Vision and Pattern Recognition (CVPR 07), Jun. 2007. [3] Huan Wang, Shuicheng Yan, Thomas Huang, Jianzhuang Liu and Xiaoou Tang, ‘Transductive Regression Piloted by Inter-Manifold Relations ’, International Conference on Machine Learning (ICML 07), Jun. 2007. [4] Huan Wang, Shuicheng Yan, Thomas Huang and Xiaoou Tang, ‘Maximum unfolded embedding: formulation, solution, and application for image clustering ’, ACM international conference on Multimedia (ACM MM07), Oct. 2006. [5] Shuicheng Yan, Huan Wang, Thomas Huang and Xiaoou Tang, ‘Ranking with Uncertain Labels ’, IEEE International Conference on Multimedia & Expo (ICME07), May. 2007. [6] Shuicheng Yan, Huan Wang, Xiaoou Tang and Thomas Huang, ‘Exploring Feature Descriptors for Face Recognition ’, IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP07 Oral), Apri. 2007. Publications