1 / 25

Semi-supervised Discriminant Analysis

Semi-supervised Discriminant Analysis. Lishan Qiao. 2009.03.13. Outline. Motivation Locality Preserving Regularization based… Laplacian Linear Discriminant Analysis(LapLDA)[1] Semi-supervised Discriminant Analysis(SDA)[2] Comments: Does Locality Preserving Reg. really work?

teige
Download Presentation

Semi-supervised Discriminant Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Semi-supervised Discriminant Analysis Lishan Qiao 2009.03.13

  2. Outline • Motivation • Locality Preserving Regularization based… • Laplacian Linear Discriminant Analysis(LapLDA)[1] • Semi-supervised Discriminant Analysis(SDA)[2] • Comments: Does Locality Preserving Reg. really work? • Opitimization based… • Semi-supervised Discriminant Analysis Via CCCP(SSDACCCP)[3] • Conclusion [1] J.H.Chen, J.P.Ye, Q.Li, Integrating global and local structures: A least squares framework for dimensionality reduction, CVPR07 [2] D.Cai, X.F.He, J.W.Han, Semi-supervised discriminant analysis, ICCV07 [3] Y. Zhang, D.Y.Yeung, Semi-supervised Discriminant Analysis Via CCCP, ECML PKDD 08 Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  3. Motivation Why to extend LDA Objective function: However, • Small Sample Size (SSS) • Global DR method • Completely supervised method Co-Training Transductive, e.g. Label Propagation Inductive, e.g. LapSVM … Semi-supervised Learning Linear Discriminant Analysis (LDA) is popular supervised DR method. PseudoLDA, PCA+LDA, NullLDA, RLDA,… 2DLDA, TensorLDA,… LapLDA, SDA, SSLDA SDA, SSLDA, SSDACCCP Besides, Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  4. LapLDA Motivation & Objective function Objective function: Motivation: LDA captures the global geometric structure of the data by simultaneously maximizing the between-class distance and minimizing the within-class distance. However, local geometric structure has recently been shown to be effective for dimensionality reduction. LapLDA = LDA + LPP Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  5. LapLDA Experiments & Discussion RLDA 90.90 0.61 ? 82.27 1.72 ↑4.54 Letter (a-m) K=1,2,3,5,10,15,20 81.02 1.31 88.67 2.32 (LapLDA) (RLDA) Does locality preserving Regularizer really work? It seems to only play the role of Tikhonov Regularizer!! Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  6. SDA Motivation & Objective function Objective function: Globality Preserving DA: Only 1 labeled training sample per class Motivation: The labeled data points are used to maximize the separability between different classes and the unlabeled data points are used to estimate the intrinsic geometric structure of the data. SDA=RLDA+LPP=LapLDA+Tikhonov Reg.=LDA+LPP+Tikhonov Reg. Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  7. SDA Experiments & Discussion 1 labeled + 29 unlabeled 1 labeled + 1 unlabeled WOptions = []; WOptions.Metric = 'Cosine'; WOptions.NeighborMode = 'KNN'; WOptions.k = 2; WOptions.WeightMode = 'Cosine'; WOptions.bSelfConnected = 0; WOptions.bNormalized = 1; options = []; options.ReguType = 'Ridge'; options.ReguAlpha = 0.01; options.beta = 0.1; No any parameter! Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  8. Discussion About Locality Preserving Reg. Issue 1 Curse of dimensionality For example, the face space is estimated to have at least 100 dimensions [4] [4] M. Meytlis, L. Sirovich. On the dimensionality of face space. PAMI, 29(7):1262–1267, 2007 1) Graph Construction Although the graph is at the heart of graph-based semi-supervised learning methods, its construction has not been studied extensively. [X. Zhu, SSL_survey, 05-08] Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  9. Discussion About Locality Preserving Reg. 0.84x103 0.92x103 1.90x103 ↑ 4.35% ■ ☆ □ ☆ LDA □ ★ Issue 2 The performance of classification relies heavily on how well the nearest neighbor criterion works in the original high-dimensional space[5]. [5] H. T. Chen, H. W. Chang, and T. L. Liu, Local discriminant embedding and its variants. CVPR, 2005. Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  10. Discussion About Locality Preserving Reg. [6] D. Zhou, O. Bousquet, B. Scholkopf. Learning with Local and Global Consistency.NIPS,2004 LDA: LapLDA: RLDA: [6] SDA: gpDA: Issue 3 Difficulty of Parameter selection Cross-validation? 2) Parameter model vs. non-parametric model Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  11. Related Works Semi-supervised DR Sparsity preserving “regularization” Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  12. Outline • Motivation • Locality Preserving Regularization based… • Laplacian Linear Discriminant Analysis(LapLDA)[1] • Semi-supervised Discriminant Analysis(SDA)[2] • Comments: Does Locality Preserving Reg. really work? • Opitimization based… • Semi-supervised Discriminant Analysis Via CCCP(SSDACCCP)[3] • Conclusion [1] J.H.Chen, J.P.Ye, Q.Li, Integrating global and local structures: A least squares framework for dimensionality reduction, CVPR07 [2] D.Cai, X.F.He, J.W.Han, Semi-supervised discriminant analysis, ICCV07 [3] Y. Zhang, D.Y.Yeung, Semi-supervised Discriminant Analysis Via CCCP, ECML PKDD 08 Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  13. SSDACCCP Motivation 1 2 C 1 0 0 1 0 0 × × × × 0 0 1 ■ × × ? ? ? × × ★ × × ? ? ? × LDA: Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  14. SSDACCCP Formulation 1 2 C 1 0 0 1 0 0 0 1 0 1 2 C 0 1 0 1 0 0 1 0 0 0 0 1 0 0 1 ? ? ? ? ? ? 0 1 0 0 0 1 Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  15. SSDACCCP Formulation Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  16. SSDACCCP Formulation Without loss of generality, D.C. Programming Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  17. SSDACCCP CCCP Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  18. SSDACCCP CCCP Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  19. SSDACCCP CCCP Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  20. SSDACCCP Formulation gradient First-order Taylor expansion Omit constant term Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  21. SSDACCCP Formulation Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  22. SSDACCCP Experiments Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  23. SSDACCCP Experiments Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  24. Conclusion × × × × × × × × × × × ■ × × × × × × × × × × × × × × ★ × × × × × × 1. Data-dependent Regularizer The power of the Locality Preserving Reg. was somewhat overstated. 2. Label estimation via optimization The prior from the practical problem is paramount important. Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

  25. Thanks! Semi-supervised Discriminant Analysis Lishan Qiao 2009-3

More Related