1 / 15

Graph Based Multi-Modality Learning

Graph Based Multi-Modality Learning. Hanghang Tong; Jingrui He Carnegie Mellon University Mingjing Li Microsoft Research Asia. Outline. Motivation Graph-based Semi-supervised learning Methods The Linear Fusion Scheme The Sequential Fusion Scheme Justifications Similarity Propagation

fanchon
Download Presentation

Graph Based Multi-Modality Learning

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Graph Based Multi-Modality Learning Hanghang Tong; Jingrui He Carnegie Mellon University Mingjing Li Microsoft Research Asia

  2. Outline • Motivation • Graph-based Semi-supervised learning • Methods • The Linear Fusion Scheme • The Sequential Fusion Scheme • Justifications • Similarity Propagation • Bayesian Interpretation • Graph-based un-supervised learning • Experimental Results • Conclusion ACM/Multimedia 2005

  3. Motivation • Multi-Modality in Multimedia • Video: • Digital Image: color vs. • Web Image: content vs. surrounding text • Traditional methods • Linear combination; super-kernel… • Co-Training… • Multi-view version of EM, DBSCAN… All Vector Model based ! ACM/Multimedia 2005

  4. Motivation (conts) • Two Key issues • How to learning within each modality • How to combine… • A more recent hot topic • Graph-based learning • Spectral Cluster; Eigen Map • Manifold Ranking… • Explore graph-based method in the context of multi-modality! ACM/Multimedia 2005

  5. Notation • n data points • c classes, • Two modalities: a and b • One Affinity Matrix for each modality: • for modality a (nxn) • for modality b (nxn) • Labeling Matrix: (nxc) • Vectorial Function: (nxc) • Learning task: • s ACM/Multimedia 2005

  6. Basic Idea • What is a ‘good’ vectorial function F? • As consistent as possible with • Info from modality a • Info from modality b • Info from Labeled points Y • How to? • Take into account the various constraints by optimization ACM/Multimedia 2005

  7. Linear Fusion Scheme Constrains. from modality b Constrains. from modality a • Optimization strategy • Optimization Solution • Iterative form • Closed form Constrains. from Labels Y Converge ACM/Multimedia 2005

  8. Sequential Fusion Scheme Constrains. from modality a and Y • Optimization strategy • Optimization Solution • Iterative form • Closed form Constrains. from modality b and Converge ACM/Multimedia 2005

  9. Similarity Propagation • Taylor expansion (linear fusion) • Similar result for sequential form Initial Label > Further propagate similarity by a and b; > Fuse the result by weighted sum > Propagate Y by a and b; > Fuse the result by weighted sum ACM/Multimedia 2005

  10. Bayesian Interpretation • Optimal F by MAP (linear form): • Assuming: • Conditional pdf • Prior by modality a • Prior by modality b • Fuse prior by • The above setting leads to… ACM/Multimedia 2005

  11. Extension to Un-Supervised Case • Compare • For one modality: • For two modalities (linear form): • Graph Laplacian Learning • Linear Form: • Sequential From: • Feed it the spectral cluster or embedding… Quite similar ! Independent on Y ! ACM/Multimedia 2005

  12. Experimental Results: Coral Image Sequential Form Linear Form Treat as one modality Color Hist Texture ACM/Multimedia 2005

  13. Experimental Results: Web Image Linear Form Sequential Form Treat as one modality Content Fea Surrounding Text ACM/Multimedia 2005

  14. Conclusion • Study multi-modality learning by graph based method • Propose two schemes for semi-supervised learning • Extend them to un-supervised learning ACM/Multimedia 2005

  15. Q&A The End Thanks ACM/Multimedia 2005

More Related