Graph-based Consensus Maximization among Multiple Supervised and Unsupervised Models

1. Graph-based Consensus Maximizationamong Multiple Supervised and Unsupervised Models Jing Gao1, Feng Liang2, Wei Fan3, Yizhou Sun1, Jiawei Han1 1 CS UIUC 2 STAT UIUC 3 IBM TJ Watson

2. A Toy Example x1 x2 x1 x2 x1 x2 x1 x2 1 1 x3 x4 x3 x4 x3 x4 x3 x4 2 2 x5 3 x6 x5 x6 x5 x6 x5 x6 3 x7 x7 x7 x7

3. Motivations • Consensus maximization • Combine outputs of multiple supervised and unsupervised models on a set of objects for better label predictions • The predicted labels should agree with the base models as much as possible • Motivations • Unsupervised models provide useful constraints for classification tasks • Model diversity improves prediction accuracy and robustness • Model combination at output level is needed in distributed computing or privacy-preserving applications

4. Related Work (1) • Single models • Supervised: SVM, Logistic regression, …… • Unsupervised: K-means, spectral clustering, …… • Semi-supervised learning, collective inference • Supervised ensemble • Require raw data and labels: bagging, boosting, Bayesian model averaging • Require labels: mixture of experts, stacked generalization • Majority voting works at output level and does not require labels

5. Related Work (2) • Unsupervised ensemble • find a consensus clustering from multiple partitionings without accessing the features • Multi-view learning • a joint model is learnt from both labeled and unlabeled data from multiple sources • it can be regarded as a semi-supervised ensemble requiring access to the raw data

6. Related Work (3)

7. Groups-Objects g1 g4 g7 x1 x2 x1 x2 x1 x2 x1 x2 1 g10 1 g12 x3 x4 x3 x4 x3 x4 x3 x4 2 g5 2 g11 g8 x5 3 x6 x5 x6 x5 x6 x5 x6 3 g3 g6 g2 g9 x7 x7 x7 x7

8. Bipartite Graph [1 0 0] [0 1 0] [0 0 1] object i group j conditional prob vector M1 adjacency …… M2 initial probability M3 …… M4 Objects Groups

9. Objective [1 0 0] [0 1 0] [0 0 1] minimize disagreement M1 Similar conditional probability if the object is connected to the group …… M2 M3 Do not deviate much from the initial probability …… M4 Objects Groups

10. Methodology [1 0 0] [0 1 0] [0 0 1] Iterate until convergence Update probability of a group M1 …… M2 Update probability of an object M3 …… M4 Objects Groups

11. Constrained Embedding groups objects constraints for groups from classification models

12. Ranking on Consensus Structure [1 0 0] [0 1 0] [0 0 1] adjacency matrix M1 …… query M2 M3 personalized damping factors …… M4 Objects Groups

13. Incorporating Labeled Information [1 0 0] [0 1 0] [0 0 1] Objective M1 Update probability of a group …… M2 M3 Update probability of an object …… M4 Objects Groups

14. Experiments-Data Sets • 20 Newsgroup • newsgroup messages categorization • only text information available • Cora • research paper area categorization • paper abstracts and citation information available • DBLP • researchers area prediction • publication and co-authorship network, and publication content • conferences’ areas are known

15. Experiments-Baseline Methods (1) • Single models • 20 Newsgroup: • logistic regression, SVM, K-means, min-cut • Cora • abstracts, citations (with or without a labeled set) • DBLP • publication titles, links (with or without labels from conferences) • Proposed method • BGCM • BGCM-L: semi-supervised version combining four models • 2-L: two models • 3-L: three models

16. Experiments-Baseline Methods (2) • Ensemble approaches • clustering ensemble on all of the four models-MCLA, HBGF

17. Accuracy (1)

18. Accuracy (2)

20. Conclusions • Summary • Combine the complementary predictive powers of multiple supervised and unsupervised models • Lossless summarization of base model outputs in group-object bipartite graph • Propagate labeled information between group and object nodes iteratively • Two interpretations: constrained embedding and ranking on consensus structure • Results on various data sets show the benefits