Active Learning for Unbalanced Data in the Challenge with Multiple Models and Biasing

1. Active Learning for Unbalanced Data in the Challenge with Multiple Models and Biasing Yukun Chen Subramani Mani Discovery Systems Lab (DSL) Department of Biomedical Informatics Vanderbilt University May 2010

2. Outline • Introduction • Datasets in the challenge • Probabilistic models • Querying methods • Other methods for active learning • Experiments and Results • Conclusion

3. Introduction • Active learning challenge is based on the pool-based active learning model. • Practically, labeling is costly but observational data is abundantly available at low cost. • Active learner could find the most informative instance and perform high learning accuracy with minimal querying cost. • In the challenge, we need to optimize the global score (ALC score) by implementing probabilistic prediction model, querying strategy, and more. • Learning from datasets in the challenge is not easy because the data is very sparse, is unbalanced for class label, has high dimensional feature space, and has missing values. • Uncertainty sampling with biasing consensus (USBC) is our basic active learning strategy for prediction and querying for labels.

4. Datasets in the challenge Development Datasets Final Datasets

5. Probabilistic Model • Random Forests (RF) classifier is the basic prediction model we used in this challenge. • We built a multi-model committee with multiple RF classifiers. • The final prediction was based on consensus posterior probability (CPP): • We also considered the variance of posterior probabilities from multiple models. The high-variance filter was used in querying method.

6. Function of Lease Confidence with Bias for Binary Class 1 LCB for pp=0.1 0.9 LCB for pp=0.3 LCB for pp=0.5 0.8 0.7 0.6 Q(x) 0.5 0.4 0.3 0.2 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 P(y=1|x) Querying Method • Querying method ranks the samples based on the informative values, and outputs the most informative sample(s) to query. • Least confidence with bias (LCB) was our basic querying method. • The informative value of sample is a function of CPP and bias factor pp (the positive fraction for the current training set in active learning). Pmax

7. Semi-supervised Learning Method • It is very important to have a good starting point on the learning curve in active learning, which is the prediction performance by knowing just one positive label. • Pure unsupervised learning method (for example the metric based on distance, similarity, clustering result) might not be good enough to make prediction. • We combined unsupervised and supervised learning: • (1) For all samples, compute the cosine similarity to the positive-labeled seed; • (2) Assign negative labels to K samples with smallest cosine similarity values; • (3) Train the training set with one given positive sample and K predicted negative samples by our multiple models, and predict for other samples. • Here is our comparison result between cosine similarity function and semi-supervised learning method for the initial AUC:

8. Batch Size Validation • For some datasets (ZEBRA, ORANGE, HIVA, and NOVA), our models did not have a good prediction when the size of training set is small. The bad initial performance could badly affect the global score based on learning curve in Log2 space (see the learning curves with respect to initial batch size). • We ran batch size validation to search for the minimal sufficient size of initial training set. • This prevented a significant drop in performance at the beginning for our prediction model. • Batch size validation result figure for ZEBRA, IBN_SINA and NOVA:

9. ZEBRA USBC IBATCH 2^14 (16384): Global score=0.5199 • ZEBRA USBC IBATCH 2^12 (4096): Global score=0.4218 ZEBRA USBC IBATCH 2^10 (1024): Global score=0.3846 1 1 1 0.95 0.95 0.95 0.9 0.9 0.9 0.85 0.85 0.85 0.8 0.8 0.8 Area under the ROC curve Area under the ROC curve Area under the ROC curve 0.75 0.75 0.75 0.7 0.7 0.7 0.65 0.65 0.65 0.6 0.6 0.6 0.55 0.55 0.55 0.5 0.5 0.5 0 5 10 15 0 5 10 15 0 5 10 15 Log (Number of labels queried) Log (Number of labels queried) Log (Number of labels queried) 2 2 2 ZEBRA USBC IBATCH 2^8 (256): Global score=0.3164 ZEBRA USBC IBATCH 2^4 (16): Global score=0.3391 ZEBRA USBC IBATCH 2^1 (2): Global score=0.2876 1 1 1 0.9 0.9 0.9 0.8 0.8 0.8 Area under the ROC curve Area under the ROC curve Area under the ROC curve 0.7 0.7 0.7 0.6 0.6 0.6 0.5 0.5 0.5 0.4 0.4 0.4 0 5 10 15 0 5 10 15 0 5 10 15 Log (Number of labels queried) Log (Number of labels queried) Log (Number of labels queried) 2 2 2 Batch Size Validation (for ZEBRA) Initial Batch: 16384 Initial Batch: 4096 Initial Batch: 1024 Initial Batch: 256 Initial Batch: 16 Initial Batch: 2

10. Experimental Setup • (1) Initialization: • (1.1) Run preprocessing steps (missing value imputation, PCA, etc) if needed. • (1.2) Assign batch size as the function of iteration, depending on the batch size validation result. • (2) Run semi-supervised learning for initial prediction and basic uncertainty sampling to rank and query samples. • (3) Run uncertainty sampling with biasing consensus (USBC) in the iterations of active learning: • (3.1) Add predicted negative samples into the training sets (if activated). • (3.2) Train by 5 RF models and predict for all unlabeled samples. • (3.3) Run high-variance filter (if activated). • (3.4) Run uncertainty sampling with bias to rank and query samples (Bias factor is the function of positive fraction and the size of training set). • (4) Output learning curves and global ALC score.

11. Results (tables for development datasets and final datasets) The results for development datasets The results for final datasets

12. Results (Active Learning Curves for final datasets) Dataset: A; Global score: 0.36 Dataset: B; Global score: 0.13 Dataset: C; Global score: 0.19 Dataset: D; Global score: 0.54 Dataset: E; Global score: 0.63 Dataset: F; Global score: 0.79

13. Discussion • For dataset E, the global score is benefited by the batch size validation. Semi-supervised learning generates a good starting point. We won on dataset E. • For dataset F, the learning curve based on USBC is acceptable except that the initial performance is not stable. We were ranked 3rd on F. • For dataset D also the batch size validation was effective. The high-variance filter successfully helped prevent a significant drop in the curve. But the starting point is quite low. • For dataset A, USBC worked well when the size of training set was at least 64. However, the initial low performance hurt our global score. • Datasets B and C are the hardest datasets like HIVA and ORANGE. Our prediction models were not effective in these datasets.

14. Conclusion • Our strategies consider more than prediction model and query model. Semi-supervised learning and batch size validation are also important parts of the active learning process. • Our methods need further evaluation using additional datasets. • The active learning challenge is still a very open problem to solve. • One possible future direction to explore is to automatically assign batch size as a function of predictive performance and informativeness.