Bayesian Learning for Conditional Models

1. Bayesian Learning for Conditional Models Alan Qi MIT CSAIL September, 2005 Joint work with T. Minka, Z. Ghahramani, M. Szummer, and R. W. Picard

2. Motivation • Two types of graphical models: generative and conditional • Conditional models: • Make no assumptions about data generation • Enable the use of flexible features • Learning conditional models : estimating (distributions of) model parameters • Maximum likelihood approaches: overfitting • Bayesian learning

3. Outline • Background • Conditional models for independent and relational data classification • Bayesian learning • Bayesian classification and Predictive ARD • Feature selection • Fast kernel learning • Bayesian conditional random fields • Contextual object recognition/Segmentation • Conclusions

4. Outline • Background • Conditional models • Bayesian learning • Bayesian classification and Predictive ARD • Bayesian conditional random fields • Conclusions

5. t1 t2 t1 t2 x1 x2 w x1 x2 w Graphical Models

6. Bayesian learning Simple: Given prior distributions and data likelihoods, estimate the posterior distributions of model parameters or the predictive posterior of a new data point. Difficult: calculating the posterior distributions in practice. • Randomized methods: Markov Chain Monte Carlo, Importance Sampling • Deterministic approximation: Varitional methods, Expectation propagation.

7. Outline • Background • Bayesian classification and Predictive ARD • Feature selection • Fast kernel learning • Bayesian conditional random fields • Conclusions

8. Goal Task 1: Classify high dimensional datasets with many irrelevant features, e.g., normal v.s. cancer microarray data. Task 2: Sparse Bayesian kernel classifiers for fast test performance.

9. Part 1: Roadmap • Automatic relevance determination (ARD) • Risk of Overfitting by optimizing hyperparameters • Predictive ARD by expectation propagation (EP): • Approximate prediction error • EP approximation • Experiments • Conclusions

10. Bayesian Classification Model Labels: t inputs: X parameters: w Likelihood for the data set: Prior of the classifier w: Where is a cumulative distribution function for a standard Gaussian.

11. Evidence and Predictive Distribution The evidence, i.e., the marginal likelihood of the hyperparameters : The predictive posterior distribution of the label for a new input :

12. Automatic Relevance Determination (ARD) • Give the classifier weight independent Gaussian priors whose variance, , controls how far away from zero each weight is allowed to go: • Maximize , the marginal likelihood of the model, with respect to . • Outcome: many elements of go to infinity, which naturally prunes irrelevant features in the data.

13. Two Types of Overfitting • Classical Maximum likelihood: • Optimizing the classifier weights w can directly fit noise in the data, resulting in a complicated model. • Type II Maximum likelihood (ARD): • Optimizing the hyperparameters corresponds to choosing which variables are irrelevant. Choosing one out of exponentially many models can also overfit if we maximize the model marginal likelihood.

14. Evd-ARD-1 Bayes Point Evd-ARD-2 Risk of Optimizing X: Class 1 vs O: Class 2

15. Predictive-ARD • Choosing the model with the best estimated predictive performanceinstead of the most probable model. • Expectation propagation (EP) estimates the leave-one-out predictive performance without performing any expensive cross-validation.

16. Estimate Predictive Performance • Predictive posterior given a test data point • EP can estimate predictive leave-one-out error probability • where q( w| t\i) is the approximate posterior of leaving out the ith label. • EP can also estimate predictive leave-one-out error count

17. Expectation Propagation in a Nutshell • Approximate a probability distribution by simpler parametric terms: • Each approximation term lives in an exponential family (e.g. Gaussian)

18. EP in a Nutshell Three key steps: • Deletion Step: approximate the “leave-one-out” predictive posterior for the ithpoint: • Minimizing the following KL divergence by moment matching: • Inclusion: The key observation: we can use the approximate predictive posterior, obtained in the deletion step, for model selection. No extra computation!

19. Comparison of different model selection criteria for ARD training • 1st row: Test error • 2nd row: Estimated leave-one-out error probability • 3rd row: Estimated leave-one-out error counts • 4th row: Evidence (Model marginal likelihood) • 5th row: Fraction of selected features The estimated leave-one-out error probabilities and counts are better correlated with the test error than evidence and sparsity level.

20. Gene Expression Classification Task: Classify gene expression datasets into different categories, e.g., normal v.s. cancer Challenge: Thousands of genes measured in the micro-array data. Only a small subset of genes are probably correlated with the classification task.

21. Classifying Leukemia Data • The task: distinguish acute myeloid leukemia (AML) from acute lymphoblastic leukemia (ALL). • The dataset: 47 and 25 samples of type ALL and AML respectively with 7129 features per sample. • The dataset was randomly split 100 times into 36 training and 36 testing samples.

22. Classifying Colon Cancer Data • The task: distinguish normal and cancer samples • The dataset: 22 normal and 40 cancer samples with 2000 features per sample. • The dataset was randomly split 100 times into 50 training and 12 testing samples. • SVM results from Li et al. 2002

23. Bayesian Sparse Kernel Classifiers • Using feature/kernel expansions defined on training data points: • Predictive-ARD-EP trains a classifier that depends on a small subset of the training set. • Fast test performance.

24. Test error rates and numbers of relevance or support vectors on breast cancer dataset. 50 partitionings of the data were used. All these methods use the same Gaussian kernel with kernel width = 5. The trade-off parameter C in SVM is chosen via 10-fold cross-validation for each partition.

25. Part 1: Conclusions • Maximizing marginal likelihood can lead to overfitting in the model space if there are a lot of features. • We propose Predictive-ARD based on EP for • feature selection • sparse kernel learning • In practice Predictive-ARD works better than traditional ARD.

26. Outline • Background • Bayesian classification and Predictive ARD • Bayesian conditional random fields • Contextual object recognition/Segmentation • Conclusions

27. Part 2: Conditional random fields (CRFs) t1 t2 x1 x2 w • Generalize traditional classification model by introducing correlation between labels. • Potential functions: • Model data with interdependence and structure, e.g, web pages, natural languages, and multiple visual objects in a picture. • Information at one location propagates to other locations.

28. Bayesian Conditional Networks • Bayesian training to avoid overfitting • Need efficient training: • The exact posterior of w • The Gaussian approximate posterior of w

29. Learning the parameter w by ML/MAP Maximum likelihood (ML) : Maximize the data likelihood where Maximum a posterior (MAP):Gaussian prior on w ML/MAP problem: Overfitting to the noise in data.

30. EP in a Nutshell • Approximate a probability distribution by simpler parametric terms (Minka 2001): • For Bayesian networks: • For Markov networks: • For conditional classification: • For conditional random fields: Each approximation term or lives in an exponential family (such as Gaussian & Multinomial)

31. EP in a Nutshell (2) • The approximate term minimizes the following KL divergence by moment matching: Where the leave-one-out approximation is

32. EP in a Nutshell (3) Three key steps: • Deletion Step: approximate the “leave-one-out” predictive posterior for the ithpoint: • Minimizing the following KL divergence by moment matching (Assumed Density filtering): • Inclusion:

33. Two Difficulties for Bayesian Training • the partition function appears in the denominator • Regular EP does not apply • the partition function is a complicated function of w

34. Turn Denominator to Numerator (1) • Transformed EP: • Deletion: • ADF: • Inclusion:

35. Turn Denominator to Numerator (2) • Power EP: • Deletion: • ADF: • Inclusion: Power EP minimizes  divergence

36. Approximating the partition function • The parameters w and the labels t are intertwined in Z(w): where k = {i, j} is the index of edges. The joint distribution of w and t: • Factorized approximation:

37. Remove the intermediate level Iterations Flatten Approximation Structure Iterations Iterations Increased efficiency, stability, and accuracy!

38. Model Averaging for Prediction • Bayesian training provides a set of estimated models • Bayesian model averaging combines predictions from all the models to eliminate overfitting • Approximate model averaging: weighted belief propagation

39. BCRFs significantly outperformed ML-trained CRFs. Results on Synthetic Data • Data generation: first, randomly sample input x, fixed true parameters w, and then sample the labels t • Graphical structure: Four nodes in a simple loop • Comparing maximum likelihood trained CRF with BCRFs: 10 Trials. 100 training examples and 1000 test examples.

40. FAQs Labeling • The dataset consists of 47 files, belonging to 7 Usenet newsgroup FAQs. Each file has multiple lines, which can be the header (H), a question (Q), an answer (A), or the tail (T). • Task: label the lines that are questions or answers.

41. FAQs Features

42. Results BCRFs outperform MAP-trained CRFs with a high statistical significance on FAQs labeling.

43. Ink Application: analyzing handwritten organization charts • Parsing a graph into different components: containers vs. connectors

44. Comparing results Results from Bayes Point Machine Results from MAP-trained CRF Results from BCRF

45. Results BCRF outperforms ML and MAP trained-CRFs. BCRF-ARD further improves test accuracy. The results are averaged over 20 runs.

46. Part 2:Conclusions • Bayesian CRFs: • Model the relational data • BCRFs improve the predictive performance over ML- and MAP-trained CRFs, especially by approximate model averaging • ARD for CRFs enables feature selection • More applications: image segmentation and joint scene analysis, etc.

47. Outline • Background • Bayesian classification and Predictive ARD • Bayesian conditional random fields • Conclusions

48. Conclusions • Predictive ARD by EP Gene expression classification: Outperformed traditional ARD, SVM with feature selection • Bayesian conditional random fields FAQs labeling and joint diagram analysis: Beats ML- and MAP-trained CRFs • Future work

49. END

50. Appendix: Sequential Updates • EP approximates true likelihood terms by Gaussian virtual observations. • Based on Gaussian virtual observations, the classification model becomes a regression model. • Then, we can achieve efficient sequential updates without maintaining and updating a full covariance matrix. (Faul & Tipping 02)