Bayesian Generalized Kernel Mixed Models

1. Bayesian Generalized Kernel Mixed Models Zhihua Zhang, Guang Dai and Michael I. Jordan JMLR 2011

2. Summary of contributions • Propose generalized kernel models (GKMs) as a framework in which sparsitycan be given an explicit treatment and in which a fully Bayesian methodology can be carried out • Data augmentation methodology to develop a MCMC algorithm for inference • Approach shown to be related Gaussian processes and provide a flexible approximation method for GPs

3. Bayesian approach for kernel supervised learning • The form of the regressor or classifier is given by • For a Mercer kernel, there exists a corresponding mapping (say ), from the input space , such that • This provides an equivalent representation in the feature space, where,

4. Generalized Kernel Models

5. Prior for regression coefficients

6. Sparse models • Recall that the number of active vectors is the number of non-zero components of • We are thus interested in a prior for which allows some components of to be zero

7. Methodology For the indicator vector

8. Graphical model

9. Inference • Gibbs for most parameters • MH for kernel parameters • Reversible jump Markov Chain for • takes 2^n distinct values • For small n, posterior may be obtained by calculating the normalizing constant by summing over all possible values of • For large n, a reversible jump MC sampler may be employed to identify high posterior probability models

10. Automatic choice of active vectors • We generate a proposal from the current value of by one of the three possible moves: Prediction :

11. Sparse Gaussian process for classification Given a function , then is a Gaussian process with zero mean and covariance function and vice versa. Also,

12. Sparse GP classification

13. Results