Clustering in Generalized Linear Mixed Model Using Dirichlet Process Mixtures

1. Clustering in Generalized Linear Mixed Model Using Dirichlet Process Mixtures Ya Xue Xuejun Liao April 1, 2005

2. Introduction • Concept drift is in the framework of generalized linear mixed model, but brings new question of exploiting the structuring of auxiliary data. • Mixtures with a countably infinite number of components can be handled in a Bayesian framework by employing Dirichlet process priors.

3. Outline • Part I: generalized linear mixed model • Generalized linear model (GLM) • Generalized linear mixed model (GLMM) • Advanced applications • Bayesian feature selection in GLMM • Part II: nonparametric method • Chinese restaurant process • Dirichlet process (DP) • Dirichlet process mixture models • Variational inference for Dirichlet process mixtures

4. Part I Generalized Linear Mixed Model

5. Generalized Linear Model (GLM) • A linear model specifies the relationship between a dependent (or response) variable Y, and a set of predictor variables, Xs, so that • GLM is a generalization of normal linear regression models to exponential family (normal, Poisson, Gamma, binomial, etc).

6. Generalized Linear Model (GLM) GLM differs from linear model in two major respects: • The distribution of Y can be non-normal, and does not have to be continuous. • Y still can be predicted from a linear combination of Xs, but they are "connected" via a link function.

7. Generalized Linear Model(GLM) DDE Example: binomial distribution • Scientific interest: does DDE exposure increase the risk of cancer? Test on rats. Let i index rat. • Dependent variables: • Independent variable: dose of DDE exposure, denoted by xi.

8. Generalized Linear Model(GLM) • Likelihood function of yi: • Choosing the canonical link , the likelihood function becomes

9. GLMM – Basic Model Returning to the DDE example, 19 labs all over the world participated this bioassay. • There are unmeasured factors that vary between the different labs. • For example, rodent diet. • GLMM is an extension of the generalized linear model by adding random effects to the linear predictor (Schall 1991).

10. GLMM – Basic Model • The previous linear predictor is modified as: , where index lab, index rat within lab . • are “fixed” effects - parameters common to all rats. • are “random” effects - deviations for lab i.

11. GLMM – Basic Model • If we choose xij = zij , then all the regression coefficients are assumed to vary for the different labs. • If we choose zij = 1, then only the intercept varies for the different labs (random intercept model).

12. GLMM - Implementation • Gibbs sampling Disadvantage: slow convergence. Solution: hierarchical centering reparametrisation (Gelfand 1994; Gelfand 1995) • Deterministic methods are only available for logit and probit models. • EM algorithm (Anderson 1985) • Simplex method (Im 1988)

13. GLMM – Advanced Applications • Nested GLMM: within each lab, rats were group housed with three cats per cage. let i index lab, j index cage and k index rat. • Crossed GLMM: for all labs, four dose protocols were applied on different rats. let i index lab, j index rat and k indicate the protocol applied on rat i,j.

14. GLMM – Advanced Applications • Nested GLMM: within each lab, rats were group housed with three cats per cage. Two-level GLMM: level I – lab, level II – cage. • Crossed GLMM: for all labs, four dose protocols were applied on different rats. • Rats are sorted into 19 groups by lab. • Rats are sorted into 4 groups by protocol.

15. GLMM – Advanced Applications • Temporal/spatial statistics: Account for correlation between the random effects at different times/locations. • Dynamic latent variable model (Dunson 2003) Let i index patient and t index follow-up time,

16. GLMM – Advanced Applications • Spatially varying coefficient processes (Gelfand 2003): random effects are modeled as spatially correlated process. Possible application: A landmine field where landmines tend to be close together.

17. Bayesian Feature Selection in GLMM Simultaneous selection of fixed and random effects in GLMM (Cai and Dunson 2005) • Mixture prior:

18. Bayesian Feature Selection in GLMM • Fixed effects: choose mixture priors for the fixed effects coefficients. • Random effects: reparameterization • LDU decomposition of the random effect covariance • Choose mixture prior for the elements in the diagonal matrix.

19. …… Berlin 1 0.01 0.00 34.10 40.90 37.50 Berlin 1 0.01 0.00 35.70 35.60 32.10 Tokyo 0 0.01 0.00 56.50 28.90 27.10 Tokyo 1 0.01 0.00 51.50 29.90 25.90 …… Missing Identification in GLMM • Data table of DDE bioassay • What if the first column is missing? • Unusual case in statistics, so few people work on it. • But this is the problem we have to solve for concept drift.

20. Concept Drift • Primary data Auxiliary data • If we treat the drift variable as random variable, concept drift is a random intercept model - a special case of GLMM.

21. Clustering in Concept Drift K = 51 clusters (including 0) out of 300 auxiliary data points Bin resolution = 1

22. Clustering in Concept Drift • There are intrinsic clusters in auxiliary data with respect to drift value. • “The simplest explanation is best.” Occam Razor Why don’t we instead give each cluster a random effect variable?

23. Clustering in Concept Drift • In usual statistics applications, we know which individuals share the same random effect . • However, in concept drift, we do not know which individuals (data points or features) share the same random-intercept. • Can we train the classifier and cluster the auxiliary data simultaneously? This is a new problem we aim to solve.

24. Clustering in Concept Drift • How many clusters (K) should we include in our model? • Does choosing K actually make sense? • Is there a better way?

25. Part II Nonparametric Method

26. Nonparametric method • Parametric method: the forms of the underlying density functions were known. • Nonparametric method is a wide category, e.g. NN, minmax, bootstrapping... • Nonparametric Bayesian method: make use of the Bayesian calculus without prior parameterized knowledge.

27. Cornerstones of NBM • Dirichlet process (DP) allow flexible structures to be learned and allow sharing of statistical strength among sets of related structures. • Gaussian process (GP) allow sharing in the context of multiple nonparametric regressions (suggest to have a separate seminar on GP)

28. Chinese Restaurant Process • Chinese restaurant process (CRP) is a distribution on partitions of integers. • CRP is used to represent uncertainty over the number of components in a mixture model.

29. Chinese Restaurant Process • Unlimited number of tables • Each table has an unlimited capacity to seat customers.

30. Chinese Restaurant Process The (m+1)th subsequent customer sits at a table drawn from the following distribution: where mi is the number of previous customers at table i and is a parameter.

31. Chinese Restaurant Process Example: The probability that next customer sits at table

32. Chinese Restaurant Process • CRP yields an exchangeable distribution on partitions of integers, i.e., the specific ordering of the customers is irrelevant. • An infinite set of random variables is said to be infinitely exchangeable if for every finite subset , we have for any permutation .

33. Dirichlet Process G0: any probability measure on the reals, : partition. A process is a Dirichlet process if the following equation holds for all partitions: where is a concentration parameter. Note: Dir– Dirichlet distribution, DP - Dirichlet process.

34. Dirichlet Process • Denote a sample from the Dirichlet process as • G is a distribution. • Denote a sample from the distribution G as Graphical model for a DP generating the parameters .

35. Dirichlet Process Properties of DP:

36. Dirichlet Process The marginal probabilities for a new This is Chinese restaurant process.

37. DP Mixtures If F is a normal distribution, this is the a Gaussian mixture model.

38. Applications of DP • Infinite Gaussian Mixture Model (Rasmussen 2000) • Infinite Hidden Markov Model (Beal 2002) • Hierarchical Topic Models and the Nested Chinese Restaurant Process (Blei 2004)

39. Implementation of DP Gibbs sampling • If G0 is a conjugate prior for the likelihood given by F: (Escobar 1995) • Non-conjugate prior: (Neal 1998)

40. Variational Inference for DPM • The goal is to compute the predictive density under DP mixture • Also, we minimized the KL distance between p and a variational distribution q. • This algorithm is based on the stick-breaking representation of DP. (I would suggest to have a separate seminar on stick-breaking view of DP and variational DP.)

41. Open Questions • Can we apply ideas of infinite models beyond identifying the number of states or components in a mixture? • Under what conditions can we expect these models to give consistent estimates of densities? • ... • Specified to our problem: Non conjugate due to sigmoid function