Bayesian Generalized Product Partition Model

1. Bayesian Generalized Product Partition Model By David Dunson and Ju-Hyun Park Presentation by Eric Wang 2/15/08

2. Outline • Introduce Product Partition Models (PPM). • Relate PPM to DP via the Blackwell-MacQueen Polya Urn scheme. • Introduce predictor dependence into PPM to form Generalized PPM (GPPM). • Discussion and Results • Conclusion

3. Product Partition Model • A PPM is formally defined as • Where is a partition of . • Let denote the data for subjects in cluster h, h = 1,…,k. • Therefore, the probability of partition is therefore the product of all its independent subsets. • The posterior cohesion on after seeing data is also a PPM, (1)

4. Product Partition Model • A PPM can also be induced hierarchically • Where if , . • Taking induces a nonparametric PPM. • A prior on the weights imposes a particular form on the cohesion: a convenient choice corresponds to the Dirichlet Process.

5. Relating DP and PPM • In DP, . • G is seen in stick breaking. If it is marginalized out, it yields the Blackwell-MacQueen (1973) formulation: • Where is the unique value taken by the ith data. • The joint distribution of the a particular set is therefore due to the independence of the data.

6. Relating DP and PPM • It can be shown directly that the Blackwell-MacQueen formulation leads to • Where is the number of data taking unique value . • is the unique value of the subject in cluster h, re-sorted by their ids: • Also, , is a normalizing constant and the cohesion is Then: (2) (3)

7. Relating DP and PPM • From slide 3, writing the prior and likelihood together: • Notice that from (1), G can be marginalized out to get the same form • Specifically, integrate over all possible unique values which can be taken by for subset h. (4)

8. Relating DP and PPM • Therefore, DP is a special case of PPM with cohesion and normalizing constant . • However, (2) follows the premise of DP that data is exhcangeable and does not incorporate dependence on predictors. • Next, PPMs will be generalized such that predictor dependence is incorporated.

9. Generalized PPM • The goal of the paper is to formulate (1) such that the cohesion depends on the subject’s predictor: • This can be done following a process very similar to the non-predictor case above. • Once again, the connection between DP and PPM will be used, this will henceforth be referred to as GPPM • The formulation is interesting because the predictors will be treated as random variables rather than known fixed values (as in KSBP).

10. GPPM • Consider the following hierarchical model • Where , constitutes a base measure on and , the parameters of the data and predictor, respectively. • This model will segment data {1,…,n} into k clusters. As before, denotes that subject i belongs to cluster h. • and , which denote the unique values of the parameters associated with the subject and its predictor, shown below

11. GPPM (5) • The joint distribution of can be developed in a similar manner to (2): • The conditional distribution of given predictors is • For comparison, (2) is shown below: • The cohesion in (6) is • (7) meets the criteria originally set out. (6) (2) (7)

12. GPPM • Some thoughts on GPPM so far: • As noted earlier the posterior distribution of PPMs are still in the class of PPMs, but with updated cohesion. • Similiarly, the posterior of a GPPM will also take the form of a GPPM • (2) and (6) are quite similar. The extra portion of (6) is the marginalized probability of the predictor . • If , then the GPPM reverts to the Blackwell-MacQueen formulation, seen clearly in the following theorem.

13. Generalized Polya Urn Scheme • The following theorem shows that the GPPM can induce a Blackwell-MacQueen Polya Urn scheme, generalized for predictor dependence:

14. Generalized Polya Urn Scheme • By the above theorem, data i will do either 1) or 2) • 1) Draw a previously unseen unique value proportional to the concentration parameter and the base measure on the predictor • 2) Draw a previously used unique value equal to the parameters of cluster h proportional to the number of data which have previously chosen that unique value and the marginal likelihoods of its predictor value across the clusters. • Further, since the predictors are treated as random variables, updating the posteriors on each cluster’s predictor parameters means that GPPM is a flexible, non-parametric way to adapt the distance measure in predictor space. • In this paper G is always integrated out; however, Dunson alludes to variational techniques which could still be developed in similar fashion following the fast Variational DP proposed by Kurihara et al (2006).

15. Generalized Polya Urn Scheme • Consider, for example, a Normal-Wishart prior on the predictor as follows • Where and are multiplicative constants and is a Wishart distribution with degrees of freedom and mean • Notice that this formulation adds another multiplier to the precision of the predictor distribution. This analogously corresponds to kernel width in KSBP, and encourages tight local clustering in predictor space. • The marginal distributions on the predictors from Theorem 1 take the forms shown on the next slide.

16. Generalized Polya Urn Scheme • The marginal distribution of the predictor in the first weight: • The marginal distribution of the predictor in the second weight has the same functional form but with updated hyperparameters: Non-central multivariate t-distribution with degrees of freedom Mean and scale where And is the empirical mean of the predictors in cluster h, without predictor i.

17. Generalized Polya Urn Scheme • Posterior updating in this model is straightforward using MCMC. The conditional posterior of the parameters is • The indicators are updated separately from the cluster parameters . The membership indicators are sampled from it multinomial posterior: • Next, update the parameters conditioned on and number of clusters k. where is the base prior updated with the data likelihood and the weights from Theorem 1

18. Results • Dunson et al. demonstrates results using the following model on conditional density regression problems • Where • Demonstrate results on 3 datasets: • Simulated Single Gaussian (p = 2) • Simulated Mixture of two Gaussians (p = 2) • Epidemiology data (p = 3) P-dimensional predictor Data likelihood Parameters of cluster h.

19. Results • Simulated single Gaussian data, 500 data points • is generated iid from a uniform distribution over (0,1). • Data was simulated using • Algorithm was run for 10,000 iterations with 1,000 iteration burn-in. Fast mixing and good estimates. Raw Data Below are conditional distributions on y for two different values of x. The dotted lines is truth, the solid line is the estimation, and the dashed lines are 99% credibility intervals y x

20. Results • Simulated 2 Gaussian results, 500 data points • is generated iid from a uniform distribution over (0,1). • Data was simulated using PPM GPPM Here, the left column of plots are for a PPM (non-generalized, while the right column plots is the GPPM on the same dataset. Notice much better fitting in the bottom plots, and that the GPPM is not dragged toward 0 as the second peak appears when approaches 0.

21. Results • Epidemiologic Application: • DDE is shown to increase the rate of pre-term birth. Two predictors and correspond to DDE dose for child i, and mother’s age after normalization, respectively. • Dataset size was 2,313 subjects. • MCMC GPPM was run for 30,000 iterations with 10,000 iteration burn-in. • The results confirmed earlier findings that DDE causes a slightly decreasing trend as DDE level rises. • These findings are similar to previous KSBP work on the same dataset, but the implementation was simpler.

22. Results Raw Data Dashed lines indicate 99% credibility intervals

23. Conclusion • A GPPM was formulated beginning with the Blackwell-MacQueen Polya Urn scheme. • The GPPM incorporates predictor dependence by treating the predictor as a random variable. • It is similar in spirit to the KSBP, but is able to bypass issues such as kernel width selection and the inability to implement a continuous distribution in predictor space. • Future research directions could explore Dunson’s mention of a variational method similar to the formulation proposed in this paper.