Describing Visual Scenes using Transformed Dirichlet Processes

1. Describing Visual Scenes using Transformed Dirichlet Processes Paper by E. B. Sudderth, A. Torralba, W. T. Freeman, and A. S. Willsky, NIPS 2005 Duke University Machine Learning Group Presented by Kai Ni May 19, 2006

2. Outline • Motivation • Hierarchical Dirichlet Processes (HDP) • Transformed Dirichlet Processes (TDP) • Application on Visual Scene

3. Motivation • Problem – Analyzing the features composing a visual scene, thereby localizing and categorizing the objects in an image. • Goal – Exploit relationships among multiple, partially labeled object categories with little manual supervision and labeling. • Application • Image object detection

4. Generative Models • Constellation models – Use fixed, small set of spatially constrained parts for single objects. • Latent Dirichlet allocation (LDA) – Use a spatially unstructured bag of features extracted from local image patches. • Transformed DP – Extension of nonparametric version LDA (which is HDP), making use of the spatial structure information.

5. Generative Models

6. Dirichlet Processes • A single clustering problem can be analyzed as a Dirichlet processes (DP).

7. Hierarchical Dirichlet Process • Mathematical form

8. HDP – Chinese Restaurant Franchise • First level: within each group, DP mixture • Φj1,…,Φj(i-1), i.i.d., r.v., distributed according to Gj; Ѱj1,…, ѰjTj to be the values taken on by Φj1,…,Φj(i-1), njk be # of Φji’= Ѱjt, 0<i’<i. • Second level: across group, sharing clusters • Base measure of each group is a draw from DP: • Ө1,…, ӨK to be the values taken on by Ѱj1,…, ѰjTj , mk be # of Ѱjt=Өk, all j, t.

9. HDP – CRF graph • The values of  are shared between groups, as well as within groups. This is a key property of HDP. Integrating out G0

10. Transformed Dirichlet Process • An extension of the HDP in which global mixture components undergo a set of random transformations before being reused in each group. • Conditioning on • The discreteness of Gj ensures that transformations are shared between observations within group j.

11. CRF view of TDP • Customers prefer tables t at which many customers njt are already seated. • Sometimes a new table t is chosen. Each new table is assigned with a dish kjt. Popular dishes are more likely to be ordered, but a new dish may also be selected. • Each time the dish is ordered, the recipe is seasoned differently according to

12. Graphical Models

13. Gibbs Sampling • Gibbs sampling variants including table assignment t, cluster assignment k, transformations , and parameters • Sampling scheme is very similar to HDP: • Conjugacy assumption of F ~ Q, H ~ Q and R ~ F to make sampling practical.

14. TDP for Visual Scenes Modeling • Observed data xji = (oji, yji), cluster parameters • Assume the location of same object is different from image to image, transformation are defined on the cluster mean: • Different translation allow the same object cluster to be reused at multiple locations within a single image.

15. Synthetic Data Results • HDP uses a large set of global clusters to discretize the transformations underlying the data, and may have poor generalization for modeling visual scenes.

16. Analyzing Street Scenes

17. Conclusion • HDP is a hierarchical, nonparametric model for sharing information between multiple groups of data. • TDP is an extension of HDP with allows the cluster parameters transform differently before reusing the sharing component. • Visual scenes modeling and detecting is an good application of TDP, which gives a better generalized model than HDP.