Learning Inhomogeneous Gibbs Models

1. Learning Inhomogeneous Gibbs Models Ce Liu celiu@microsoft.com

2. How to Describe the Virtual World

3. Histogram • Histogram: marginal distribution of image variances • Non Gaussian distributed

4. Texture Synthesis (Heeger et al, 95) • Image decomposition by steerable filters • Histogram matching

5. FRAME (Zhu et al, 97) • Homogeneous Markov random field (MRF) • Minimax entropy principle to learn homogeneous Gibbs distribution • Gibbs sampling and feature selection

6. Our Problem • To learn the distribution of structural signals • Challenges • How to learn non-Gaussian distributions in high dimensions with small observations? • How to capture the sophisticated properties of the distribution? • How to optimize parameters with global convergence?

7. Inhomogeneous Gibbs Models (IGM) A framework to learn arbitrary high-dimensional distributions • 1D histograms on linear features to describe high-dimensional distribution • Maximum Entropy Principle– Gibbs distribution • Minimum Entropy Principle– Feature Pursuit • Markov chain Monte Carlo in parameter optimization • Kullback-Leibler Feature (KLF)

8. 1D Observation: Histograms • Feature f(x): Rd→ R • Linear feature f(x)=fTx • Kernel distance f(x)=||f-x|| • Marginal distribution • Histogram

9. Intuition

10. Learning Descriptive Models =

11. Learning Descriptive Models • Sufficient features can make the learnt model f(x) converge to the underlying distribution p(x) • Linear features and histograms are robust compared with other high-order statistics • Descriptive models

12. Maximum Entropy Principle • Maximum Entropy Model • To generalize the statistical properties in the observed • To make the learnt model present information no more than what is available • Mathematical formulation

13. Intuition of Maximum Entropy Principle

14. Inhomogeneous Gibbs Distribution • Solution form of maximum entropy model • Parameter: Gibbs potential

15. Estimating Potential Function • Distribution form • Normalization • Maximizing Likelihood Estimation (MLE) • 1st and 2nd order derivatives

16. Parameter Learning • Monte Carlo integration • Algorithm

17. y x Gibbs Sampling

18. Minimum Entropy Principle • Minimum entropy principle • To make the learnt distribution close to the observed • Feature selection

19. Feature Pursuit • A greedy procedure to learn the feature set • Reference model • Approximate information gain

20. Proposition The approximate information gain for a new feature is and the optimal energy function for this feature is

21. Kullback-Leibler Feature • Kullback-Leibler Feature • Pursue feature by • Hybrid Monte Carlo • Sequential 1D optimization • Feature selection

22. Acceleration by Importance Sampling • Gibbs sampling is too slow… • Importance sampling by the reference model

23. Flowchart of IGM Obs Samples Obs Histograms IGM MCMC Syn Samples Feature Pursuit KL Feature KL<e N Y Output

24. Toy Problems (1) Feature pursuit Synthesized samples Gibbs potential Observed histograms Synthesized histograms Circle Mixture of two Gaussians

25. Toy Problems (2) Swiss Roll

26. Applied to High Dimensions • In high-dimensional space • Too many features to constrain every dimension • MCMC sampling is extremely slow • Solution: dimension reduction by PCA • Application: learning face prior model • 83 landmarks defined to represent face (166d) • 524 samples

27. Face Prior Learning (1) Observed face examples Synthesized face samples without any features

28. Face Prior Learning (2) Synthesized with 10 features Synthesized with 20 features

29. Face Prior Learning (3) Synthesized with 30 features Synthesized with 50 features

30. Observed Histograms

31. Synthesized Histograms

32. Gibbs Potential Functions

33. Learning Caricature Exaggeration

34. Synthesis Results

35. Learning 2D Gibbs Process

37. CSAIL Thank you! celiu@csail.mit.edu