Discussion led by Chunping Wang ECE, Duke University July 10, 2009

1. Simulation of the matrix Bingham-von Mises-Fisher distribution, with applications to multivariate and relational data Peter D. Hoffto appear in Journal of Computational and Graphical Statistics Discussion led by Chunping Wang ECE, Duke University July 10, 2009

2. Outline • Introduction and Motivations • Sampling from the Vector Von Mises-Fisher (vMF) Distribution (existing method) • Sampling from the Matrix Von Mises-Fisher (mMF) Distribution • Sampling from the Bingham-Von Mises-Fisher (BMF) Distribution • One Example • Conclusions 1/21

3. Introduction Stiefel manifold: set of rank- orthonormal matrices, denoted The matrix Bingham-von Mises-Fisher distribution The matrix von Mises-Fisher distribution – linear term The matrix Bingham distribution – quadratic term 2/21

4. Motivations Sampling orthonormal matrices from distributions is useful for many applications. Examples: • Factor analysis observed matrix latent latent Given uniform priors over Stiefel manifold, 3/21

5. Motivations • Principal components observed matrix, with each row with Eigen-value decomposition Likelihood Posterior with respect to uniform prior 4/21

6. Motivations • Network data , symmetric binary observed matrix, with the 0-1 indicator of a link between nodes i and j. E: symmetric matrix of independent standard normal noise Posterior with respect to uniform prior 5/21

7. Sampling from the vMF Distribution (wood, 1994) the modal vector; , concentration parameter A distribution on the -sphere in constant distribution for any given angle defines the modal direction. 6/21

8. Sampling from the vMF Distribution (wood, 1994) (1) A simple direction ( Proposal envelope ) (2) An arbitrary direction For a fixed orthogonal matrix , 7/21

9. Sampling from the mMF Distribution Rejection sampling scheme 1: uniform envelope rejection region a bound Sample accept when Acceptance region Extremely inefficient 8/21

10. Y Y Y Sampling from the mMF Distribution Rejection sampling scheme 2: based on sampling from vMF Proposal samples are drawn from vMF density functions with parameter , constrained to be orthogonal to other columns of . 9/21

11. Y Y Y Sampling from the mMF Distribution Rejection sampling scheme 2: based on sampling from vMF Proposal samples are drawn from vMF density functions with parameter , constrained to be orthogonal to other columns of . 9/21

12. Y Y Y Sampling from the mMF Distribution Rejection sampling scheme 2: based on sampling from vMF Proposal samples are drawn from vMF density functions with parameter , constrained to be orthogonal to other columns of . Rotate the modal direction 9/21

13. Y Y Y Sampling from the mMF Distribution Rejection sampling scheme 2: based on sampling from vMF Proposal samples are drawn from vMF density functions with parameter , constrained to be orthogonal to other columns of . Rotate the sample to be orthogonal to the previous columns 9/21

14. Y Y Y Sampling from the mMF Distribution Rejection sampling scheme 2: based on sampling from vMF Proposal samples are drawn from vMF density functions with parameter , constrained to be orthogonal to other columns of . Proposal distribution 9/21

15. Sampling from the mMF Distribution Rejection sampling scheme 2: based on sampling from vMF Sample scheme: 10/21

16. Sampling from the mMF Distribution A Gibbs sampling scheme Sample iteratively • Note that . When . • remedy: sampling two columns at a time • Non-orthogonality among the columns of add to the autocorrelation in the Gibbs sampler. • remedy: performing the Gibbs sampler on 11/21

17. Sampling from the BMF Distribution The vector Bingham distribution 12/21

18. Sampling from the BMF Distribution The vector Bingham distribution 12/21

19. Sampling from the BMF Distribution The vector Bingham distribution Better mixing 12/21

20. Sampling from the BMF Distribution The vector Bingham distribution From variable substitution, rejection sampling or grid sampling 12/21

21. Sampling from the BMF Distribution The vector Bingham distribution The density is symmetric about zero 12/21

22. Sampling from the BMF Distribution The vector Bingham-von Mises-Fisher distribution The density is not symmetric about zero any more, is no longer uniformly distributed on . The update of and should be done jointly. The modified step 2(b) and 2(c) are: 13/21

23. Sampling from the BMF Distribution The matrix Bingham-von Mises-Fisher distribution Rewrite 14/21

24. Sampling from the BMF Distribution The matrix Bingham-von Mises-Fisher distribution Sample two columns at a time Parameterize 2-dimensional orthonormal matrices as Uniform pairs on the circle Uniform 15/21

25. Sampling from the BMF Distribution The matrix Bingham-von Mises-Fisher distribution 16/21

26. Example: Eigenmodel estimation for network data 17/21

27. Example: Eigenmodel estimation for network data , symmetric binary observed matrix, with the 0-1 indicator of a link between nodes i and j. E: symmetric matrix of independent standard normal noise Posterior with respect to uniform prior BMF distribution with 18/21

28. Example: Eigenmodel estimation for network data Samples from two independent Markov chains with different starting values 19/21

29. Example: Eigenmodel estimation for network data 20/21

30. Conclusions • The sampling scheme of a family of exponential distributions over the Stiefel manifold was developed; • This enables us to make Bayesian inference for those orthonormal matrices and incorporate prior information during the inference; • The author mentioned several application and implemented the sampling scheme on a network data set. 21/21

31. References • Andrew T. A. Wood. Simulation of the von Mises Fisher distribution. Comm. Statist. Simulation Comput., 23:157-164, 1994 • G. Ulrich. Computer generation of distributions on the m-sphere. Appl. Statist., 33, 158-163, 1984 • J. G. Saw. A family of distributions on the m-sphere and some hypothesis tests. Biometrika, 65, 69-74, 1978