ベーテ自由エネルギーに対する CCCP アルゴリズムの拡張

1. ベーテ自由エネルギーに対するCCCPアルゴリズムの拡張ベーテ自由エネルギーに対するCCCPアルゴリズムの拡張 東京工業大学総合理工学研究科 知能システム科学専攻　渡辺研究室　 　　西山　悠，　渡辺澄夫

2. Background Probabilistic information processing • Inference for Bayesian networks • Error correcting codes noise decode 000111 000101 000101 • CDMA multi-user detection • Probabilistic image processing restored image degrade image Task High-dimensional distribution Marginals Intractable

3. Task Intractable • Belief propagation (Sum product algorithm) Bethe free energy Optimization techniques • BO (M. Welling, et. al., 2001) • CCCP (A. L. Yuille, 2002) • SEQ (Tonosaki, et. al., 2007) (Approximate) marginals

4. Performance of CCCP CCCP • guarantees to monotonically decrease Bethe free energy unlike belief propagation. • requires huge computational costcompared to belief propagation. • does not always converge for synchronous update of inner loop.

5. Purpose We extend CCCP algorithm for Bethe free energy and present a new CCCP (NCCCP) algorithm. • is more stable even for synchronous inner loop. New CCCP (NCCCP) algorithm • includes conventional CCCP. • guarantees to monotonically decrease Bethe free energy. • can reduce huge computational costunderlying CCCP.

6. Convex Concave Concave and convex procedure(CCCP) CCCP algorithm guarantees to monotonically decrease the function(al) by the update rule Extremum

7. Bethe free energy Bethe free energy Concave Convex Conventional CCCP Approximate marginals

8. Trivial pair creation Main result (Key idea) NCCCP an arbitrary convex functional Bethe free energy

9. Free parameters NCCCP Particularly, CCCP

10. NCCCP algorithm CCCP algorithm Outer loop Outer loop Inner loop Inner loop Outer loop Outer loop Inner loop Inner loop Outer loop Outer loop Approximate marginals Approximate marginals

11. where satisfies for all NCCCP algorithm for Bethe free energy Theorem (Outer loop) Outer loop of NCCCP algorithm for minimizing Bethe free energy is given by as follows:

12. NCCCP algorithm for Bethe free energy Theorem (Inner loop) Inner loop of NCCCP is given by as follows:

13. Update Manner of Inner Loop Theorem (Inner loop) Inner loop of NCCCP is given by as follows: Asynchronous update time time time CCCP guarantees to converge. Synchronous update time time time CCCP does not always converge. NCCCP converges.

14. Role of free parameters Whenare large Whenare small Outer loop Outer loop fast slow Outer loop or fast Outer loop Outer loop convergence slow fast Outer loop or Outer loop fast Outer loop convergence Approximate marginals Approximate marginals There exist optimal values .

15. Numerical Results(1/2) (i) Asynchronous inner loop CCCP CCCP

16. Numerical Results (2/2) (ii) Synchronous inner loop CCCP CCCP

17. Conclusion We presented a new CCCP (NCCCP) algorithm for Bethe free energy. New CCCP (NCCCP) algorithm • includes conventional CCCP. • guarantees to monotonically decrease Bethe free energy. • is more stable even for synchronous inner loop. • can reduce huge computational costunderlying CCCP.

18. Future works • To design efficient NCCCP algorithm based on the optimality of free parameters. • To apply NCCCP algorithm to practical problems such as CDMA multi-user detection problems or decoding algorithm for LDPC codes. 1.外崎幸徳，樺島祥介, “CCCPに基づくCDMAマルチユーザ検出アルゴリズム,” 電子情報通信学会論文誌 D, vol. J89-D, no. 5, pp. 1049-1060, 2006. 2.T. Shibuya, K. Harada, R. Tohyama, and K. Sakaniwa, “Iterative Decoding Based on the Concave-Convex Procedure,” IEICE Trans. Fundam. Electron. Commu. Comput. Sci., vol. E88-A, no. 5, pp. 1346-1364, 2005.

19. Example: Gaussian Distributions NCCCP algorithm= Outer loop + Inner loop Theorem (Outer loop in Gaussian distributions) Theorem (Inner loop in Gaussian distributions)