菊池自由エネルギーに対する CCCP アルゴリズムの拡張

菊池自由エネルギーに対するCCCPアルゴリズムの拡張菊池自由エネルギーに対するCCCPアルゴリズムの拡張東京工業大学総合理工学研究科知能システム科学専攻　渡辺研究室　　　西山　悠，　渡辺澄夫

Outline High-dimensional distribution marginals Functional Bethe free energy Kikuchi free energy Optimization techniques (Approximate) marginals BO(M. Welling, et. al., 2001) CCCP(A. L. Yuille, 2002) New CCCP (NCCCP) SEQ(Tonosaki, et. al., 2007)

Performance of CCCP CCCP • guarantees to monotonically decrease Kikuchi free energy unlike belief propagation. • requires huge computational cost compared to belief propagation. • does not always converge for synchronous update of inner loop.

Contribution Purpose We extend CCCP algorithm for Kikuchi free energy and present a new CCCP (NCCCP) algorithm. NCCCP algorithm • includes conventional CCCP • guarantees to monotonically decrease Kikuchi free energy. • is more stable even for synchronous inner loop. • can reduce huge computational cost underlying CCCP.

concave convex Key Idea • NCCCP is based on the generalization of concave and convex decomposition in Kikuchi free energy. Kikuchi free energy = + = + Parameters change the decomposition. = +

Background High-dimensional distribution Marginals Direct calculation is intractable. e.g. Maximization of the Posterior Marginals (MPM) estimation

Energy is Free Energy Minimization of Kullback-Leibler distance from a trial distribution to the target distribution is equivalent to that of the following functional with respect to : (1) Helmholtz Free Energy Entropy is

Kikuchi Free Energy Kikuchi free energy, which is defined on set of regions , is given by as follows: Entropy term consists of weighted sum of entropies in Energy term in (1) is exactly rewritten as this form. is given by Overcounting number where for basic clusters .

Bethe Free Energy When the set of regions is given by Bethe free energy

Example (1) Basic clusters Set of regions Region Graph Intersections of basic clusters Intersections of basic clusters

Overcounting number Kikuchi free energy

Example (2) Basic clusters Set of regions Overcounting number Region Graph Bethe free energy

Concave Convex Procedure (CCCP) CCCP algorithm guarantees to monotonically decrease the function(al) , where is convex and is concave, by the update . Kikuchi free energy consists of such functions. Conventional CCCP Double loop algorithm = Outer loop + Inner loop

Trivial pair creation Particularly, let convex funtional be of the form Free parameters Main Results New CCCP (NCCCP) Parameters satisfy

CCCP A high-dimensional vector changes the decomposition.

NCCCP Free Energy The update rule is obtained by differentiating NCCCP free energy subject to normalization and consistency properties of marginals. NCCCP free energy Lagrange functional

NCCCP Algorithm for Kikuchi (1/2) Theorem (Outer Loop of NCCCP) Outer loop of NCCCP algorithm for minimizing Kikuchi free energy is given by as follows: Here :Lagrange multipliers :overcounting numbers :free parameters :set of regions

NCCCP Algorithm for Kikuchi (2/2) Theorem (Inner Loop of NCCCP) Inner loop of NCCCP is given by as follows:

Remark NCCCP algorithm guarantees to monotonically decrease Kikuchi free energy even if free vector is changed within at every outer loop (i.e, ). CCCP NCCCP algorithm Outer loop Inner loop Outer loop Inner loop Outer loop Approximate marginals

Update Manner of Inner Loop Asynchronous update time time time CCCP guarantees to converge. Synchronous update time time time CCCP does not always converge. NCCCP converges.

Role of the free vector Whenis large Whenis small Outer loop Outer loop slow fast Outer loop or convergence fast Outer loop Outer loop slow fast Outer loop or Outer loop fast convergence Outer loop Approximate marginals Approximate marginals There exists an optimal series of vector .

Example: Gaussian Distributions We consider the special case where Kikuchi free energy is Bethe free energy (i.e., ). the target distribution is Gaussian distributions. Target distribution: Marginals and Lagrange multipliers:

NCCCP for Bethe free energy Theorem (Outer loop in Gaussian distributions) Theorem (Inner loop in Gaussian distributions) Here satisfies

Abbreviated Case We consider the special case where inverse covariance matrix satisfies Corollary (Outer and Inner loops which satisfy (2)) Outer loop Inner loop Here

Stability Expansion of inner loop around the fixed-point Necessary condition for the convergence of inner loop: Conventional CCCP : The condition is very tight. CCCP does not converge for the dense graph. NCCCP The condition can always be satisfied for arbitrary dense graphs by making parameter large.

Numerical Results (Synchronous) CCCP Convergence

Numerical Results (Asynchronous) CCCP

Conclusion We presented a new CCCP (NCCCP) algorithm for Kikuchi free energy. NCCCP algorithm • includes conventional CCCP • guarantees to monotonically decrease Kikuchi free energy. • is more stable even for synchronous inner loop. • can reduce huge computational cost underlying CCCP.

Future works • To apply NCCCP algorithm to practical problems • such as CDMA multi-user detection problems or • decoding algorithm for LDPC codes. Reference M. Welling and Y. W. Teh proceeding of UAI (2001) A. L. Yuille Neural Computation 14 1691 (2002) Y. Tonosaki and Y. Kabashima Interdisciplinary Information Sciences 13 57 (2007) To design efficient NCCCP algorithm based on the optimality of free parameters. T. Shibuya et. al. IEICE Trans. On Fundamentals E88-A5 1346 (2005)

菊池自由エネルギーに対する CCCP アルゴリズムの拡張