Symmetry Breaking Bifurcations of the Information Distortion

1. Symmetry Breaking Bifurcations of the Information Distortion Dissertation Defense April 8, 2003 Albert E. Parker III Complex Biological Systems Department of Mathematical Sciences Center for Computational Biology Montana State University

2. Goal: Solve the Information Distortion Problem The goal of my thesis is to solve the Information Distortion problem, an optimization problem of the form maxqG(q) constrained byD(q)D0 where •  is a subset of Rn. • G and D are sufficiently smooth in . • G and D have symmetry: they are invariant to some group action. Problems of this form arise in the study of clustering problems or optimal source coding systems.

3. Goal: Another Formulation Using the method Lagrange multipliers, the goal of finding solutions of the optimization problem can be rephrased as finding stationary points of the problem maxqF(q,) = maxq(G(q)+D(q)) where •  [0,). •  is a subset of RNK. • G and D are sufficiently smooth in . • G and D have symmetry: they are invariant to some group action.

4. How: Determine the Bifurcation Structure We have described the bifurcation structure of stationary points to any problem of the form maxqF(q,) = maxq(G(q)+D(q)) where •  [0,). •  is a linear subset of RNK. • G and D are sufficiently smooth in . • G and D have symmetry: they are invariant to some group action.

5. Thesis Topics • The Data Clustering Problem • The Neural Coding Problem • Information Theory / Probability Theory • Optimization Theory • Dynamical Systems • Bifurcation Theory with Symmetries • Group Theory • Continuation Techniques

6. Outline of this talk • The Data Clustering Problem • A Class of Optimization Problems • Bifurcation with Symmetries • Numerical Results

7. The Data Clustering Problem • Data Classification: identifying all of the books printed in 2002 which address the martial art Kempo • Data Compression: converting a bitmap file to a jpeg file YN Y q(YN|Y) : a clustering N objects {yNi} K objects {yi}

8. A Symmetry: invariance to relabelling of the clusters of YN class 1 class 2 YN Y q(YN|Y) : a clustering N objects {yNi} K objects {yi}

9. A Symmetry: invariance to relabelling of the clusters of YN class 2 class 1 YN Y q(YN|Y) : a clustering N objects {yNi} K objects {yi}

10. Requirements of a Clustering Method • The original data is represented reasonably well by the clusters • Choosing a cost function, D(Y,YN) , called a distortion function, rigorously defines what we mean by the “data is represented reasonably well”. • Fast implementation

11. Examplesoptimizing at a distortion level D(Y,YN)  D0 • Deterministic Annealing(Rose 1998) A Fast Clustering Algorithm • max H(YN|Y) constrained by D(Y,YN)  D0 • Rate Distortion Theory(Shannon ~1950) Minimum Informative Compression • min I(Y,YN) constrained by D(Y,YN)  D0

12. Inputs and Outputs and Clustered Outputs Inputs Outputs Clusters Y YN X q(YN |Y) p(X,Y) L objects {xi} N objects {yNi} K objects {yi} • The Information Distortion method clusters the outputs Y into • clusters YNso that the information that one can learn about X • by observing YN , I(X;YN), is as close as possible to the • mutual information I(X;Y) • The corresponding information distortion function is • DI(Y;YN)=I(X;Y) - I(X;YN )

13. Two optimization problems which use the information distortion function • Information Distortion Method(Dimitrov and Miller 2001) • max H(YN|Y) constrained by DI(Y,YN) D0 • max H(YN|Y) +  I(X;YN) • Information Bottleneck Method (Tishby, Pereira, Bialek 1999) • min I(Y,YN) constrained by DI(Y,YN)  D0 • max –I(Y,YN) +  I(X;YN)

14. An annealing algorithm to solve • maxqF(q,) = maxq(G(q)+D(q)) • Let q0 be the maximizer of maxqG(q), and let 0=0. For k 0, let (qk , k)be a solution to maxqG(q) +  D(q). Iterate the following steps until • K=  max for some K. • Perform  -step: Let k+1 =k + dk where dk>0 • The initial guess for qk+1 at k+1 is qk+1(0) = qk +  for some small perturbation . • Optimization: solve maxq (G(q) + k+1 D(q)) to get the maximizer qk+1 , using initial guess qk+1(0) .

15. Application of the annealing method to the Information Distortion problem maxq (H(YN|Y) +  I(X;YN)) when p(X,Y) is defined by four gaussian blobs q(YN |Y) Y YN p(X,Y) Y X 52 objects N objects I(X;YN)=D(q(YN|Y)) 52 objects 52 objects Inputs Outputs

16. Observed Bifurcations for the Four Blob problem: We just saw the optimal clusterings q* at some *=  max. What do the clusterings look like for <  max??

17. q*  Conceptual Bifurcation Structure Bifurcations of q*() Observed Bifurcations for the 4 Blob Problem

18. q*  Observed Bifurcations for the 4 Blob Problem Conceptual Bifurcation Structure ?????? Why are there only 3 bifurcations observed? In general, are there only N-1 bifurcations? What kinds of bifurcations do we expect: pitchfork-like, transcritical, saddle-node, or some other type? How many bifurcating solutions are there? What do the bifurcating branches look like? Are they subcritical or supercritical ? What is the stability of the bifurcating branches? Is there always a bifurcating branch which contains solutions of the optimization problem? Are there bifurcations after all of the classes have resolved ?

19. Bifurcations with symmetry • To better understand the bifurcation structure, we capitalize on the symmetriesof the function G(q)+D(q) • The “obvious” symmetry is that G(q)+D(q) is invariant to relabelling of the N classes of YN • The symmetry group of all permutations on N symbols is SN. switch labels 1 and 3

20. Symmetry Breaking Bifurcations q* 

21. Symmetry Breaking Bifurcations q* 

22. Symmetry Breaking Bifurcations q* 

23. Symmetry Breaking Bifurcations q* 

24. Symmetry Breaking Bifurcations q* 

25. q*  Existence Theorems for Bifurcating Branches • Given a bifurcation at a point fixed by SN , • Equivariant Branching Lemma • (Vanderbauwhede and Cicogna 1980-1) • There are N bifurcating branches, each which have symmetry SN-1. • The Smoller-Wasserman Theorem • (Smoller and Wasserman 1985-6) • There are bifurcating branches which havesymmetry <(N-cycle)p>for every prime p|N, p<N.

26. q*  Existence Theorems for Bifurcating Branches • Given a bifurcation at a point fixed by SN-1 , • Equivariant Branching Lemma • (Vanderbauwhede and Cicogna 1980-1) • Gives N-1 bifurcating branches which have symmetry SN-2. • The Smoller-Wasserman Theorem • (Smoller and Wasserman 1985-6) • Gives bifurcating branches which havesymmetry <(M-cycle)p>for every prime p|N-1, p<N-1 . • When N = 4, N-1=3, there are no bifurcating branches given by SW Theorem.

27. Bifurcation Structure corresponds with Group Structure

28. A partial subgroup lattice for S4 and the corresponding bifurcating directions given by the Equivariant Branching Lemma

29. A partial subgroup lattice for S4 and the corresponding bifurcating directions given by the Smoller-Wasserman Theorem

30. q*  Conceptual Bifurcation Structure

31. The Equivariant Branching Lemma shows that the bifurcation structure from SM to SM-1is … q*  Conceptual Bifurcation Structure Group Structure

32. The Equivariant Branching Lemma shows that the bifurcation structure from SM to SM-1is … q*  Conceptual Bifurcation Structure q* Group Structure 

33. The Smoller-Wasserman Theorem shows additional structure … q*  Conceptual Bifurcation Structure q* Group Structure 

34. The Smoller-Wasserman Theorem shows additional structure … 3 branches from the S4 to S3 bifurcation only. q*  Conceptual Bifurcation Structure q* Group Structure 

35. If we stay on a branch which is fixed by SM , how many bifurcations are there? q*  Conceptual Bifurcation Structure q* 

36. q*  Theorem: There are at exactly K/N bifurcations on the branch (q1/N ,  ) for the Information Distortion problem Conceptual Bifurcation Structure q* There are 13 bifurcations on the first branch Group Structure 

37. Bifurcation theory in the presence of symmetriesenables us to answer the questions previously posed …

38. q*  Observed Bifurcations for the 4 Blob Problem Conceptual Bifurcation Structure ?????? Why are there only 3 bifurcations observed? In general, are there only N-1 bifurcations? What kinds of bifurcations do we expect: pitchfork-like, transcritical, saddle-node, or some other type? How many bifurcating solutions are there? What do the bifurcating branches look like? Are they subcritical or supercritical ? What is the stability of the bifurcating branches? Is there always a bifurcating branch which contains solutions of the optimization problem? Are there bifurcations after all of the classes have resolved ?

39. q*  Conceptual Bifurcation Structure Observed Bifurcations for the 4 Blob Problem ?????? Why are there only 3 bifurcations observed? In general, are there only N-1 bifurcations? There are N-1 symmetry breaking bifurcations from SMto SM-1 for M  N. What kinds of bifurcations do we expect: pitchfork-like, transcritical, saddle-node, or some other type? How many bifurcating solutions are there? There are at least N from the first bifurcation, at leastN-1 from the next one, etc. What do the bifurcating branches look like? They are subcritical or supercritical depending on the sign of the bifurcation discriminator(q*,*,uk) . What is the stability of the bifurcating branches? Is there always a bifurcating branch which contains solutions of the optimization problem? No. Are there bifurcations after all of the classes have resolved ? In general, no.

40. We can explain the bifurcation structure of all problems of the form maxqF(q, ) = maxq (G(q)+D(q)) where •  [0,). •  is a subset of RNK. • G and D are sufficiently smooth in . • G and D are invariant to relabelling of the classes of YN • The blocks of the Hessian q(G+ D) at bifurcation satisfy a set of generic conditions. This class of problems includes the Information Distortion problem.

41. chapter 8 chapter 4 chapter 6 chapter 6 Non-generic Symmetry breaking bifurcation Impossible scenario Saddle-node bifurcation Impossible scenario

42. Continuation techniques providenumerical confirmation of the theory

43. Previously Observed Bifurcation Structure for the Four Blob problem:

44. Previous results: Actual structure: Singularity of F: Singularity of L : * Equivariant Branching Lemma: Previous vs. Actual Bifurcation Structure We used Continuation Techniques and the Theory of Bifurcations with Symmetries on the 4 Blob Problem using the Information Distortion method to get this picture.

45. q* 

46. q*  Smoller-Wasserman Theorem: there are bifurcating branches with symmetry <(1324)2>=<(12)(34)>

47. q*  A closer look …

48. q*  Bifurcation from S4toS3…

49. The bifurcation from S4toS3 is subcritical … (the theory predicted this since the bifurcation discriminator (q1/4,*,u)<0)

50. q*  Bifurcation from S3toS2…