Unsupervised Recursive Sequence Processing

1. Unsupervised Recursive Sequence Processing Marc Strickert, Barbara Hammer http://www.inf.uos.de/lnm University of Osnabrück, Germany 23 April 2003

2. What we want… Background: from SOM to Hyperbolic SOM for Sequences (H-SOM-S) Examples:Mackey-Glass time series / Reber grammar Outlook and discussion Outline ESANN 2003 · 23 April · Unsupervised Recursive Sequence Processing · Marc Strickert

3. But we do not want to fix the historic context length. Wish List Example: Mackey-Glass chaotic time series Structure within the training sequence should be properly represented in a trained low-dim map. We want to learn the current state, using past events. ESANN 2003 · 23 April · Unsupervised Recursive Sequence Processing · Marc Strickert

4. Helge Ritter (1999) Hyperbolic Self Organizing Maps (HSOM) Con-Fusions Map topology Thomas Voegtlin (2002) Recursive Self Organizing Maps (RSOM) Context representation Markus Hagenbuchner et al.(2003) Self Organizing Maps for Structured Data (SOM-SD) Indexing scheme Hyperbolic Self Organizing Maps for Sequences (H-SOM-S) ESANN 2003 · 23 April · Unsupervised Recursive Sequence Processing · Marc Strickert

5. From SOM to H-SOM J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J J Topology of the neural grid rectangular, Euclidean circular triangle mesh, 6 neighbors, -> Euclidean Exponentially growing contexts -> sequence processing circular triangle mesh, 7 neighbors, -> hyperbolic ESANN 2003 · 23 April · Unsupervised Recursive Sequence Processing · Marc Strickert

6. What’s inside a Neuron? J • Weight • = Sequence element B. Historic context = previous winning position in grid J is a currently presented real number or vector given as a triangle coordinate tuple ESANN 2003 · 23 April · Unsupervised Recursive Sequence Processing · Marc Strickert

7. The Winner Neuron Context control =0->std. SOM • Best matching Neuron: fulfilling both criteria of • having a weight close to the presented sequence element where ‘close‘ is expressed in terms of the Euclidean distance • AND • having a context nearby the previous winner neuron where `nearby`is expressed as a shortest path length in the grid. • Best matching Neuron: fulfilling both criteria of • having a weight close to the presented sequence element where ‘close‘ is expressed in terms of the Euclidean distance • AND • having a context nearby the previous winner neuron where `nearby`is expressed as a shortest path length in the grid. both normalized to [0;1] Dist(Neuroni, SymbolElem) = (1- ) · DistEUC(NeuroniWeight, SymbolElem) +  · DistGRID(NeuroniContext, LastWinner) WinnerIndex = argMini { Dist(Neuroni, SymbolElem) } ESANN 2003 · 23 April · Unsupervised Recursive Sequence Processing · Marc Strickert

8. H-SOM-S Context Update in Action J Coding a neuron‘s context Context update ... as locally Euclidian triangle coordinates Assume N3to be the winner neuron of the previous training step. 1 N3 1 N1 1 N2 coding the context as recent winning location on the map ESANN 2003 · 23 April · Unsupervised Recursive Sequence Processing · Marc Strickert

9. Trained map● 100 neurons● 6 neighbors Example 1: Mackey-Glass Series Receptive fields: 30 time steps Local distortions ESANN 2003 · 23 April · Unsupervised Recursive Sequence Processing · Marc Strickert

10. Temporal quant. Error for Mackey-Glass data past present ESANN 2003 · 23 April · Unsupervised Recursive Sequence Processing · Marc Strickert

11. Example 2: Reber Grammar • Training features: • 7 Symbols, equidist. coded in multi-dimensional Euclidean space. • map radius 5; 7N-hyperb. grid  617 Neurons. •  from 1 ( 0.8 => 20% of historic influence. • 3*106 letters from Reber words presented. ESANN 2003 · 23 April · Unsupervised Recursive Sequence Processing · Marc Strickert

12. Map Ordering for current Reber Letter E B P S S P V V X X E B T T 6-neighbor topology 7-neighbor topology ESANN 2003 · 23 April · Unsupervised Recursive Sequence Processing · Marc Strickert

13. Reber Triplet Reconstruction from H-SOM-S Frequencies of triplet occurrence VPX TTT ESANN 2003 · 23 April · Unsupervised Recursive Sequence Processing · Marc Strickert

14. Summary, Conclusion and Outlook • Recursive data processing is achieved by incorporating both the current pattern and a reference to the last winner. • Hyperbolic neuron grid topology matches requirements of exponentially growing contexts. • Regularities in the data are reflected in the temporal and similarity based ordering in the trained map. • The 2D map can be easily navigated and analyzed. • More sophisticated grid topologies are possible;e.g. tetrahedron meshing instead of triangular mesh. ESANN 2003 · 23 April · Unsupervised Recursive Sequence Processing · Marc Strickert

15. Questions • What does topology preservation mean in the presenceof temporal processing ? • Is there a particularly good choice for the final value of the context control ? ? Your questions, comments or suggestions ? ESANN 2003 · 23 April · Unsupervised Recursive Sequence Processing · Marc Strickert

16. ESANN 2003 · 23 April · Unsupervised Recursive Sequence Processing · Marc Strickert

17. H-SOM-S Neural Reber Word Representation ESANN 2003 · 23 April · Unsupervised Recursive Sequence Processing · Marc Strickert