On the capacity of unsupervised recursive neural networks for symbol processing

1. On the capacity of unsupervised recursive neural networks for symbol processing Prof. Dr. Barbara Hammer Computational Intelligence Group Institute of Computer Science Clausthal University of Technology Nicolas Neubauer, M.Sc. Neural Information Processing Group Department of Electrical Engineering & Computer Science Technische Universität Berlin 29. 8. 2006

2. Overview Introduction GSOMSD models and capacities Main part Implementing deterministic push-down automata in RecSOM Conclusion

3. Unsupervised neural networks Clustering algorithms • Each neuron/prototype i has a weight vector wi • Inputs x are mapped to a winning neuron i such that d(x,wi) is minimal • Training: Adapt weights to minimize some error function Self-Organizing Maps (SOMs): Neurons arranged on lattice • During training, adapt neighbours also • After training, similar inputs  neighbouring neurons • variants without fixed grid (e.g. neural gas) Defined for finite-dimensional input vectors • Question: How to adapt algorithms for inputs of non-fixed length like time series? • E.g. time windowing, statistical analysis, ... Rn R?

4. x1 Cinit Recursive processing of time series • Process each input of time series separately • Along with a representation C of the map’s response to the previous input (the context) • function rep: RN Rr (N=number of neurons) • Ct = rep(d1(t-1), ..., dN(t-1)) • Neurons respond not only to input, but to context also • Neurons require additional context weights c • distance of neuron i at timestep t:di(t) = αd(xt,wi) + βdr(Ct,ci) x2 xt ... rep rep rep C2 Ct C

5. Generalized SOM for Structured Data (GSOMSD) [Hammer, Micheli, Sperduti, Stricker, 04] Unsupervised recursive algorithms: Instances of GSOMSD Varying in • context function rep, distances d, dr • and lattice (metric, hyperbolic, neural gas, ...) Example • Recursive SOM (RecSOM) context stores all neurons’ activations r=N, rep(d1,...,dN) = -exp(d1),...,-exp(dN) each neuron needs N context weights! (memory ~ N^2) • other models store • properties of winning neuron • previous activations only for single neurons

6. Computational capacity 0111…1 != 1111…1 • Ability to keep state information: Equivalence to Finite State Automata (FSA) / regular languages • Decaying context will eventually „forget“ leading 0 ([(…)]) != ([(…)]] • Ability to keep stack information: Equivalence to Pushdown Automata (PDA) / context-free languages • Finite context cannot store potentially infinite stack • … Ability to store at least two binary stacks: Turing Machine Equivalence  connecting context models to Chomsky hierarchy

7. Why capacity matters • Explore dynamics of algorithms in detail • Distinguish power of different models • different contexts within GSOMSD e.g., justify huge memory costs of RecSOM compared to other models • to other approaches e.g., supervised recurrent networks • Supervised recurrent networks: Turing machine equivalence • in exponential time for sigmoidal activation functions[Kilian/Siegelmann ’96] • in polynomial time for semilinear activation functions[Siegelmann/Sontag ’95]

8. [Strickert/Hammer,05] Various recursive models and their capacity * for WTA semilinear context

9. Overview Introduction GSOMSD models and capacities Main part Implementing deterministic push-down automata in RecSOM Conclusion

10. Goal: Build a Deterministic PDA Using • the GSOMSD recursive equation di(t) = αd(xt,wi) + βdr(Ct,ci) • L1 distances for d, dr (i.e., d(a,b) = |a-b|) • parameters • α=1 • β=¼ • modified RecSOM context: • instead of original exp(-di): max(1-di,0) • similar overall shape • easier to handle analytically • additionally, winner-takes-all • required at one place in the proof... • makes life easier overall, however

11. ~ weight space! (not lattice) 1 a 0 b 0 1 Three levels of abstraction • Layerstemporal („vertical“) grouping  feed-forward architecture • Operatorsfunctional grouping  defined operations • Phasescombining operators -> automaton simulations ~

12. First level of abstraction: Feed-forward • One-dimensional input weights w • Encoding function enc: Σ NlInput symbol σi series of inputs (i, ε, 2ε, 3ε, …, (l-1)ε) with ε >> max(i)=|Σ| • Each neuron is active for at most one component of enc • resulting in l layers of neurons • In layer l, we know that only neurons from layer l-1 have been active, i.e. are represented >0 in context • pass on activity from layer to layer

13. (l-1)ε|(…) (l-1)ε|(…) ...ε ... aux2 aux1 σ1,b ε |(0,0,1,1) ε|(1,1,0,0) σ0,b σ1,a σ0,a 1|… 1|(0,1) 1|(…,0,1) 1|(…,1,0) 1|(1,0) 1|… 0|(0,1) 0|… 0|(…,0,1) 0|(…,1,0) 0|… 0|(1,0) Feed-forward: Generic Architecture • Sample architecture • 2 states – S={a,b} • 2 inputs – Σ={σ0,σ1} • Output layer • Network represents state a  a active  C=(0,0,...,1,0) • Network represents state b  b active  C=(0,0,...,0,1) Simulation of a state transition function δ: S x Σ S a b (l-1)ε “Hidden” layers arbitrary intermediate computations ε • Input layer • encoding input X state • get input via input weight • get state via context weight 0,1 enc w|c

14. Second Layer of Abstraction: Operators We might be finished • In fact, we are - for the FSA case However, what about the stack? • looks like γ0 γ1γ1 ... • how to store potentially infinite symbol sequences? General idea: • Encode stack in the winner neuron’s activation • Then build ‘operators’ to • read • modify or • copy the stack by changing the winner’s activation

15. Encoding the stack in neurons’ activations To save a sequence of stack symbols within the map, • turn γ0 γ1γ1 into binary sequence alpha=011 • f4(alpha) = • f4 (□) = 0 • f4 (0) = ¼ • f4 (1) = ¾ • f4 (01) = ¼ + 3/16 • f4 (011) = ¼ + 3/16 + 3/64 • push(s,γ1) = ¼s + ¾ • pop(s,γ0 ) = (s- ¼)*4 • Encode stack in activation: Activation a = 1–¼s  push(a, γ1)= 13/16 -1/16s • pop(a, γ0) = 5/4 – s

16. Operators COPY • copy activation into next layer TOP • identify top stack symbol OR • get activation of active neurons(if any) PUSH • modify activation for push • push(a, γ1)= 13/16 -1/16s POP • modify activation for pop • pop(a, γ0) = 5/4 – s

17. σ1,b σ0,b σ1,a σ0,a Third abstraction: Phases a b

18. The final architecture 0S2 1S3

19. Overview Introduction GSOMSD models and capacities Main part Implementing deterministic push-down automata in RecSOM Conclusion

20. Conclusions • RecSOM: stronger computational capacity than SOMSD/MSOM • Does this mean it‘s worth the cost? • Simulations not learnable with Hebbian learning: Practical relevance questionable • Anyway: Elaborate context (costly) rather hindering for simulations too much context: results in a lot of noise maybe better: simpler models, slightly enhanced for example: MSOM, SOMSD with context variable indicating last winner‘s activation • Turing machines also possible? • Storing two stacks into a real number is possible • Reconstructing two stacks from real number is hard particularly when using only differences • may have to leave constant-size simulations • Other representation of stacks may be required

21. Thanks

22. Aux slide: PDA definition

23. Aux slide: PDA definition suitable for map construction