A hierarchical neural network structure for text learning is obtained through self-organization

1. LTM cell (“ARRAY”) Layer 2 … PN Layer 3 Layer 4 Layer 6 “A” “Y” “R” Fig. 1 LTM cell with minicolumns [4] Hierarchical Neural Network for Text Based LearningJanusz A Starzyk, BasawarajOhio University, Athens, OH X4 B B A B C A A • Rules Contd. A B B X2 X3 X4 X1 X1 X3 X2 B A Fig. 3 If create a new node C. Fig. 2 If create a new node C. Fig. 4 Neuron “A” with single output is merged with Neuron “B”, and “A” is removed. Introduction This leads to a sparse hierarchical structure. The higher layers represent more complex concepts. Basic nodes in this network are capable of differentiating input sequences. Sequence learning is prerequisite to building spatio-temporal memories. This is performed using laminar minicolumn [3] LTM cells (Fig.1) In such networks, the interconnection scheme is naturally obtained through sequence learning and structural self-organization. No prior assumption about locality of connections or structure sparsity is made. Machine learns only inputs useful to its objectives, a process that is regulated by reinforcement signals and self organization. References Traditional approach is to describe semantic network structure and/or probabilities of transition in associated Markov models Biological networks learn Different Neural Network structures, but common goal • Simple and efficient to solve the given problem • Sparsity is essential Size of the network and time to train important for large data sets Hierarchical structure of identical processing units was proposed [1] Layered organization and sparse structure is biologically inspired Neurons on different layers interact through trained links Mountcastle, V. B., et. al, Response Properties of Neurons of Cat’s Somatic Sensory Cortex to Peripheral Stimuli, J. Neurophysiology, vol. 20, 1957, pp. 374-407. Rogers, T. T., McClelland, J. L., Semantic Cognition text: A parallel Distributed Processing Approach, 2004, MIT Press . Grossberg, S., How does the cerebral cortex work? Learning, attention and grouping by the laminar circuits of visual cortex. Spatial Vision, 12, 163-186,1999. Starzyk J.A., Liu Y., Hierarchical spatio-temporal memory for machine learning based on laminar minicolumn structure, 11th ICCNS, Boston, 2007. Hierarchical Network A hierarchical neural network structure for text learning is obtained through self-organization Similar representation for text based semantic network was used [2] An input layer takes in characters, then learns and activates words stored in memory Direct activation of words requires large computational cost for large dictionaries Extension to phrases, sentences or paragraphs would render such a network impractical due to associated computational cost Computer memory required would also be tremendously large Batch Mode: All words used for training are available at initiation. Network simplification & optimization is done by processing - all the words in the training set. Total number of neurons is 23% higher than the reference (6000) Dynamic Mode: Words used for training are increased incrementally, - one word at a time Simplification & optimization is done by processing - one word at a time. Total number of neurons is 68% higher than the reference (6000) Implementation Tests were run with dictionary up to 6000 words The percent reduction in number of interconnections increases (by up to 65 – 70%) as the number of words increase. The time required to process network activation for all the words used decreases as the number of words increases (reduction by a factor of 55, in batch mode; and 35, in dynamic mode; for 6000 words). Dynamic implementation takes longer compared to the batch implementation, mainly due to the additional overhead required for bookkeeping. The savings (connections and activations) obtained in case of dynamic implementation are less compared to the batch implementation Combination of both methods is advisable for continuous learning and self-organization. Results and Conclusion Fig. 5 Neuron “B” with single input is merged with Neuron “A”, and “B” is removed. Network Simplification • Proposed approach uses intermediate neurons to lower the computational cost • Intermediate neurons decrease number of activations associated with higher level neurons • This concept can be extended to associations of words • Small number of rules for concurrent processing are used • We can arrive at local optimum of network structure / performance • The network topology is self-organizing through addition and removal of neurons and redirecting of neuron connections • Neurons are described by their sets of input and output neurons • Local optimization criteria are checked by searching the set SLA before the structure is updated when creating or merging the neurons. • where ILX is the input list of neuron X and • OLA is the output list of neuron A Rules for Self-Organization • Few simple rules used for self organization X4 X3 X2 X1 X4 X1 X2 X3 C A B A