A self-organizing map for adaptive processing of structured data

1. A self-organizing map for adaptive processing of structured data Advisor ：Dr. Hsu Reporter：Chun Kai Chen Author：Markus Hagenbuchner and Alessandro Sperduti 2003 IEEE TNN

2. Outline • Motivation • Objective • Introduction • Supervised Learning: Recursive Neural Networks • Unsupervised Learning within an SOM Framework • Experimental Evaluation • Conclusions • Personal Opinion

3. Motivation • Recent developments in the area of neural networks produced models capable of dealing with structured data.

4. Objective • Here, we propose the first fully unsupervised model • an extension of traditional self-organizing maps (SOMs) • for the processing of labeled directed acyclic graphs (DAGs)

5. Introduction(1/3) • MANY natural and artificial systems are more appropriately modeled using data structures • This structured representation conveys much more information than a “flat” one, e.g., a vector of numerical features extracted from the image • Supervised: Use feedback from knowledge of correct classification. • Unsupervised: No knowledge of correct classification needed.

6. Introduction(2/3) • In recent years, supervised neural networks have been developed • able to deal with structured data encoded as labeled directed acyclic graphs (DAGs) • Supervised information, however, either may not be available or very expensive to obtain • Thus it is very important to develop models which are able to deal with structured data in an unsupervised fashion

7. Introduction(3/3) • In this paper • we will show how self-organizing maps (SOMs) [22] can be extended to treat complex objects represented by data structures • we extend the standard SOM model [22], so as to allow the mapping of structured objects into a topological map

8. Fundamental Concepts of Data Structures • Given a DAG D • v • denote by ch[v] the set of children of v • the kth child of v by chk[v] • The data structures • we consider are labeled DAGs • labels are tuples of variables • In the following • denote by #(C) the class of DAGs with maximum outdegree c

9. Supervised Learning: Recursive Neural Networks(1/3) • Recursive neural networks described in [29] are neural networks capable of performing mappings from a set of labeled graphs to a set of real vectors

10. Supervised Learning: Recursive Neural Networks(2/3) • Recursive neural networks described in [29] are neural networks capable of performing mappings from a set of labeled graphs to a set of real vectors • IRm where denotes the label space, • while the remaining domains represent the encoded subgraphs spaces up to the maximum outdegree of the input domain I#, • c is the maximum out-degree of DOAGs in I#, • s = source(D) , • ys is the label attached to the super-source of D, and • D(1)…D(c)are the subgraphs pointed by s

11. Supervised Learning: Recursive Neural Networks(3/3) • The function τ • maps from a higher dimensional space (i.e., m+c˙n) to a lower dimensional space (i.e., n). • the role of consists of compressing the information about a node (i.e., the label, and the compressed information about the children of the node) in a vector of dimension • This observation is • fundamental to understand how (1) can be adapted to unsupervised learning within a SOM approach. • In fact, the aim of the SOM learning algorithm is to learn a feature map • which given a vector in the spatially continuous input space I • returns a point in the spatially discrete output display space A

12. Unsupervised Learning within an SOM Framework • We are interested in generalizing (4) to deal with the case , i.e., the input space is a structured domain with labels in . • nilΑ • is a special coordinate vector into the discrete output space • Is represented by the coordinates (-1 -1).

13. Model of Mnode

14. BMU mi=[7, 5, 8] After training x1=[8, 5, 9] x1=[8, 5, 9] x2=[7, 4, 2] … training Data Training Algorithm for Mnode • Initialize SOM • For each input data • Step 1—Competitive Step: • Identify its Best Matching Unit (BMU) • Step 2—Cooperative Step: • Adjust BMU and its neighborhood • Repeat Step 2 till stop criteria met

15. Training Algorithm ForM#

16. 4. Experimental Evaluation • For the experiments, we used an extended version of the Policemen Benchmark, • is a set of labeled DOAGs extracted from images produced • the dataset consists of visual patterns and associated graph structures from three different domains

17. d=1 d=2 overlap

19. Only 3% neurons are utilized over fitting problem Fig. 14. Network performance versus size of the training set. over fitting Fig. 15. Network performance versus the total number of iterations trained.

20. Fig. 16. Network performance versus initial neighborhood radii

21. Fig. 17. Network performance versus the initial learning parameter

22. Conclusion • In this paper • we have described a new way of addressing unsupervised learning problems involving structured objects using an SOM technique • The experiments • clearly show that the model is able to encode structural features in a coherent way • optimal parameters • a network of size 114×87 (number of neurons approximately 1/3 the number of nodes in the training set) • the best set of parameters is • the number of training iterations is greater than 450

23. Personal Opinion • Advantage • Specifically, we showed how an SOM network can be used recursively in order to learn a map defined on DAGs • The major innovation is to treat each leaf node as a standard input for an SOM, and the coordinates of the winning neuron as pointer within a parent node • Disadvantage • The information on the labels, on the other hand, can be encoded finely only if the map is sufficiently large and training parameters are chosen appropriately