Institute for Advanced Studies in Basic Sciences – Zanjan

1. Institute for Advanced Studies in Basic Sciences – Zanjan Supervised Kohonen Artificial Neural Networks Mahdi Vasighi

2. Introduction Data Analysis: Linear transformations and data reduction techniques like Principal Component Analysis (PCA) • Advantage: • Projection from high dim. onto a low dim. coordinate system. • Does not require a high level of modelling expertise. • Disadvantages: • Assuming the topology of the input data can be reduced in a linear fashion. • Outliers disturb the quality of the projection. • Visualization power deteriorates considerably if the number of relevant dimensions in the multivariate space remains high after a PCA analysis.

3. An alternative is nonlinear mapping: This explicitly aims to map objects in a low-dimensional space (usually two-dimensional) in such a way that the distances between objects are preserved in the mapping. The Kohonen maps (self organizing maps) incorporates in an unsupervised way the topology present in the data. The unsupervised problem means that one deals with a set of experimental data which have no specific associated answers (or supplemental information) attached.

4. X (input) Y (output) • A supervised modelling technique is able to capture the relationship between the input data (measurements, observations) and output data (properties of interest). • MLR, PLS, for regression problems • LDA for classification problems • These modelling techniques fail if: • Nonlinearity or topologically incoherent relationship and Considerable number of outliers. ANNs and SVMs can tackle such nonlinear relationships in a convincing way. However, the visualization and interpretation of these models is severely hard due to the fact that they are more or less ‘black-box’ techniques.

5. Kohonen Artificial Neural Networks The Kohonen network is probably the closest of all artificial neural networks architectures and learning schemes to the biological neuron network As a rule, the Kohonen type of net is based on a single layer of neurons arranged in a two-dimensional plane having a well defined topology A defined topology means that each neuron has a defined number of neurons as nearest neighbors, second-nearest neighbor, etc.

6. W Input vector Similarity is the basis of selection of the winner neuron. In other words, there is a competition between neurons for winning. (competitive learning) Weight vector Output

7. The Kohonen learning concept tries to map the input so that similar signals excite neurons that are very close together.

8. XS a a c b d b c d e e Top Map Trained KANN Top Map After the training process accomplished, the complete set of the training vectors is once more run through the KANN. In this last run the labeling of the neurons excited by the input vector is made into the table called top map.

9. 3 5 6 2 2 1 1 1 2 1 1 4 2 3 1 1 0 1 1 0 2 1 1 1 0 1 2 0 3 0 XS Input Vector 3 1 2 0 0 2 0 1 0 3 1 0 3 0 0 0 1 0 2 0 H H H H H H H H Top Map L L L L L L L Weight Map The number of weights in each neuron is equal to the dimension m of the input vector. Hence, in each level of weight only data of one specific variable are handled. Trained KANN

10. Applications of Kohonen ANNs • Sampling from large amount of data • Disease Diagnosis • QSAR & QSPR • Analysis of genomics data Making model using corrupt object (missing values) Typical similarity criteria Specific similarity criteria Number of missing values

11. Counter Propagation network (CPN) CPANN has the same structure as Kohonen network with an additional output layer with same layout as input layer.

12. Input Target Based on the location of the winning unit in the input map (i.e., the unit which is most similar or closest to the presented object X), the input map and the output map are updated simultaneously at the same spatial locations. Input Kohonen layer Input Similarity map Winner Output layer If the CPN network is trained, it can be used for prediction. Simply, an unknown input object is presented to the network. The position of the winning unit in the input map then is used to look-up the predicted value of the corresponding unit in the output map.

13. Applications of CP-ANN • Building predictive calibration and classification models Spectrum Molecular Descriptor Process Descriptor Reaction Descriptor Class Membership Medicinal Activity Process Condition Reaction Mechanism

14. Detection of faulty condition in a process

15. in a CPN network the flow of information is directed from the input layer units towards the output layer. For this reason, we prefer to denote the CPN as being a pseudo-supervised strategy. • CPN was not able to model the inverse relationship between the output and the input. • The output properties are not involved in the formation of the directing Kohonen input map. Hence, the CPN model cannot be considered as being a true supervised method. • One can easily look inside the driving input layer and relationship with output.

16. Supervised Kohonen network (SKN) In a SKN network, the input layer and the output layer are ‘glued’ together, thereby forming a combined input-output layer. Because in a SKN information present in the objects X and Yis used explicitly during the update of the units in the map, the topological formation of the concatenated map is driven by X and Y in a truly supervised way.

17. After training, the input and output maps are decoupled. Then, for a new input object its class membership is estimated according to the procedure outlined for the CPN network. The variables of the objects X and Y in the training set must be scaled properly , but it is not trivial how to deal with the relative weight of the number of variables in X and the number of variables in Y during the training of a SKN network. • User must determine beforehand the proper balance between the influence of the input and output objects: in general, correct scaling of the input and output variables is of utmost importance.

18. Input Input Kohonen layer Output layer Target XY-Fused networks Chemo. Int. lab. 83 (2006) 99–113 Input Similarity map Fused Similarity map Winner Output Similarity map By using a ‘fused’ similarity measure based on a weighted combination of the similarities between an object X and all units in the input layer, and the similarities between the corresponding target and the units in the output layer, the common winning unit for both maps is determined. α(t) decreases linearly in epochs

19. Class (1400) Simulated data sets Odd–Even data set The first data set, referred to as Odd–Even, contains 400 rows (input objects) with 8 variables: each integer valued variable was varied randomly between 1 and 100. Data Matrix (8400) if per row the total number of even values was greater than the total number of odd values, the class membership for that row was assigned to be 1, otherwise the class membership was set to −1

20. Input Class CPNN Output map by CPNN for Odd-Even data Input layer Output layer • Scattered pattern of output map • No real relationship between the multivariate topological structure present in the input space and the associated class membership of the objects. • Does not take into account the class information during the formation of the input and output maps.

21. SKNN Kohonen layer Output map by SKNN for Odd-Even data Input Class • Scattered pattern of output map • Imbalance between the number of input (8) and output (1) variables • Does not take into account the class information during the formation of the input and output maps.

22. Input Class Output map by XYFN for Odd-Even data X-Y Fused Network Input Kohonen layer Input Similarity map Fused Similarity map Output Similarity map Output layer • Nice coherent output maps • Indicating a certain ‘hidden’ relationship present between the input and output space.

23. Class 1 Class 2 Class 3 Input layer Output layer 1 0 0 0 1 0 0 0 1 Overlap data set Three normally distributed clouds of data points in three dimensions. the first 150 objects belong to a multivariate normal distribution around the origin (class 3), whereas the other two classes 1 and 2, each consisting of 150 objects as well, are normally distributed around the centroids (5,3,4) and (5.5, 3.5, 4.5). We will compares the quality of the 8×8 input and output maps for the CPN and XYF networks for the Overlap data set. Data Matrix (3450) X Set 1 Set 2 Set 3 Y Z Class 1 Class 2 Class 1

24. Ymap Xmap Class map CPNN Input layer Output layer

25. Ymap Xmap XY-Fused Network for Overlap data set Class map

26. References • Neural Networks For Chemists, An Introduction. (Weinheim/VCH Publishers) • Chem.Int.Lab.sys.83 (2006) 99–113 • Chem.Int.Lab.sys 90 (2008) 84–91 • Chem.Int.Lab.sys. 38 (1997) 1-23 • Current Computer-Aided Drug Design, 2005, 1, 73-78

27. Thanks

28. Input vector Winner 4×4×2 output

29. amax=0.9 amin=0.1 t=1 (first epoch) Neighbor function: Linear  winner d Input vector × 0.4×0.9 1×0.9× 0.8×0.9× 0.6×0.9×