CS-485: Capstone in Computer Science

1. CS-485: Capstone in Computer Science Artificial Neural Networks and their application in Intelligent Image Processing Spring 2010

2. Organizational Details • Class Meeting:12:25-3:45pm Tuesday, SCIT213 • Class webpage http://www.eagle.tamut.edu/faculty/igor/CS-485.htm • Instructor: Dr. Igor Aizenberg • Office: Science and Technology Building, 104C • Phone (903 334 6654) • e-mail: igor.aizenberg@tamut.edu • Office hours: • Monday, Thursday 10-30 – 6-30 • Tuesday, Wednesday 4-30 – 6-30

3. 1) I. Aizenberg, “Advances in Neural Networks”, University of Dortmund, 2005, Class notes (available from the class webpage) 2) Additional materials will also be available from the class webpage Text Book

4. Introduction to Neural Networks Artificial Intellect: Who is stronger and why? • Applied Problems: • Image, Sound, and Pattern recognition • Decision making • Knowledge discovery • Context-Dependent Analysis • … NEUROINFORMATICS - modern theory about principles and new mathematical models of information processing, which based on the biological prototypes and mechanisms of human brain activities

5. Applied Problems Recognition of Images Natural language understanding (Translation of the texts) Decision Making Learning and Adaptation Reasoning and Prediction Knowledge Discovery Cognitive analysis Team behavior Fuzzy Logic

6. The History of Neuroscience Renaissance of connectionism from the papers by Hopfield, and popularizing the back-propagation algorithm for multiplayer feed-forward networks 1982 End of Perceptron era: Work “Perceptron” by Minsky and Papert 1969 Frank Rosenblatt invented the modern “perceptron” style of NN, composed of trainable threshold units 1957 Ashby puts the idea that intelligence could be created by the use of “homeostatic” devices which learn through a kind of exhaustive search 1952 Minsky’s builts the first actual neural network learning system 1951 1949 Hebb hypothesis that human and animal long-term memory is mediated by permanent alterations in the synapses. Notion of Wiener about key role of connectionism and feedback loops as a model for learning in neural networks 1948 1943 McCulloch and Pitts introduced the fundamental ideas of analyzing neural activity via thresholds and weighted sums

7. ANN as a Brain-Like Computer NN as an model of brain-like Computer • An artificial neural network (ANN) is a massively parallel distributed processor that has a natural propensity for storing experimental knowledge and making it available for use. It means that: • Knowledge is acquired by the network through a learning (training) process; • The strength of the interconnections between neurons is implemented by means of the synaptic weights used to store the knowledge. • The learning process is a procedure of the adapting the weights with a learning algorithm in order to capture the knowledge. On more mathematically, the aim of the learning process is to map a given relation between inputs and output (outputs) of the network. • Brain • The human brain is still not well understood and indeed its behavior is very complex! • There are about 10 billion neurons in the human cortex and 60 trillion synapses of connections • The brain is a highly complex, nonlinear and parallel computer (information-processing system)

8. Rules & Knowledge Productions Engineering Experiment Characteristics & Estimations Signals & parameters Adaptive Machine Learning via Neural Network Interpretation and Decision Making Data Analysis Data Acquisition Data Analysis Data Acquisition Decision Making Knowledge Base Intelligent Data Analysis in Engineering Experiment

9. yi xi m1 m2 mp m3 Mathematical Interpretation of Classification in Decision Making 1. Quantization of pattern space into p decision classes 2. Mathematical model of quantization: “Learning by Examples” Input Patterns Response:

10. The learning involves change of structure Responce Neuroprocessor Learning Rule Teacher Input Images Learning via Self-Organization Principle Self-organization – basic principle of learning: Structure reconstruction

11. Machine Vision Artificial Intellect with Neural Networks Applications of Artificial Neural Networks IntelligentControl Advance Robotics TechnicalDiagnistics Intelligent Data Analysis and Signal Processing Image & Pattern Recognition Intelligent Expert Systems Intelligentl Medicine Devices Intelligent Security Systems

12. Theory What we will learn and do? • Artificial Neural Networks • And Its Applications • You will learn: • Contemporary theoretical principles and paradigms of Neuroinformatics, • Mathematical models and algorithms of neural network techniques for experimentation, • Applications of Neuroinformatics to engineering and sciences problems, • Computer-Aided Technology for Instrumentation Self-Paced Work Practice

13. What we will learn and do? • General principles of artificial neural networks • General principles of learning algorithms • Feedforward neural network and backpropagation learning • Multi-valued neurons and a feedforward neural network based on multi-valued neurons • Basic ideas of image processing • Edge detection on noisy images using a neural network

14. Which way of imagination is best for you ? Pattern recognition Symbol Manipulation or Pattern Recognition ? • Ill-Formalizable Tasks: • Sound and Pattern recognition • Decision making • Knowledge discovery • Context-Dependent Analysis What is difference between human brain and traditional computer via specific approaches to solution of ill-formalizing tasks (those tasks that can not be formalized directly)? Symbol manipulation • Dove flies • Lion goes • Tortoise scrawls • Donkey sits • Shark swims

15. Connectionism Brain computer is a highly interconnected neurons system in such a way that the state of one neuron affects the potential of the large number of other neurons which are connected according to weights or strength. The key idea of such principle isthe functional capacity of biological neural nets determs mostly not so of a single neuron but of its connections Associative distributed memory Storage of information in a brain is supposed to be concentrated in synaptic connections of brain neural network, or more precisely, in the pattern of these connections and strengths (weights) of the synaptic connections. Massive parallelism Brain computer as an information or signal processing system, is composed of a large number of a simple processing elements, called neurons. These neurons are interconnected by numerous direct links, which are called connection, and cooperate which other to perform a parallel distributed processing (PDP) in order to soft a desired computation tasks. Principles of Brain Processing How our brain manipulates with patterns ? A process of pattern recognition and pattern manipulation is based on:

16. Brain-Like Computer ? Brain-like Computer Artificial Neural Network – Mathematical Paradigms of Brain-Like Computer The new paradigm of computing mathematics consists of the combination of such artificial neurons into some artificial neuron net. Neurons and Neural Net Brain-like computer – is a mathematical model of humane-brain principles of computations. This computer consists of those elements which can be called the biological neuron prototypes, which are interconnected by direct links called connections and which cooperate to perform parallel distributed processing (PDP) in order to solve a desired computational task.

17. Y A Principles of Neurocomputing Connectionizm NN is a highly interconnected structure in such a way that the state of one neuron affects the potential of the large number of another neurons to which it is connected accordiny to weights of connections Not Programming but Training NN is trained rather than programmed to perform the given task since it is difficult to separate the hardware and software in the structure. We program not solution of tasks but ability of learning to solve the tasks Distributed Memory NN presents an distributed memory so that changing-adaptation of synapse can take place everywhere in the structure of the network.

18. A A Principles of Neurocomputing Learning and Adaptation NN are capable to adapt themselves (the synapses connections between units) to special environmental conditions by changing their structure or strengths connections. Non-Linear Functionality Every new states of a neuron is a nonlinear function of the input pattern created by the firing nonlinear activity of the other neurons. Robustness of Assosiativity NN states are characterized by high robustness or insensitivity to noisy and fuzzy of input data owing to use of a highly redundance distributed structure