1 / 62

Machine Learning Algorithm

Machine Learning Algorithm. By: Jzau-Shenlg Lin. Outline. Introduction Artificial Neural Networks (ANN) Fuzzy-, Possibilistic-, and Rough- Systems Cerebellar Model Arithmetic Computer (CMAC) Genetic Algorithm (GA) Artificial Immune System (AIS) Ant Colony System (ACS)

toni
Download Presentation

Machine Learning Algorithm

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Machine Learning Algorithm By: Jzau-Shenlg Lin

  2. Outline • Introduction • Artificial Neural Networks (ANN) • Fuzzy-, Possibilistic-, and Rough- Systems • Cerebellar Model Arithmetic Computer (CMAC) • Genetic Algorithm (GA) • Artificial Immune System (AIS) • Ant Colony System (ACS) • Support Vector Machine (SVM) • Conclusions

  3. Introduction(1/2) • Machine learning is a research strategy, in which computers can modeling or implement the humans’ learning behaviors. • It also reconstructs the intelligent architecture in its intelligent base to reinforce the performance for itself. • H.A. Simon indicated that learning is an adaptive activity for a system to causes the system doing the same or similar task more effectively. • R.S. Michalski thought that learning is the representation to configure or revise the experimental tasks. • The experts who design expert systems presented that learning is extracting intelligence.

  4. Introduction(2/2) • Learning is a very important feature for the intelligent behavior. • The applications for the machine learning include: • Robots • Computer game • Signal processing – Compressing, Recognition, watermarking, … • Network topology – The shortest path, Channel assignment, … • Several optimization problems in engineering

  5. Synapse(神經連接線) --依電位變化傳遞 Dendrites(神經樹) --輸入路徑 Axon(神經軸)—輸出路徑 Soma(神經核)—細胞本體 Neurobiological model 1.Artificial Neural Networks (ANN) (1/6) • An ANN is a Massively parallel distributed Processor that has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects: (a) Knowledge is acquired by the network through a learning process. (b) Interneuron connection strengths known as synaptic weights are used to store the knowledge. • Neural Networks are referred to in the literature asneurocomputers, connectionist networks, parallel distributed processors,etc.

  6. Bias input Activation function Output : : Summing junction Threshold Synaptic Weights 1. Artificial Neural Networks (ANN) (2/6) • Models of a Neuron

  7. 1. Artificial Neural Networks (ANN) (3/6) • Type of Activation Function (a) Threshold Function (b) Piecewise-linear Function (c) Sigmoid Function Slope=a

  8. Input layer of source nodes Output layer of neurons Input layer of source nodes Layer of hidden neurons Output layer of neurons 1. Artificial Neural Networks (ANN) (4/6) • Feedforward Architecture

  9. Z-1 Z-1 Z-1 Z-1 1. Artificial Neural Networks (ANN) (5/6) Input layer of source nodes • Recurrent and 2-D Lattice Networks

  10. Neural Networks Optimal Nets ●Hopfield-Tank net ● Annealing net ● Bolzmann machine Fixed Nets ● Hamming net ● Hopfield net ● Bi-direction associative memory Unsupervised Nets ●Self- organization map ● ART ● Neocognitron ● Competitive learning ● Principle component analysis (PCA) ● Independent component analysis (ICA) Supervised Nets ● Perceptron ● Back-Propagation delay ● Probabilistic net ● Multilayer Perceptron ● ADALINE ● LVQ ● Counter propagation net 1. Artificial Neural Networks (ANN) (6/6) • Classifications

  11. 2.1 Fuzzy-Systems • Fuzzy C-Means (FCM) Hard-C-Means (HCM)Fuzzy-C-Means (FCM)

  12. 2.1 Fuzzy-Systems • Penalized FCM and Compensated FCM Penalized FCM (PFCM)Compensated FCM (CFCM)

  13. 2.1 Fuzzy-Systems • The curves of ln(i)and tanh (-i)within 0  i  1

  14. x3 x12 x15 x6 x8 x2 x5 x11 x14 x7 x9 x4 x10 x13 x1 2.1 Fuzzy-Systems • Fuzzification in the training example --Butterfly

  15. 2.2 Possibilistic-System • Possibilistic C-Means (PCM) – Proposed by Krishnapuram and Keller : Scale parameter at the i-th cluster. : Possibilistic typicality value of training sample belonging to the i-th cluster.

  16. 2.3 Fuzzy-, Possibilistic-Systems(1/7) • Fuzzy Possibilistic C-Means (FPCM) -- Proposed by Pal, Pal, and Bezdek

  17. Cluster Cluster Training sample 1 2 ….. c 1 2 ….. c z1 z2 Typicality function zn Membership function 2.3 Fuzzy-, Possibilistic-Systems(2/7) • FPCM-- Membership and Typicality

  18. x12 x8 x2 x1 x3 x5 x7 x9 x11 x6 x4 x10 2.3 Fuzzy-, Possibilistic-Systems(3/7) • Simulated data set

  19. 2.3 Fuzzy-, Possibilistic-Systems(4/7) • Penalized FPCM =JFPCM - scale factors based on clusters and training samples

  20. 2.3 Fuzzy-, Possibilistic-Systems(5/7) • Compensated FPCM =JFPCM + scale factors based on clusters and training samples

  21. Codebook PFPCM/ CFPCM DCT AC coefficients DC coefficient Original Image Encoder DC + Index Transmission DC + Index IDCT DC coefficient Codebook AC coefficients Decoder Reconstructed Image 2.3 Fuzzy-, Possibilistic-Systems(6/7)

  22. Original Image LBG DCT + LBG(VQ) DCT + CFPCM(VQ) 2.3 Fuzzy-, Possibilistic-Systems(7/7)

  23. 2.4 Rough- System (1/5) • Rough set • Let R be a binary equivalence relation defined on a universal set Z is a subset of the Cartesian product, . • An equivalence relation is a binary relation, R, that satisfies • R is reflexive : • R is symmetric : • R is transitive : • can be defined as the union of all equivalence classes in Z/R that are contained in A such that • can be also defined as the union of all equivalence classes in Z/R that overlap with A like the following equation

  24. 2.4 Rough- System (2/5) • Rough set • A rough set can be represented by and with the given set A as • And the rough boundary of A by the equivalence classes Z/R is distinct as • Interconnection models in the architecture of rough neurons (a) Fully connected (b) Exciting model (c) Inhibiting model

  25. 2.4 Rough- System (3/5) • Rough Neurons (Proposed by Lingras in 1998) • Definition for the Exciting model in the rough neurons

  26. 2.4 Rough- System (4/5) • Rough Fuzzy Hopfield Neural Network (RFHNN) Netx,i : Netx,i :

  27. Morphology Processing Result 2.4 Rough- System (5/5) • Processing to the Multi-Spectral Image using RFHNN

  28. 2.5 Fuzzy-, Possibilistic-, Rough- Systems + Artificial Neural networks • Fuzzy Competitive Learning Network (FCLN) • Penalized Fuzzy Competitive Learning Network (PFCLN) • Compensated Fuzzy Competitive Learning Network (CFCLN) • Rough Fuzzy Competitive Learning Network (RFCLN) • Fuzzy Hopfield Neural Network (FHNN) • Penalized Fuzzy Hopfield Neural Network (PFHNN) • Compensated Fuzzy Hopfield Neural Network (CFHNN) • Fuzzy-Possibilistic Hopfield Neural Network (FPHNN) • Rough Fuzzy Hopfield Neural Network (RFHNN)

  29. 3. Cerebellar Model Arithmetic Computer (CMAC) (1/4) • CMAC, named Cerebella Model Articulation Controller, was proposed by J.S. Albus in1975. • CMAC is a model of associate memory network. • In the training phase, the CMAC updates the weights in memory by using a transformation from input samples. • It can easily obtain the outputs by looking up the weights in memory in accordance with the input vectors in the recognition phase. • Due to a simple manner with memory architecture, the CMAC can be easily implemented into hardware circuit.

  30. a w a1 a2 a4 a5 a6 a7 aN w1 w2 w4 w5 w6 w7 wN y Input Vector 3. Cerebellar Model Arithmetic Computer (CMAC) (2/4) • Traditional CMAC Architecture

  31. 1 m94 m2 . . . a0 ~ a15 . . . m1 2 . . . coding 1 Sum 94 3 4 2 Class 1 m94 5 m2 m1 . . . a0 ~ a15 . . . 6 . . . coding 7 Input signals with quantizing binary code 8 Class 2 m94 61 m2 Sum 2 m1 . . . a0 ~ a15 . . . 62 N . . . coding Sum 1 63 64 Input Pattern Class 16 Output Weights Memory 3. Cerebellar Model Arithmetic Computer (CMAC) (3/4) • Modified CMAC Architecture with Clustering Memory

  32. 3. Cerebellar Model Arithmetic Computer (CMAC) (4/4) • Applied CMAC to Character Recognition 8 error pixels in characters 14 error pixels in characters 18 error pixels in characters

  33. 4. Genetic Algorithm (GA)(1/4) • Evolutionary computing was introduced in the 1960s by I. Rechenberg in his work "Evolution strategies" (Evolutions strategie in original). • His idea was then developed by other researchers. Genetic Algorithms (GAs) were invented byJohn Hollandand developed by him and his students and colleagues at University of Michigan, 1970’s . • Directed search algorithms based on the mechanics of biological evolution and Survival with a fitness function. • Functions of GA: • Chromosome • string of DNA • consists of genes • a solution of the problem • Fitness • measure the chromosome • survival or not • Reproduction • crossover • two chromosomes • combine the genes from parents • form new chromosomes • mutation • occurs on single chromosome • elements of DNA are a bit changed

  34. 4. Genetic Algorithm (GA)(2/4) Simple Genetic Algorithm() { Randomly initialize population; evaluate population; while(termination criterion not reached) { select solutions for next population with a fitness function (reproduction); perform crossover and mutation; evaluate population (to produce new offspring); } }    

  35. 4. Genetic Algorithm (GA)(3/4) Population New Offspring mutation crossover Evaluate Roulette-Wheel Selection Reproduction Fitness Function Evolution Circumstance Evolutionary Procedure

  36. 4. Genetic Algorithm (GA)(4/4) • Combing GA with other systems • GA + Fuzzy Algorithm • GA + Possibilistic Algorithm • GA + Rough algorithm • GA + Artificial Neural Network

  37. 5. Artificial Immune System (AIS)(1/9) • The AIS transfers the characteristics of natural immune system with mathematic model into computing system in algorithm manner to solve the engineering problems. • The AIS is based on Jerne’s idiotypic network theory (Jerne, 1973), which suggests that the immune system maintains a network of interconnected B-cells.

  38. 細胞及分泌物 吞噬細胞 淋巴細胞 補體 B細胞及抗體 T細胞及淋巴球 5. Artificial Immune System (AIS)(2/9) • Natural immune system in human body 自然殺手細胞

  39. 5. Artificial Immune System (AIS)(3/9) • The structure of multi-protection and -defense system in immune system

  40. 5. Artificial Immune System (AIS)(4/9) • Models in AIS 1.Antibody Network 2.Evolutionary Algorithm • Immune Genetic Algorithm (IGA) • Immune Evolutionary Programming (IEP) • Immune Evolutionary Strategy (IES) 3.Colonel Selection Principle (CSP)

  41. 抗原激發 抗體群 Ab population 選擇抗體 Ab Selection 相似成熟度 Affinity Maturation Selection 重選抗體 Ab Re-selection 激發細胞 非激發細胞 繁殖Clone 衰亡 Death 5. Artificial Immune System (AIS)(5/9) • Antibody Network (1)

  42. Start Immune System Input external Ags B-Cell’s surface Character of Ags displayed by B-cell Th cells secrete to start immune reaction Activate lymphoid cells All Ags are removed? N Generate Abs Continuously? N Y Stop immune reaction Y Activate Ts to secrete IL¯ Inhibit immune reaction Generate Abs 5. Artificial Immune System (AIS)(6/9) • Antibody Network (2)

  43. 抗原與目的函數、制約條件的對應 記憶親和度(R1) 高的抗體 促進和抑制 生成起始抗體群 生成新抗體 抗體與抗原親和度(R1)計算 抗體與記憶親和度(R2) 的計算 排除適量親和度(R2) 高的抗體 親和度(R1)=1? N Y 抗原排除 (結束) 5. Artificial Immune System (AIS)(7/9) • Immune Genetic Algorithm with Constraint (IGAC)

  44. 啟動 族群更新及適應因子計算 最佳族群? Y 疫苗注射 停止 N 免疫選擇 選擇、交配、及突變 5. Artificial Immune System (AIS)(8/9) • Immune Genetic Algorithm with Vaccination (IGAV)

  45. 5. Artificial Immune System (AIS)(9/9) • Antibody Network (2) + Fuzzy Algorithm to Image segmentation

  46. 6. Ant Colony System (ACS)(1/4) • Ant system algorithm, based on behavior of real ants, is a natural approach to establish from their nest to food source. • An ant moves randomly and detects a previously laid pheromone on a path in order to find the shortest way between their nest and the food source. • Ant system algorithm is an important methodology to apply on non-linear optimal problems recently. • It is a parallel architecture to force ants move simultaneously, independently, and without supervisor.

  47. 6. Ant Colony System (ACS)(2/4) • Each ant chooses the next position to visit in accordance with the visibility of the position and the pheromone intensity. • The k-th ant starting from position i decides to visit position j with the probability defined as follows: where is the visibility of position j from position i, and are two heuristically defined parameters.

  48. 6. Ant Colony System (ACS)(3/4) • We define the pheromone intensity on path (i, j) at time t to be and to assign a random value to it when t = 0. • Along the path from i to j, a trail substance is laid on path (i, j) and defined as: where Q is a constant and is the tour length of the k-th ant.

  49. 6. Ant Colony System (ACS)(4/4) • When the ant has completed a position and a cycle of n iterations is consisted, the laid trail substance is used to update the amount of substance previously laid as the following equation: and where is a coefficient of persistence of the tail and is the quantity of trail substance laid on path (i, j) by the k-th ant during a cycle( between time t and t + n).

  50. 6.1 Annealing algorithm + ACS (AACS) • In the scheme of ant system algorithm, the total cost function for the network topology from node i to k and cooling schedule can be defined as where • The probability that the k-th ant starting from node i to visit node j undergo random thermal perturbations at a given temperature T conforms to a Boltzmann distribution and

More Related