NEURAL NETWORK THEORY NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

1. NEURAL NETWORK THEORY NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS • 欢迎大家提出意见建议！ • 2003.10.15

2. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS NEURONS AS FUNCTIONS Neurons behave as functions. Neurons transduce an unbounded input activation x(t) at time t into a bounded output signal S(x(t)).

3. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS NEURONS AS FUNCTIONS The transduction description: a sigmoidal or S-shaped curve the logistic signal function: The logistic signal function is sigmoidal and strictly increases for positive scaling constant c >0.

4. -∞ - + +∞ 0 NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS NEURONS AS FUNCTIONS S(x) x Fig.1 s(x) is a bounded monotone-nondecreasing function of x If c→+∞，we get threshold signal function (dash line), Which is piecewise differentiable

5. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS SIGNAL MONOTONICITY In general, signal functions are monotone nondecreasing S’>=0. This means signal functions have an upper bound or saturation value. The staircase signal function is a piecewise-differentiable Monotone-nondecreasing signal function.

6. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS SIGNAL MONOTONICITY An important exception: bell-shaped signal function or Gaussian signal functions The sign of the signal-activation derivation s’ is opposite the sign of the activation x. We shall assume signal functions are monotone nondecreasing unless stated otherwise.

7. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS SIGNAL MONOTONICITY Generalized Gaussian signal function define potential or radial basis function : input activation vector: variance: mean vector: we shall consider only scalar-input signal functions:

8. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS SIGNAL MONOTONICITY neurons are nonlinear but not too much so ---- a property as semilinearity Linear signal functions - make computation and analysis comparatively easy - do not suppress noise - linear network are not robust Nonlinear signal functions - increases a network’s computational richness - increases a network’s facilitates noise suppression - risks computational and analytical intractability - favors dynamical instability

9. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS SIGNAL MONOTONICITY Signal and activation velocities the signal velocity: =dS/dt Signal velocities depend explicitly on action velocities

10. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS BIOLOGICAL ACTIVATIONS AND SIGNALS Fig.2 Neuron anatomy 神经元(Neuron)是由细胞核(cell nucleus)，细胞体(soma)，轴突(axon)，树突(dendrites)和突触(synapse)所构成的

11. x1 w1 x2 w2 net=XW o=f（net） ∑ f … xn wn NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS BIOLOGICAL ACTIVATIONS AND SIGNALS X=（x1，x2，…，xn） W=（w1，w2，…，wn） net=∑xiwi net=XW

12. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS BIOLOGICAL ACTIVATIONS AND SIGNALS Competitive Neuronal Signal The neuron “wins” at time t if , “loses” if and otherwise possesses a fuzzy win-loss status between 0 an 1. a. Binary signal functions : [0,1] b. Bipolar signal functions : [-1,1] McCulloch—Pitts (M—P) neurons logical signal function ( Binary  Bipolar )

13. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS NEURON FIELDS Neurons within a field are topologically ordered, often by proximity. zeroth-order topology : lack of topological structure Denotation: , , neural system samples the function m times to generate the associated pairs , ... , The overall neural network behaves as an adaptive filter and sample data changed network parameters.

14. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS NEURONAL DYNAMICAL SYSTEMS Description:a system of first-order differential or difference equations that govern the time evolution of the neuronal activations or membrane potentials Activation differential equations: (1) (2) in vector notation: (3) (4)

15. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS NEURONAL DYNAMICAL SYSTEMS Neuronal State spaces So the state space of the entire neuronal dynamical system is: Augmentation:

16. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS NEURONAL DYNAMICAL SYSTEMS Signal state spaces as hyper-cubes The signal state of field at time t: The signal state space: an n-dimensional hypercube The unit hypercube : or , The relationship between hyper-cubes and the fuzzy set : , subsets of correspond to the vertices of

17. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS NEURONAL DYNAMICAL SYSTEMS Neuronal activations as short-term memory Short-term memory(STM) : activation Long-term memory(LTM) : synapse

18. S k x o NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS SIGNAL FUNCTION (ACTIVATION FUNCTION) 1、Liner Function S(x) = cx + k , c>0

19. S r -θ θ x -r NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS SIGNAL FUNCTION (ACTIVATION FUNCTION) 2. Ramp Function r if x≥θ S(x)= cx if |x|<θ -r if x≤-θ r>0, r is a constant.

20. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS SIGNAL FUNCTION (ACTIVATION FUNCTION) 3、threshold linear signal function: a special Ramp Function Another form:

21. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS SIGNAL FUNCTION (ACTIVATION FUNCTION) 4、logistic signal function: Where c>0. So the logistic signal function is monotone increasing.

22. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS SIGNAL FUNCTION (ACTIVATION FUNCTION) 5、threshold signal function: Where T is an arbitrary real-valued threshold,and k indicates the discrete time step.

23. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS SIGNAL FUNCTION (ACTIVATION FUNCTION) 6、hyperbolic-tangent signal function: Another form:

24. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS SIGNAL FUNCTION (ACTIVATION FUNCTION) 7、threshold exponential signal function: When ,

25. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS SIGNAL FUNCTION (ACTIVATION FUNCTION) 8、exponential-distribution signal function: When ,

26. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS SIGNAL FUNCTION (ACTIVATION FUNCTION) 9、the family of ratio-polynomial signal function: An example For ,

27. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS SIGNAL FUNCTION (ACTIVATION FUNCTION)

28. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS SIGNAL FUNCTION (ACTIVATION FUNCTION)

29. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS PULSE-CODED SIGNAL FUNCTION Definition: (5) (6) where (7)

30. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS PULSE-CODED SIGNAL FUNCTION Pulse-coded signals take values in the unit interval [0,1]. Proof: when when

31. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS PULSE-CODED SIGNAL FUNCTION Velocity-difference property of pulse-coded signals The first-order linear inhomogenous differential equation: (8) The solution to this differential equation: (9) (5) A simple form for the signal velocity: (10) (11)

32. NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS PULSE-CODED SIGNAL FUNCTION (10) The central result of pulse-coded signal functions: The instantaneous signal-velocity equals the current pulse minus the current expected pulse frequency. ------------- the velocity-difference property of pulse-coded signal functions