Didday's Model of Winner-Take-All Networks: A Biological Example of Neural Competition

1. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence • Lecture 7. Didday Model of Winner-Take-All • Reading Assignments: • TMB2: • 4.3, pp. 194-197. Prey Selection - or Winner Takes All • 4.4. A Mathematical Analysis of Neural Competition

2. The Prey-Selector Model of Didday • Consider how the frog's brain might select one of several visually presented prey objects. • The task: to design a distributed network(not a serial scan strategy) that could take a position-tagged "foodness array" and ensure that usually the strongest region of activity would influence the motor control system. • The prey-selector: an early, biological example of what connectionists call a winner-take-all network. • Didday’s model adds a relative foodness layer of cells in topographic correspondence to the retinotopic foodness layer. • The new layer yields the input to the motor circuitry.

3. Winner-take-all Networks • Goal: given an array of inputs, enhance the strongest (or strongest few) and suppress the others No clear strong input yields global suppression Strongest input is enhanced and suppresses other inputs

4. Winner-Take-All Networks • Basic architecture: • - topographically organized • layer of neurons • - each neuron receives excitatory • input (e.g., sensory) • - each neuron also sends inhibition • to neighboring neurons, • possibly via an inhibitory • inter-neuron strong stimulus weak stimulus a b Example of 2-neuron WTA with spiking neurons

5. Didday’s Model = inhibitory inter-neurons retinotopic input = copy of input = receives excitation from foodness layer and inhibition from S-cells

6. Didday's transformation scheme from foodness to relative-foodness • It uses a population of S “sameness“ cells in topographic correspondence with the other layers. • Each S-cell inhibits the activity that cells in its region of the relative-foodness layer receive from the corresponding cells in the foodness layer by an amount that augments with increasing activity outside its particular region. • This ensures that high activity in a region of the foodness layer penetrates only if the surrounding areas do not contain sufficiently high activity to block it.

7. Hysteresis • How does the network respond to a change in the maintained stimulus pattern? • Once in equilibrium, one may increase a non-maximal stimulus s2 so that it becomes larger than the previously largest stimulus s1, yet not switch activity to the corresponding element. • In neural networks with loops - an internal state resists dependence on input: buildup of excitation and inhibition precludes the system's quick response to new stimuli. • For example, if one of two very active regions were to suddenly become more active, then the deadlock should be broken quickly. • In the network so far described, however, the new activity cannot easily break through the excitation of its competitor and the global inhibition: there is hysteresis.

8. Hysteresis in a Ferromagnet Direction of External Magnetic Field • In a ferromagnetic material, the atoms act like magnets which tend to align themselves with their neighbors -- this cooperation is opposed by thermal noise. • A ferromagnet becomes magnetized when the atomic magnets have high probability of pointing in the same direction. Not a single function magnetic field  direction but a history-dependent relation.

9. Didday's Transformation Scheme From Foodness to Relative-Foodness • Didday uses a local mechanism, to combat hysteresis, introducing a Newness cell - "N-cell" for each S-cell to monitor temporal changes in the activity of its region. • For a dramatic increase in the region's activity, it overrides the inhibition on the S-cell and permits this new level of activity to enter the relative foodness layer.

10. Formalizing: An "N-cell" for each S-cell • Introducing an "N-cell" for each S-cell • Formally: neurons Ni have activity ki proportional to dSi/dt. • We then postulate that the • input to the ith element to • our array is no longer si • but si + ki. • If ki is large enough for • long enough, the relative • foodness Ei can gain • ascendancy. Si dSi/dt

11. Formalizing Didday's Model • The rest of this lecture provides a mathematical analysis • from TMB2, Section 4.4, and is based on the paper: • Arbib, M.A., and Amari, S.I., (1977) Competition and Cooperation in Neural Nets, in Systems Neuroscience (J. Metzler, Ed.), New York: Academic Press, pp. 119-165. • To see the mathematics, switch to the Word file: • 7W. Amari-Arbib Mathematics