Nature requires Nurture

1. Nature requires Nurture • Initial wiring is genetically controlled • Sperry Experiment • But environmental input critical in early development • Occular dominance columns • Hubel and Wiesel experiment

3. Critical Periods in Development • There are critical periods in development (pre and post-natal) where stimulation is essential for fine tuning of brain connections. • Other examples of columns • Orientation columns

4. Pre-Natal Tuning: Internally generated tuning signals • But in the womb, what provides the feedback to establish which neural circuits are the right ones to strengthen? • Not a problem for motor circuits - the infant moves its limbs to refine the feedback and control networks. • But there is no vision in the womb. • --Systematic moving patterns of activity are spontaneously generated pre-natally in the retina. • A predictable pattern, changing over time, provides excellent training data for tuning the connections between visual maps. • The pre-natal development of the auditory system • Research indicates that infants, immediately after birth, preferentially recognize the sounds of their native language over others. The assumption is that similar activity-dependent tuning mechanisms work with speech signals perceived in the womb.

5. Post-natal environmental tuning • The pre-natal tuning of neural connections using simulated activity can work quite well – • a newborn colt or calf is essentially functional at birth. • This is necessary because the herd is always on the move. • For many animals, including people, experience is absolutely necessary for normal development (as in the kitten experiment). • For a similar reason, if a human child has one weak eye, the doctor will sometimes place a patch over the stronger one, forcing the weaker eye to gain experience.

6. Adult Plasticity and Regeneration The brain has an amazing ability to reorganize itself through new pathways and connections rapidly. • Through Practice: • London cab drivers, motor regions for the skilled • After damage or injury • Undamaged neurons make new connections and take over functionality or establish new functions • But requires stimulation • Stimulation standard technique for stroke victim rehabilitation

7. When nerve stimulation changes, as with amputation, the brain reorganizes. In one theory, signals from a finger and thumb of an uninjured person travel independantly to separate regions in the brain's thalamus (left). After amputation, however, neurons that formerly responded to signals from the finger respond to signals from the thumb (right).

8. Summary • Both genetic factors and activity dependent factors play a role in developing the brain architecture and circuitry. • There are critical developmental periods where nurture is essential, but there is also a great ability for the adult brain to regenerate. • Next: What computational models satisfy some of the biological constraints. • Question: What is the relevance of development and learning in language and thought?

9. Connectionist Models: Basics Srini Narayanan CS182/CogSci110/Ling109 Spring 2008

11. Neural networks abstract from the details of real neurons • Conductivity delays are neglected • An output signal is either discrete (e.g., 0 or 1) or it is a real-valued number (e.g., between 0 and 1) • Net input is calculated as the weighted spatial sum of the input signals • Net input is transformed into an output signal via a simple function (e.g., a threshold function)

12. yj wij yi xi f ti : target xi = ∑j wij yj yi = f(xi – qi) The McCullough-Pitts Neuron yj: output from unit j Wij: weight on connection from j to i xi: weighted sum of input to unit i Threshold

13. Neural nets: Mapping from neuron

14. Simple Threshold Linear Unit

15. Simple Neuron Model 1

16. Simple Neuron Model 1 1 1 1

17. Simple Neuron Model 1 1 1 1 1

18. Simple Neuron Model 0 1 1 1

19. Simple Neuron Model 0 1 0 1 1

20. o u t p u t y { 1 if net > 0 0 otherwise w0 I0 = 1 w1 w2 wn . . . i1 i2 in i n p u t i Abstract Neuron

21. Computing with Abstract Neurons • McCollough-Pitts Neurons were initially used to model • pattern classification • size = small AND shape = round AND color = green AND location = on_tree => unripe • linking classified patterns to behavior • size = large OR motion = approaching => move_away • size = small AND direction = above => move_above • McCollough-Pitts Neurons can compute logical functions. • AND, NOT, OR

22. i1 w01 w02 i2 y0 b=1 w0b x0 f Computing logical functions: the OR function • Assume a binary threshold activation function. • What should you set w01, w02 and w0b to be so that you can get the right answers for y0?

23. i2 i1 Many answers would work y = f (w01i1 + w02i2 + w0bb) recall the threshold function the separation happens when w01i1 + w02i2 + w0bb = 0 move things around and you get i2 = - (w01/w02)i1 - (w0bb/w02)

24. Decision Hyperplane • The two classes are therefore separated by the `decision' line which is defined by putting the activation equal to the threshold. • It turns out that it is possible to generalise this result to TLUs with n inputs. • In 3-D the two classes are separated by a decision-plane. • In n-D this becomes a decision-hyperplane.

25. Linearly separable patterns PERCEPTRON is an architecture which can solve this type of decision boundary problem. An "on" response in the output node represents one class, and an "off" response represents the other. Linearly Separable Patterns

26. The XOR function

27. The Input Pattern Space

28. The Decision planes

29. Multiple Layers y 0.5 1 -1 0.5 1.5 1 1 1 1 I1 I2

30. Multiple Layers y 0.5 1 -1 0.5 1.5 1 1 1 1 I1 I2 0 1

31. Multiple Layers y 0.5 1 -1 0.5 1.5 1 1 1 1 I1 I2 1 1

32. Types of abstract neuron parameters • The form of the combination function - e.g. linear, sigma-pi, cubic. • The activation-output relation - linear, hard-limiter, or sigmoidal. • The nature of the signals used to communicate between nodes - analogue or boolean. • The dynamics of the node - deterministic or stochastic. • Spatio temporal information encoding: • Pulse coding and Spiking Neurons

33. Types of Activation functions

34. The Sigmoid Function y=a x=neti

35. The Sigmoid Function Output=1 y=a Output=0 x=neti

36. The Sigmoid Function Output=1 Sensitivity to input y=a Output=0 x=neti

37. Changing the exponent k(neti) K >1 K < 1

38. Radial Basis Function

39. Stochastic units • Replace the binary threshold units by binary stochastic units that make biased random decisions. • The “temperature” controls the amount of noise temperature

40. Spiking Neurons and Pulse coding • Rate coding (ex. Sigmoid units) • Spatial summation of input • Output is the average number of spikes in some time window (normalized between 0 and 1). • Pulse coding (More realistic) • Look at each individual spike (the time it is generated) • Can take into account refractory period • EXAMPLE: Integrate and fire neurons • EXAMPLE: Time to first spike (Thorpe 1996). • Adds power to the basic neuron by adding temporal information

41. Triangle Nodes: Encoding relational information with abstract neurons • The triangle node (aka 2/3 node) is a useful function that activates its outputs (3) if any (2) of its 3 inputs are active • Such a node will be useful for lots of representations.

42. Triangle nodes and McCullough-Pitts Neurons? Relation Object Value A B C

43. Networks of Triangle nodes: example sentence “They all rose” triangle nodes: when two of the abstract neurons fire, the third also fires model of spreading activation

44. Basic Ideas behind connectionist models • Parallel activation streams. • Top down and bottom up activation combine to determine the best matching structure. • Triangle nodes bind features of objects to values • Mutual inhibition and competition between structures • Mental connections are active neural connections

45. 5levels of Neural Theory of Language Pyscholinguistic experiments Spatial Relation Motor Control Metaphor Grammar Cognition and Language Computation Structured Connectionism abstraction Neural Net SHRUTI Computational Neurobiology Triangle Nodes Biology Neural Development Quiz Midterm Finals