Biological and cognitive plausibility in connectionist networks for language mode l ling

1. Biological and cognitive plausibility in connectionist networks for language modelling Maja Anđel Department for German Studies University of Zagreb

2. Connectionist networks • AI networks for simulating cognitive processes – language • implementing the “computer metaphor” • architectural inspiration: human brain (network of artificial neurons)

3. Connectionist networks • subsymbolic, not symbolic processing = no division in hardware/software – software “built in” to the hardware structure

4. Connectionist networks

5. Connectionist networks

6. Language modeling in connectionism

7. Models in history (’80) • Rumelhart & McClelland (1986): English past tense – modeling the U-shaped learning curve

8. Models in history (’80) • McClelland & Kawamoto (1986): Thematic roles in sentences • “How to represent sequences?” w i c k e l f e a t u r e s w i c i c k c k e k e l e l f l f e ...

9. outputs hidden units inputs Models in history (’80) • Elman (1990): Words of different length • “How to represent sequences?” context units

10. Today – biological plausibility • O’Reilly (1998): Six principles of biological plausibility: • biological realism • distributed representations • inhibitory competition • bidirectional propagation of activation (top-down and bottom-up) • error-driven task learning • Hebbian learning New algorithms that satisfy all the constraints!

11. Today – biological plausibility • O’Reilly (1998): Six principles of biological plausibility: • biological realism

12. Today – biological plausibility • O’Reilly (1998): Six principles of biological plausibility: • distributed representations

13. Today – biological plausibility • O’Reilly (1998): Six principles of biological plausibility: • distributed representations

14. Today – biological plausibility • O’Reilly (1998): Six principles of biological plausibility: • inhibitory competition

15. Today – biological plausibility • O’Reilly (1998): Six principles of biological plausibility: • bidirectional propagation of activation (top-down and bottom-up) conceptual perceptual d o g

16. Today – biological plausibility • O’Reilly (1998): Six principles of biological plausibility: • error-driven task learning desired outcome error computation transfer function error backpropagation input signal

17. Today – biological plausibility • O’Reilly (1998): Six principles of biological plausibility: • error-driven task learning network state transfer function input signal desired outcome error computation

18. Today – biological plausibility • O’Reilly (1998): Six principles of biological plausibility: • Hebbian learning "The general idea is an old one, that any two cells or systems of cells that are repeatedly active at the same time will tend to become 'associated', so that activity in one facilitates activity in the other." (Hebb 1949, p. 70) "When one cell repeatedly assists in firing another, the axon of the first cell develops synaptic knobs (or enlarges them if they already exist) in contact with the soma of the second cell." (Hebb 1949, p. 63)

19. Today – biological plausibility • New algorithms – combination of new principles • Complexity brings better results • Old models sucessfully transposed • morphology acquisition • syntax processing • semantic categorization • Computational neuroscience

20. Thank you!