Language

1. Cognitive Neuroscience and Embodied Intelligence Language Based on book Cognition, Brain and Consciousness ed. Bernard J. Baars courses taught byProf. Randall O'Reilly, University of Colorado, and Prof. Włodzisław Duch, Uniwersytet Mikołaja Kopernika and http://wikipedia.org/ http://grey.colorado.edu/CompCogNeuro/index.php/CECN_CU_Boulder_OReilly http://grey.colorado.edu/CompCogNeuro/index.php/Main_Page Janusz A. Starzyk

2. Models account for: distributed lexicon, orthography, phonology, semantics. The same learning mechanisms in the brain, but different inputs/outputs. Levels of processing: phonemes/syllables, letters, words, ideas, phrases, sentences, situations, stories. Distributed representations, great possibilities of combining many representations Modeling speech Semantic representations of word co-occurrence. Semantic representations on the level of sentence shapes. Phonological neighborhood density of words = the number of words that sound similar to a given word, so creating similar activations in the brain. Semantic neighborhood density of words = the number of words with a similar meaning (widened activation subnetwork).

3. Learning based on processing temporal sequences Word sequences must produce meaning representations Language is the result of unpacking distributed meaning representations in the brain and communicating them to other people through communication channels, with the expectation that their corresponding representations will be created in the brain of the receiver Modeling speech Learning to read dyslexia Sign recognition, mapping orthography onto phonology (not trivial for English) and intonations (important in Chinese) Regularities and exceptions creating too-regular past tenses.

4. Biological foundations Controlling the vocal apparatus is responsible for the correct pronunciation of syllables. Mainly responsible for this control is Broca's area in the frontal cortex; for speech analysis, Wernicke's area in the superior temporal lobe. Broca's: surface representation, Wernicke's: deep representation.

5. Neuron uczy się prawdopodob. warunkowego, korelacji pomiędzy pożądaną aktywnością a sygnałami wejściowymi; optymalna wartość 0.7 osiągana jest szybko tylko przy małej stałej uczenia 0.005 Organization of phonemes Phonemes: consonants and vowels, IPA alphabet. 4 dimensions characterizing vowels, tongue positions. Coding: 7 positions for front/back, 6 for up/down, 4 for the rest (shape of the lips and length). Consonants: 3 dimensions. Coding: 7 positions for location (loc), 5 for manner (Mnr), 2 for voicing (Vce).

6. We will try, with the help of computer simulations, to find and verify with the help of models, the answers to several questions: What processes are involved in the reading process and why do they sometimes let us down (dyslexia)? How do we read known words: cat, yacht, and how do we read invented words, eg. nust, deciding on some pronunciation? Why do children say "I goed” instead of "I went”? Where does the meaning of words come from? How to go from words to sentences? Questions

7. Phonological level of dyslexia: nonexistent words don't activate deeper areas (Wernicke's). Deep level: phonological and semantic errors (cat – cot, cat - dog), mistakes in sign recognition. Distributed lexicon and dyslexia Surface dyslexia: new words don't create a problem, but there is a lack of access to the semantic level + difficulties in reading exception words + mistakes in recognition. A model of reading and dyslexia has two paths from orthography to phonology: direct (by mapping) and indirect, via semantics. Uncommon and difficult words are pronounced through the indirect pathway.

8. 40 words, 20 concrete and 20 abstract; activations in the model show phonological-semantic similarities. Words to read

9. TestLog shows the word, distance, most similar, error sm_nm Model Project dyslex.proj.gz The network was trained because it requires 250 epochs. Training: random selection of one of the 3 layers (orthography, phonology, semantics) as input and the other 2 layers as outputs, mapping one aspect onto the two others. kWTA = 25% for hidden layers View TestLog, StepTest

10. Dyslexia: depending on the degree of damage and the pathway damaged, we get different forms of dyslexia: phonological, deep and surface. lesion_path = Semantics Turns off the whole layer. Errors: Trial2_TestLog need, loan, flow, past => coat Hire and coat are the most frequently mistaken. Phonological distance from 0 to 1 measured by cos(S1,S2) = S1*S2/|S1||S2| Simulating dyslexia

11. Effect of lesions on dyslexia Errors of hidden neurons OS_Hid orthographic-semantic SP_Hid semantic-phonetic Error types for semantic pathway lesions.

12. Complete lesions Errors resulting from semantic and direct pathway lesions.

13. Direct pathway lesions Errors resulting from direct pathway lesions, with both no lesions in the semantic pathway (Full Semanitcs) and with a complete semantic pathway lesion (No Semantics).

14. Reading models: mapping orthography onto phonology. Two issues: can one system learn to pronounce regular words and simultaneously deal with exception words? simulating pronunciation of nonexistent words requires the discovery of subtle regularities of pronuncation. Mint, hint, flint => "i" is the same, but in pint it's different... Regularities are often modified, depending on the context, they have groupings (neighborhoods), and exceptions are on the extremes of these modifications. Regularities and exceptions form a continuum. Reading

15. Representations are not localized in one region. Interactions lead to an interesting division of labor. Reading: distributed lexical model

16. Exchange between dependent and independent is similar in object recognition We need constants but we also need feature connections Reading as object recognition

17. Model ss.proj.gz Chapt. 10.4.2 7 blocks of 3*9 = 189 inputs, 5*84 = 420 in orthography, and 600 hidden, 7 blocks of Reading model 2*10 = 140 phonological elements. Input: words up to 7 letters, completion, eg. best = bbbestt

18. Tests of regularity: Głuszko Pseudo-homophony phyce => Choyce Network relaxation times as a function of frequency and consistency of words. Human reading speeds shows similar dependencies. Regularities

19. Union of semantics and phonology Project pt.proj.gz Regularization tendencies change with maturation and learning new words. Initial training first on irregular words and then on regular ones, this is controversial but gives a U-shaped curve. Models gradually changing the learning environment but without Hebbian learning don't work. Past tenses

20. Network: semantic input, hidden layer + phonological. Data: 389 verbs, including 90 irregular in the past tense, 4 possible regular endings: -ed, -en, -s, -ing, total 1945 examples. Leabra model Cooperation + competition + Hebbian learning gives a network, in which mapping regular and irregular verbs reaches a dynamic equilibrium. Priming after several exposures changes the behavior of the network.

21. Cooperation + competition + Hebbian learning gives a network, in which mapping regular and irregular verbs reaches a dynamic equilibrium. Priming after several exposures changes the behavior of the network. Leabra model

22. Idea semantics is the result of activations distributed across many areas. Simplest model: Strong Hebbian correlations between words, like correlations between elements of images or phonemes creating syllables. LSA- Latent Semantic Analysis, type of PCA, which can be realized by Hebbian learning. Word meaning

23. Words in the brain Action networks – observations, findings of ERP and fMRI tests. Psycholinguistic experiments about speech show that in the brain we have discrete phonological representations, and not acoustic ones. Acoustic signal => phonemes => words => semantic concepts. Semantic activations follow 90 ms after phonological activations (N200 ERPs). F. Pulvermuller (2003) The Neuroscience of Language. On Brain Circuits of Words and Serial Order. Cambridge University Press.

24. Words: simple model A model inspired by overlapping brain processes which happen during invention of new words. Given is a set of key words, which activate the auditory cortex. Phonemes are resonances, orderly activation of phonemes activates known words and new combinations equally; context + inhibition in the winner-takes-all processleaves one word. • Creativity = imagination (fluctuations) + filtering (competition) • Imagination: many temporary resonances arise in parallel, activating representations of words and non-words, depending on the connection strength of oscillators. • Filtering: associations, emotions, phonological/semantic density. Goals: making the simplest model of creative thinking; creating interesting new names, conveying product features; understanding new words, which aren't in the dictionary.

25. Associations - revision For difficult associations, adding noise in weakly creative people won't help because of a lack of connections, priming words cause only chaos. For orthographically similar priming words with close associations, this activates the representation of the second word, always increasing the chance of resonance and shortening latency. Why does priming neutral for simple associations and nonsensical words worsen results for creative people? Weak creativity = weak associations (connections) between oscillators;adding noise (nonsensical words) strengthens already overlapping oscillations, enabling mutual activations; for a strongly connected neural network and simple associations, it leads to confusion, when it activates many states.

26. Project sem.proj.gz, description 10.6.2 An already trained network responds to questions... Quiz

27. Traditional approach: grammatical breakdown of sentences. Alternative approach: distributed representations, gestalt of the sentence. Like in recognition of 3D objects, there is no central representation. Sentence meaning Small world: sentences containing people's names, active and passive actions, objects and places. People: busdriver, teacher, schoolgirl,pitcher. Actions: eat, drink, stir, spread, kiss, give, hit, throw, drive, rise. Objects: Spot (the dog), steak, soup, ice cream, crackers, jelly, iced tea, kool aid, spoon, knife, anger, rose, bat (animal), bat (baseball), ball, ball (party), bus, pitcher, fur Places: kitchen, living room, shed, park.

28. Project sg.proj.gz, description 10.7.2 The input presents words, localized representations, in the Encode layer are created distributed representations, integrated in time in the Gestalt and Gestalt_Context layers, questions are connected with roles (agent, patient, instrument ...), the network decodes representations, completing them (Filler) Network and project Eg. bat (animal) and bat (baseball) requires differentiation.

29. Role assignment, word ambiguity, concept instantiation, role elaboration, conflict resolution. Tests Small world: sentences containing people's names, active and passive actions, objects and places.

30. Unambiguous verbs, after training of the network, have these cosine similarities of internal activations. Verb similarity

31. Unambiguous nouns, after training of the network, have these cosine similarities of internal activations. Noun similarity

32. Similarities of sample sentences: sc_ = schoolgirl bu_ = busdriver te_ = teacher pi_ = pitcher _at = ate _dr = drank _st = stirred _so = soup _st = steak _ic = ice tea _ko = Kool-Aid Shapes of sentences

33. What processes are involved in the reading process and why do they sometimes betray us (dyslexia)? Distributed lexical representations, interactions between sign recognition, level of spelling (orthography), phonology and semantics. How do we read known words: cat, yacht, and how do we read invented words, eg. nust? Thanks to contextually activated representations, giving a continuum between regular forms and exceptions. Why do children say "I goed” instead of "I went”? Because of the dynamic equilibrium between mapping regular forms and exceptions. Where does the meaning of words come from? Statistics of co-occurrence, interactions with representations of sensory data. How to go from words to sentences? This is enabled by the "Sentence Gestalt" (a theory in psychology). Questions/answers concerning language