Reading, Language, Brain: The role of computational models

1. Reading, Language, Brain:The role of computational models Mark S. Seidenberg University of Wisconsin-Madison

2. I am very happy to be here. And to visit the National Key Laboratory of Cognitive Neuroscience and Learning The modern study of reading in Chinese began with the pioneering work of ShuHua. In the West, this is the most famous laboratory for scientific studies of reading and language in China. And China is a very big country.

3. My Own Work Connectionist models that simulate detailed aspects of acquisition, skilled performance. Dyslexia = anomalies in how system develops Children, adults Normal, dyslexic English, Serbian, Chinese, others Mainly at Medical College of Wisconsin. Jeff Binder, Lab director. Brain circuits Computational models Behavior

4. Computational models Seidenberg & McClelland, 1989 (about the time neuroimaging came into psychology)

5. Computational models Harm and Seidenberg, 1999: phonology, dyslexia Zevin and Seidenberg, 2006: consistency effects, nonwords, individual differences

6. Computational models Joanisse & Seidenberg, 2000: verb morphology. The “past tense debate”

7. Computational Models Harm & Seidenberg, 2004: computing MEANING

8. Many other models Many people, labs

9. Articulatory speech Insular-Opercularis (Output) Inferior frontal/insular cortex Opercularis-Triangularis Supramarginal gyrus Inferolateral anterior temporal region (Semantics) Anterior STG/STS Inferior SMG buffer Temporal lobe Middle STG/STS Primary auditory area/ and surroundings (Input) Activation of meaning (from other modalities) and input to spontaneous speech production Sound + auditory word input

10. Ueno, Lambon Ralph, Rogers, 2012 Output sound Hidden layer Semantics (Input & Output) Hidden layer Hidden layer Elman Hidden layer Input Sound

11. The reading models instantiated several theoretical principles. General, not linked to details of any single model Are they correct? Are they still relevant? Needed? What good are these models???

12. context meaning spelling sound Everytheory/model must have these basic elements: What reading IS. But, how are these components, processes realized? What is in the ovals? What does an arrow represent? How is the system learned? How is it represented in the brain? What is different about “dual-route” models (e.g., DRC, CDP+) 1. only about pronunciation aloud: no semantics 2. a second mechanism for pronouncing some words: adds word frequency 3. “functional architecture”: independent of brain 4. no learning

13. Let’s look at some of the basic principles. And what has happened to them.

14. 1. Distributed representations Current status: • fMRI evidence for word-specific representations • univariate methods biased against finding highly distributed representations? Much research using other methods (MVPA and related tools) Chris Cox (Wisconsin grad student), Tim Rogers, M. Seidenberg How are categories represented? Artifacts, animals, etc.

15. 2. Variable mappings between codes Arbitrary, correlated, to what degree? Quasiregularity Depends on writing system More about this later! Current status: many investigations across different writing systems Dual-route models: DRC: GPC rules. Very different. CDP+: rules are gone. Replaced by connectionist network.

16. 3. Statistical learning Learning based on frequencies, distributions of events gradual, structures emerge over time Current status: 1. Using models to look at reading development under atypical conditions perceptual, learning, experiential deficits dialect differences in US teaching methods

17. 3. Statistical learning Learning based on frequencies, distributions of events gradual, structures emerge over time Current status: 1. Using models to look at reading development under atypical conditions perceptual, learning, experiential deficits dialect differences in US teaching methods 2. Statistical learning in language acquisition

18. Where are the words? Related mechanisms in reading Chinese? 刘川生书记在致辞中代表学校向莅临大会的领导和专家表示热烈的欢迎，对北京市委、市政府长期以来关心和支持北师大建设表示衷心的感谢。刘川生书记指出，文化是民族的精神家园，推进文化创新发展是时代赋予我们的神圣使命。大学是文化传承创新的重要阵地，在中华文化创新和传播的伟

19. 4. Processing by satisfying multiple constraints. A general way of solving complex problems. Part of what makes humans intelligent! Key idea: nonlinear combination of clues in isolation, not very informative together, very informative What is it? A living thing

20. 4. Processing by satisfying multiple constraints. A general way of solving complex problems. Part of what makes humans intelligent! Key idea: nonlinear combination of clues in isolation, not very informative together, very informative What is it? A living thing university professor

21. 4. Processing by satisfying multiple constraints. A general way of solving complex problems. Part of what makes humans intelligent! Key idea: nonlinear combination of clues in isolation, not very informative together, very informative What is it? A living thing university professor Visiting NKLCNL

22. 4. Processing by satisfying multiple constraints. A general way of solving complex problems. Part of what makes humans intelligent! Key idea: nonlinear combination of clues in isolation, not very informative together, very informative What is it? A living thing university professor Visiting NKLCNL American ME!

23. semantics orthography phonology In reading aloud Activation from both parts

24. Recent research: is semantics used in reading words aloud (in English)? Dual-route models: No Some studies show such effects (Strain, Patterson, Seidenberg, 1996) but not all.

25. Individual differences among readers? Will Graves, Medical College of Wisconsin  Rutgers University Imageability effects on reading words aloud Skilled adult readers Highly educated

26. DTI volume of sem-phon pathways AG – pSTG ITG – pMTG Correlate with imageability effect For skilled readers of English

27. So: skilled readers (in English) differ. Related to anatomical differences in relevant parts of reading circuit. Need to look at other individual differences.

28. semantics orthography phonology Combining constraints in reading for meaning:

29. “radical” Reading Chinese

30. But contributions vary with type of character Transparency of radical? Consistency of phonetic? Characters with other structures N-N compounds like TELEVISION 电视 more semantics, less phonology?

31. “Division of labor” depends on writing system, word, task, reader

32. About morphology Much debate about what is “morphological” in Chinese. Much debate about what is “morphological” in English. Simple view: minimal units of meaning. Discrete. Combined like beads on a string BOAT HOUSE HOUSE BOAT

33. But, many partial regularities BAKER: person who bakes BAKE + ER TALKER: person who talks TALK + ER GROCER: person who sells food *GROCE CORNERa vegetable not CORN + ER DISLIKE DIS + LIKE not LIKE DISAGREE DIS + AGREE not AGREE with DISCOVER DIS + COVER *not COVER SWEETBREADS not sweet, not bread BOOTLEG make illegal alcohol SLAPSTICK a kind of humor

34. Similarity ratings: TEACHER TEACH BACKER BACK CORNER CORN

35. Seidenberg and Gonnerman, TICS, 2000 Morphemes are graded, not discrete Reflect correlations between form and meaning Which vary in degree Many similarities to Chinese

36. 5. Interconnectivity Representations determined by functions in circuits

37. 6. Models perform TASKS Activation is task dependent. Circuits both represent and process information Many studies showing task-dependence of activation in areas like pOTS Yang et al. Mano et al. (in press): in naming vs. visual discrimination tasks others

38. Conclusions The models are a useful tool. They are not literally correct. The principles that govern them are relevant to understanding brain, behavior. More so than the specific architectures that were proposed; too simple! 1. distributed representations 2. variable mappings 3. statistical learning 4. constraint satisfaction 5. Interactivity, feedback 6. task orientation (others)

39. Thank you! Thanks to collaborators Modeling Tim Rogers (Wisconsin) Chris Cox (Wisconsin) Jason Zevin (Weill Cornell Sackler) Michael Harm (Google) Marc Joanisse (Western Ontario) David Plaut (CMU) Jay McClelland (Stanford) Imaging (Medical College of Wisconsin Jeff Binder, lab director Will Graves Rutvik Desai Quintino Mano Chinese language Tianlin Wang (University of Wisconsin)

40. Binder, Graves, Desai, Seidenberg articles