Presentation Transcript

1. BrainNet: Combining evidence from corpora and from the brain to study conceptual representations

   Massimo PoesioUniEssex, Language & ComputationUni Trento, CIMEC/CLIC
2. COLLABORATORS Trento: ANDREW ANDERSONYUQIAO GUYUAN TAOMARCO BARONIGABRIELE MICELI Essex:HEBA LAKANY (now Strathclyde)FRANCISCO SEPULVEDA BRIAN MURPHY(CMU, was Trento)
3. MOTIVATIONS Research on conceptual knowledge is carried out in Artificial Intelligence, Computational Linguistics, Neural Science, and Psychology But there is limited interchange between AI, CL and the other disciplines studying concepts Except indirectly through the use of WordNet This line of research: use evidence from Neural Science, work on (vector-space) models in CL, and psychology to rethink the design of lexical repositories such as WordNet
4. THE (LEXICAL) SEMANTICS REVOLUTION IN CL AND AI The availability of repositories of lexical knowledge such as ConceptNet, Cyc, FrameNet, and especially WordNet, has had a dramatic impact on research and development in HLT and AI, leading to the development of the first HLT systems able to do (some form of) lexical semantic interpretation on large amounts of data This extensive use however has also highlighted the limitations of such resources (focusing here on WordNet as it’s the best known)
5. LIMITATIONS OF WORDNET Already familiar from the CL literature: Coverage Overly fine-grained distinctions More fundamental problems: Evidence for categorical distinctions Assumptions about taxonomic structure Lack of information about function / perceptual properties Emotional import
6. ENCOUNTERING WORDNET’S LIMITATIONS: A TYPICAL EXAMPLE Between 2003 and 2006 AbdulrahmanAlmuhareb and myself ran a series of studies on ontology learning from text (Poesio & Almuhareb, 2008) We used WordNet to identify the categories of interest and to evaluate the results of our system
7. QUANTITATIVE EVALUATION ATTRIBUTES PROBLEM: can’t compare against WordNet Precision / recall against hand-annotated datasets Human judges (ourselves): We used the classifiers to classify the top 20 features of 21 randomly chosen concepts We separately evaluated the results CATEGORIES: Clustering of the balanced dataset PROBLEM: The WordNet category structure is highly subjective
8. CLUSTERING: ERROR ANALYSIS
9. CLUSTERING: ERROR ANALYSIS
10. CLUSTERING: ERROR ANALYSIS IN WORDNET: PAIN
11. EXAMPLE: STATES AND RELATIONS WORDNET FEELING STATE ATTRIBUTE PLAUSIBLE ALTERNATIVE? STATE ATTRIBUTE FEELING
12. LIMITS OF THIS TYPE OF EVALUATION No way of telling how complete / accurate are our concept descriptions Both in terms of relations and in terms of their relative importance No way of telling whether the category distinctions we get from WordNet are empirically founded
13. EVIDENCE FROM OTHER AREAS OF COGNITIVE SCIENCE Attributes: evidence from psychology Association lists (priming) E.g., use results of association tests to evaluate proximity (Lund et al, 1995; Pado and Lapata, 2008) Comparison against feature norms: Schulte imWalde, 2008) Feature norms Category distinctions: evidence from neural science
14. USING BRAIN DATA TO IDENTIFY CATEGORY DISTINCTIONS Studies of brain-damaged patients have been shown to provide useful insights in the organization of conceptual knowledge in the brain Warrington and Shallice 1984, Caramazza & Shilton 1998 fMRI has been used to identify these distinctions in healthy patients as well E.g., Martin & Chao See, e.g., Capitani et al 2003 for a survey
15. Magnetic Resonance ImagingScanner
16. fMRI Setup SETUP
17. Simple Paradigms Image visualisation Property elicitation Silent naming Concept “simulation”
18. ANIMALS VOXEL TOOLS CATEGORY DISTINCTIONS IN THE BRAIN
19. A MORE COMMON CASE d. RED: Law, BLUE: Music
20. MVPA: USING SUPERVISED LEARNING TO CLASSIFY ACTIVATION PATTERNS Simple experiment: Show subjects pictures of different objects (e.g., shoes vs. bottles) on different trials of different runs
21. FROM WORDNET TO BRAINNET Neural evidence, unlike the evidence used to compile dictionaries and WordNet, and like the evidence one gathers from corpora and certain behavioral experiments, is entirely objective (although it can be subjective in the sense of differing from subject to subject) The objective of our research is to combine evidence from brain data, from corpora, and from behavioral experiments (all of which is rather noisy) to develop a new architecture for conceptual knowledge: BrainNet
22. FIRST CASE STUDY: ABSTRACT CONCEPTS Until recently, most work on concepts in CL / neuroscience / psychology focused on concrete concepts But the type of conceptual knowledge that really challenges traditional assumptions about its organization are `abstract concepts’ – or to be more precise, the set of categories of non-concrete concepts Events / actions States ‘Urabstract’ concepts: LAW, JUSTICE, ART We are carrying out explorations of abstract knowledge using fMRI
23. THEORIES OF ABSTRACT CONCEPTS IN COGNITIVE NEUROSCIENCE In CL/AI: TAXONOMIC organization for both abstract and concrete concepts Best known Cognitive Neuroscience: Paivio’s DUAL CODE theory (Paivio, 1986) CONCRETE: verbal system & visual system ABSTRACT: verbal system only Schwanenflugel & Akin 1994: CONTEXT AVAILABILITY Barsalou’s SCENARIO-BASED MODEL (Barsalou, 1999): Abstract knowledge organized around SCENARIOS
24. THE OBJECTIVES OF OUR EXPERIMENT Identify the representation in the brain of a variety of WordNet categories exemplifying both concrete and abstract concepts (abstract words chosen by inspecting the words rated as most abstract in the De Rosa et al norms 2005) Really abstract: ATTRIBUTE, COMMUNICATION, EVENT, LOCATION, ‘URABSTRACT’ A category of concrete objects: TOOLS A complex category: SOCIAL-ROLE Comparing two types of classification: TAXONOMIC (as in WordNet) DOMAIN (cfr. Barsalou’s hypothesis about abstract concepts being ‘situated’) Two domains: LAW and MUSIC Using WordNet Domain
25. STIMULI
26. STIMULI, 2: URABSTRACTS
27. STIMULI, 3: SOCIAL ROLES
28. ABSTRACT CONCEPTS: DATA COLLECTION AND ANALYSIS 7 right-handed native speakers of Italian Task: Words presented in white on grey screen for 10 sec Cross in between, 7 sec Subjects had to think of a situation in which the word applied Scanner: 4T BrukerMedSpec MRI scanner, EPI pulse sequence TR=1000ms, TE=33ms, 26° flip angle. Voxeldimensions 3mm*3mm*5mm Preprocessing: using UCL’s Statistical Parameter Mapping Software Data corrected for head motion Classification: using a single layer NN
29. MAIN QUESTIONS Can the taxonomic and domain classes be distinguished from the fMRI data? Is there a difference in classification accuracy between taxonomy and domain? Can the taxonomic and domain classes be predicted across participants?
30. RESULTS WITHIN PARTICIPANTS (CATEGORY DISTINCTIONS) ALL CATEGORICAL DISTINCTIONS CAN BE PREDICTED ABOVE CHANCE THERE ARE SIGNIFICANT DIFFERENCES BETWEEN CATEGORIES
31. RESULTS WITHIN PARTICIPANTS(DOMAIN)
32. WITHIN PARTICIPANTS RESULTS SUMMARY Can discriminate with accuracy well above chance both taxonomic and domain distinctions Easiest categories to recognize: TOOL, ATTRIBUTE, LOCATION, Then SOCIAL ROLE, COMMUNICATION Main confusions: communication / event
33. CATEGORY LOCALIZATION IN THE BRAIN Red: Attribute Blue: Tool Green: Location R+G=Yellow G+B=Cyan R+B=Pink R+G+B=White
34. Red: Social-role Green: Communication Blue: Event R+G=Yellow G+B=Cyan R+B=Pink R+G+B=White Red: Social-role Green: Attribute Blue: Urabstract
35. CROSS PARTICIPANTS RESULTS SUMMARY Concrete taxonomic classes tool and location can be predicted across participant, attribute can also be significantly classified, but less concrete classes become conflated with attribute. In general domain can be predicted across participants, however domain membership is much better classified in the most abstract taxonomic classes (attribute, communication and urabstract) Visually apparent inter-region differences in activation. The precuneus appears to contain voxels systematically associated with independent taxonomic/topical categories.
36. CROSS PARTICIPANTS RESULTS SUMMARY Concrete categories TOOL and LOCATION can be predicted across participant; ATTRIBUTE can also be significantly classified; but less concrete classes become conflated with ATTRIBUTE. In general DOMAIN can be predicted across participants, however domain membership is much better classified in the most abstract taxonomic classes (attribute, communication and urabstract)
37. TAXONOMIC / DOMAIN ORGANIZATION LAW MUSIC Attribute giurisdizione jurisdiction sonorita' sonority cittadinanza citizenship ritmo rhythm impunita' impunity melodia melody legalita' legality tonalita’ tonality illegalita' illegality intonazione pitch communicationdivieto prohibition canzone song verdetto verdict pentagramma stave ordinanza decree ballata ballad addebito accusation ritornello refrain ingiunzione injunction sinfonia symphony event arresto arrest concerto concert processo trial recital recital reato crime assolo solo furto theft festival festival assoluzione acquittal spettacolo show social-rolegiudice judge musicista musician ladro thief cantante singer imputato defendant compositore composer testimone witness chitarrista guitarist avvocato lawyer tenore tenor toolmanette handcuffs violino violin toga robe tamburo drum manganello truncheon tromba trumpet cappio noose metronomo metronome grimaldello skeleton key radio radio Locationtribunale court/tribunal palco stage carcere prison auditorium auditorium questura police station discoteca disco penitenziario penitentiary conservatorio conservatory patibolo gallowsteatro theatre urabstractsgiustizia justicemusica music liberta' liberty blues blues legge law jazz jazz corruzione corruption canto singing refurtiva loot punk punk
38. WHAT THE DATA SUGGESTS
39. EEG vsfMRI A question about conceptual organization that can be clearly investigated using neural evidence is: Which categories can be distinguished? But: fMRI too expensive to carry out systematic investigations (~500 eurosx hour) Alternative: EEG Used in BCI for a variety of ‘mind reading’ tasks Also used to study semantics with ERPs
40. EEG vs. fMRI
41. Gaussian Naive-Bayes, SM Log. Regression, Linear SVM USING EEG TO STUDY SEMANTICS: ERP Features: signal amplitude and slope at range of resolutions gives compact representation of waveform N400 Violations of person and number in pronoun-verb agreement Up to 70% detection on single trials
42. EEG Spectral Analysis of Concepts? Participants presented with aural or visual concept stimuli EEG apparatus records electrical activity on the scalp Waveforms can be reduced to frequency components
43. EEG pros and cons Pros: Lighter Cheaper Better temporal resolution (ms) Cons: Coarser spatial resolution (cm) Noisy (e.g., very sensitive to skull depth)
44. EEG CAN BE USED TO IDENTIFY MAJOR CATEGORICAL DISTINCTIONS Murphy et al, Brain and Language 2011: 7 Italian subjects 30 animals, 30 tools Each presented 6 times Task: silent naming
45. STIMULI
46. EEG SIGNALS: TIME-FREQUENCY (PER CHANNEL)
47. Classification System Schematic 64 channels preprocessed data X channels filtered data* “Tool” component Feature vector Filter by Time, Freq and Eelectr. CSSD Decomposition Vector Transform var(“tool”), var(“animal”) SupVec Machine Answer ? “Animal” component Data analysis
48. RESULTS Time/Freq window optimisation, CSP extraction of class-sensitive sources, 5-fold cross-validated SVM With group analysis, 98% accuracy categorising mammals vs tools Murphy et al, 2011, Brain and Language
49. PRELIMINARY CONCLUSIONS EEG can be used to decode broad categorical distinctions May need to use fMRI to study Finer grained distinctions Cross-language distinctions
50. BRAIN EVIDENCE AND CORPUS EVIDENCE Can we find ways of combining evidence about strength of categorial distinctions coming from EEG / fMRI with the evidence coming from corpora? First question: what is the relation between the conceptual spaces induced from corpora and the conceptual spaces elicited using EEG?
51. PREDICTING BRAIN (FMRI) ACTIVATION USING CONCEPT DESCRIPTIONS T. Mitchell, S. Shinkareva, A. Carlson, K. Chang, V. Malave, R. Mason and M. Just. 2008. Predicting human brain activity associated with the meanings of nouns. Science320, 1191–1195
52. MITCHELL ET AL 2008: METHODS Record fMRI activation for 60 nominal concepts And extract 200 ‘best’ features, or VOXELs Build conceptual descriptions for these concepts from corpora (the Web) 25 features for each concept 25 verbs expressing typical properties of living things / tools Collect strength of association between these features and each concept Learn association between each voxel and the 25 verbal features using 58 concepts Use learned model to predict activation of 2 held-out data (compare using Euclidean distance) Accuracy: 77%
53. MITCHELL ET AL 2008
54. MITCHELL ET AL 2008: VERB FEATURES
55. MITCHELL ET AL: LEARNING ASSOCIATIONS
56. OUR EXPERIMENTS Replicate the Mitchell et al study using EEG data instead of fMRI Different feature selection mechanisms Compare different methods for building concept descriptions In addition to hand-picked, also a variety of standard corpus models For Italian B. Murphy, M. Baroni, and M. Poesio, EEG responds to conceptual stimuli and corpus semantics, EMNLP 2009
57. RESULTS USING THE HAND-PICKED FEATURES
58. AA-MP MITCHELL ET AL RESULTS USING AUTOMATICALLY SELECTED FEATURES
59. INTERIM SUMMARY It is possible to establish systematic links between knowledge about concepts acquired from corpora and knowledge extracted from brain data These links may be used for instance to compare ontology learning methods (need however to extend the investigation of categorial distinctions discussed above)
60. APPLICATIONS ‘Mind-reading’ techniques can be used for a variety of other studies of interest to CL types DEEP RELATIONS: fMRIcan be used to extract information about POLARITY This can be used for sentiment analysis in text ADAM: being able to distinguish between ANIMALS and TOOLS using EEG can be used as an early predictor of certain classes of semantic dementia
61. CONCLUSIONS Evidence from neuroscience, combined with evidence from corpora and from behavioral studies, may be used to put our theories of the lexicon on a firmer empirical footing The resulting resources may be more useful both for HLT and for other applications
62. THANKS!