1 / 37

WP 4 Language Emergence

WP 4 Language Emergence. Britta Wrede (BIEL) Gerhard Sagerer, Katharina Rohlfing, Karola Pitsch, Katrin Lohan, Lars Schillingmann, BIEL Jun Tani, RIKEN Stefano Nolfi, CNR Angelo Cangelosi, Martin Peniak PLYM

deloresc
Download Presentation

WP 4 Language Emergence

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WP 4 Language Emergence Britta Wrede (BIEL) Gerhard Sagerer, Katharina Rohlfing, Karola Pitsch, Katrin Lohan, Lars Schillingmann, BIEL Jun Tani, RIKEN Stefano Nolfi, CNR Angelo Cangelosi, Martin Peniak PLYM Chrystopher Nehaniv, Kerstin Dautenhahn, Yo Sato, Joe Saunders, Frank Förster, Caroline Lyon, UH Kerstin Fischer, Arne Zeschel, USD

  2. Overview Task 4.1 Generalization as a basis for emergence of symbolic systems (start: M7) Task 4.2 Acoustic Packaging and the learning of words (start: M13) Task 4.3 From single word lexicons to compositional languages (start: M13) Task 4.4 Constructional grounding and primary scenes (start: M19) Task 4.5 Evolutionary origins of action nd language compositionality (start: M31) ITALK Year 1 Review Düsseldorf, 30 June 2009

  3. Objectives & Goals GrammaticalConstructions 4.4 Lexicon Construction Speech 4.3 ActionHierarchy 4.1 Action Acoustic Packages 4.2 ITALK Year 1 Review Düsseldorf, 30 June 2009

  4. Objectives Year 1 • Robotics experiments on the development of action categorisation as a basis for linguistic communicative capabilities • Analysis of the role of temporal synchrony between speech and action as a basis for the implementation of a module that fuses two modalities to segment actions ITALK Year 1 Review Düsseldorf, 30 June 2009

  5. Task 4.1 Generalization as a basis for emergence of symbolic systems (Start: M7) • Emergence of action structure through use of slow vs fast context units in a Multiple Timescale Recurrent Neural Network (MTRNN) • Fast context units are able to find primitives • Slow context units are able to sequence primitives without explicit learning ITALK Year 1 Review Düsseldorf, 30 June 2009

  6. c a b b a c b c Homunculus?? a b a Reparatory of primitives Combined streams Compositionality in Action Generation • How to acquire set of reusable behavior primitives to be combined to generate actions? ITALK Year 1 Review Düsseldorf, 30 June 2009

  7. Teach time Method: Self-Organization of Functional Hierarchy [Yamashita & Tani, 2008] Behavioral Compositionality MTRNN τ= 50.0 slow time constant Slow τ= 5.0 Fast time constant Fast ITALK Year 1 Review Düsseldorf, 30 June 2009

  8. 4.1 Summary • Emergence of action structure through fast and slow context units • Slow context units are able to learn composition / sequencing of primitives 4.1 Outlook • Extend Tani’s Recurrent Networks, e.g. using (pre)trained networks on actions and teach words for action and object categories/properties • Evaluation of MTRNN on Motionese Corpus (input: hand trajectories) • Hypothesis about relation of action hierarchy with language hierarchy: verbs related to slow context units, objects to fast context units

  9. Overview GrammaticalConstructions 4.4 Lexicon Construction Speech 4.3 ActionHierarchy 4.1 Action Acoustic Packages 4.2 ITALK Year 1 Review Düsseldorf, 30 June 2009

  10. 4.2 Acoustic PackagingBackground(Start: M13) • How to associate information in different modalities for language learning? • Synchrony [Zukow-Goldring, 1997] [Matatyaho, Mason & Gogate et al., 2007] • Synchronous object movement and verbal labeling enhances object learning • More low-level synchrony in ACI than in AAI [Rolf et al., 2009] • Acoustic Packaging [Brand et al, 2007] • Synchrony between language and events helps to divide sequence of events into units [Hirsh-Pasek & Golinkoff, 1996] • Speech segment determines perceived (end of) action

  11. Acoustic Packaging [Brand et al., 2007] Question: Does speech influence how action is structured by infants? Experiment: 32 Infants of 7.5 – 11.5 months of age; Preferential Looking A B A C Vision Familia-rization Wow! Do you see what she‘sdoing? She‘s blixing! Audio Preferred Sequence A B B A Test: Split Screen • Non-packaged sequence perceived as new • Speech structures action !

  12. Computational Model of Acoustic Packaging [Schillingmann et al., 2009, best paper award at the ICDL09] Long term goals • Temporal segmentation of actions • Generating appropriate feedback • Integration with imitation learning approaches Evaluation • Does model reflect structural properties of tutoring behavior? ITALK Year 1 Review Düsseldorf, 30 June 2009

  13. Computational Model of Acoustic Packaging Segmentation • Speech: by ASR (ESMERALDA) Temporal Association • Acoustic Package created if segments overlap • Action: by motion history images ITALK Year 1 Review Düsseldorf, 30 June 2009

  14. Computational Model of Acoustic Packaging ITALK Year 1 Review Düsseldorf, 30 June 2009

  15. Computational Model of Acoustic Packaging ITALK Year 1 Review Düsseldorf, 30 June 2009

  16. Evaluation Data • Videos from Motionese corpus (11 AAI, 11 ACI) and from babyface study (11 ARI) • Task: stacking cups Analysis • Automatic detection of Acoustic Packages • Measurements: • number of Acoustic Packages (#AP) • mean number of motions per Acoustic Package (#motions / AP) Hypothesis • ACI more structured than AAI • More #AP and less #motions / AP in ACI ITALK Year 1 Review Düsseldorf, 30 June 2009

  17. Results • Sig. more Acoustic Packages in ACI and ARI • Sig. less Motions per Acoustic Packages in ACI and ARI • Hypothesis confirmed • Automatically detected Acoustic Packages find more structure in ACI and ARI ITALK Year 1 Review Düsseldorf, 30 June 2009

  18. 4.2 Outlook Interaction • Designing feedback of iCub robot based on AP for user studies (Live-Demo!) • Hyp. 1: Signalling detected AP ends to user will increase contingency in interaction • Hyp. 2: APs are related to action structure and therefore signal understanding ITALK Year 1 Review Düsseldorf, 30 June 2009

  19. 4.2 Outlook Learning • Use AP as units for • Speech learning • Action learning • Comparison with other segmentation approaches (e.g. MTRNNs) ITALK Year 1 Review Düsseldorf, 30 June 2009

  20. Overview GrammaticalConstructions 4.4 Lexicon Construction Speech 4.3 ActionHierarchy 4.1 Action Acoustic Packages 4.2 ITALK Year 1 Review Düsseldorf, 30 June 2009

  21. 4.3 From single word to compositional lexicons(Start: M13) Relation to Task 1.4 – Hierarchical Actions with RNNPB In T1.4 the main aim is to develop neural architectures capable of learning compositional and hierarchical actions This is currently being investigated through the use of Tani’s Recurrent Neural Network : (i) RNNPB with Parametric Bias for hierarchical architecture and (ii) TRNN fully time recurrent for emergence hierarchy. Future Work in Task 4.3 Extend Tani’s Recurrent Networks, e.g. using (pre)trained networks on actions and teach words for action and object categories/properties

  22. 4.3 and 4.4 Grammar learning in children Tomasello (2003): From holophrases… lemme-see (=let me see) …to pivot schemas… lemme Z (=let me Z) …to item-based constructions… X let Y Z …to abstract constructions SUBJ VERB OBJ COMP

  23. 4.3 and 4.4 Grammar learning in children Tomasello (2003): From holophrases… lemme-see …to pivot schemas… lemme Z …to item-based constructions… X let Y Z …to abstract constructions SUBJ VERB OBJ COMP starting point: concrete, simplex

  24. 4.3 and 4.4 Grammar learning in children Tomasello (2003): From holophrases… lemme-see …to pivot schemas… lemme Z …to item-based constructions… X let Y Z …to abstract constructions SUBJ VERB OBJ COMP starting point: concrete, simplex end point: abstract, complex

  25. 4.3 and 4.4 Grammar learning in children Tomasello (2003): From holophrases… lemme-see …to pivot schemas… lemme Z …to item-based constructions… X let Y Z …to abstract constructions SUBJ VERB OBJ COMP starting point: concrete, simplex end point: abstract, complex in between: increasing abstraction & complexity

  26. 4.3 and 4.4 Grammar learning Argument structure constructions Elementary blueprints for predicating basic event types intransitive He walks N V intransitive motion He walks through the park N V OBL transitive He walks the dog N V N transitive motion He walks the dog through the N V N OBL park Holistic constructions are associated with semantic frames Formal constituents in syntax map to conceptual constituents in event structure

  27. 4.3 and 4.4 Grammar learning (Adapted from Goldberg 1995) Sem CAUSE-MOVE AGT THM LOC Syn V SUBJ OBJ OBL

  28. 4.3 and 4.4 Grammar learning (Adapted from Goldberg 1995) Semantics: CAUSED MOTION frame Sem CAUSE-MOVE AGT THM LOC Syn V SUBJ OBJ OBL

  29. 4.3 and 4.4 Grammar learning (Adapted from Goldberg 1995) Semantics: CAUSED MOTION frame Sem CAUSE-MOVE AGT THM LOC Syn V SUBJ OBJ OBL Syntax: Complex transitive complementation

  30. 4.3 and 4.4 Grammar learning (Adapted from Goldberg 1995) Semantics: CAUSED MOTION frame Sem CAUSE-MOVE AGT THM LOC Syn V SUBJ OBJ OBL Syntax: Complex transitive complementation Correspondence links

  31. 4.3 and 4.4 Planned Experiments Associate holistic utterance with complex event structure → holophrase learning Segment/decompose sequence into constitutive elements → word/morpheme learning Identify recurrent functional relationships between constituent markings and meanings → construction learning

  32. 4.3 and 4.4 Planned Experiments • Generalisation within construction (Replication Sugita & Tani, 2005)V N • Generalisation across 2 constructionsV | V N V N | V N OBLpush blockkick ball to-the-boxpush block to-the-box • Generalisation across N+1 constructions (increase complexity)V N | V N OBL | V | V N N • Experiment with acquisition order • Generalisation across N+1 constructions with empirically motivated statistical biases ITALK Year 1 Review Düsseldorf, 30 June 2009

  33. 4.3 and 4.4Empirically motivated statistical biases Experimentally vary… • Statistical learning cues available to learner, e.g. • relative constructional frequencies in the input • lexical type frequencies per constructional slot • availability of ‘pathbreaking’ verbs for particular event types • amount of lexical overlap between constructions → input modeled on basis of WP 3.1 grammar classifications • Effect of combining the availability of linguistic input-statistical with tutor-provided paralinguistic cues ITALK Year 1 Review Düsseldorf, 30 June 2009

  34. WP4 Summary • Extension of the work at RIKEN on the use of generalisation as a basis for compositional linguistic communication • Multiple timescales recurrent neural network (MTRNN) for action and language learning experiments. • For experiments planned jointly by PLYM and USD on compositional action and language learning • Analysis of the phenomenon of synchrony between verbal utterances and action for an objective measurement of synchrony in multimodal behaviour • Integration of acoustic packaging modules on the iCub platform using the XCF integration framework ITALK Year 1 Review Düsseldorf, 30 June 2009

  35. WP4 Outlook • Interaction studies to analyse feedback produced wrt Acoustic Packages • Use MTRNN for joint learning of action and speech as a basis for construction learning • Data analysis and first modeling approaches of construction grammar ITALK Year 1 Review Düsseldorf, 30 June 2009

  36. Thank you! GrammaticalConstructions 4.4 Lexicon Construction Speech 4.3 ActionHierarchy 4.1 Action Acoustic Packages 4.2 ITALK Year 1 Review Düsseldorf, 30 June 2009

  37. Thank you! ITALK Year 1 Review Düsseldorf, 30 June 2009

More Related