1 / 68

WP3 Social Learning and Interaction

This study investigates the role of social learning and imitation in shared communication, studying actions and gestures in learning of multimodal behavior, variability in infant-directed actions, and user expectations for correlations and hierarchies of robot capabilities.

menard
Download Presentation

WP3 Social Learning and Interaction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WP3 Social Learning and Interaction Joe Saunders1, Chrystopher L. Nehaniv1, Kerstin Dautenhahn1 Frank Förster1, Yo Sato1, Caroline Lyon1, Gerhard Sagerer2, Britta Wrede2, Katherina Rolhfing2, Karola Pitsch2, Katrin Lohan2, Lars Schillingmann2, Kerstin Fischer3, Arne Zeschel3, Angelo Cangelosi4 1University of Hertfordshire, 2University of Bielefeld, 3University of Southern Denmark. 4University of Plymouth Düsseldorf, Germany 30 June – 1 July 2009

  2. WP 3 Objectives • to investigate the role of social learning and imitation for shared communication • to study how socially constructed linguistic knowledge can indicate negation • to study actions and gestures in the learning of multimodal task-oriented behaviour • to study the variability in infant-directed actions to understand emulation of actions • to implement a “tutoring spotter” to exploit regularities in child-directed interactions • to study user expectations for correlations and hierarchies of robot capabilities • to study the differences between infant and robot directedspeech to inform developmental robotics research Düsseldorf - June 2009

  3. WP 3 Participants UH USD BIEL PLYM WP 3 Start Month • WP3 officially started in month 7 of year 1 of ITALK Düsseldorf - June 2009

  4. WP3 Summary Progress and Integration WP 3.1 - An empirically motivated ‘constructicon’ for HRI and communication experiments constructicon working paper WP 3.2 – HRI experiments on deixis, gesture and reference ITALK consortium HRI Experiments: PLYM and P.Dominey HRI Experiments Socially Acquired Language iCub/Kaspar Learning Architecture WP 3.4 – Analysis Adult/Child and Adult/Adult tutoring WP 3.3 – HRI experiments on negation WP 3.5 – Expectations in HRI of robotic linguistic capability tutoring spotter contingent behaviour variability of tutoring behaviour HRI Experiments Düsseldorf - June 2009

  5. Background “In the beginning, there was the act.” - Goethe A robot will use what it learns individually and from others socially through grounded sensorimotor interaction to bootstrap the acquisition of language. This creates a positive feedback cycle between using language and developing other cognitive abilities. Düsseldorf - June 2009

  6. Background A view of Language Competence: (Langacker 1987-2000 Croft 1992,2001 Goldberg 1995 Givon 1995 Bybee1985-2002 Tomasello 1998-present) Cognitive-functional linguistics Usage-Based linguistics Language structure emerges from language use “When human beings use symbols to communicate with one another, stringing them together into sequences, patterns of use emerge and become consolidated into grammatical constructions”(Tomasello 2003) Düsseldorf - June 2009

  7. Background Usage Based Linquisitics: Language structure emerges from language use When humans use symbols to communicate, they string the symbols together in sequences Patterns of use emerge Grounded in sensory-motor processes Consolidated into grammatical constructions Düsseldorf - June 2009

  8. Background Usage Based Linguistics - Implications Constructions may be acquired with the same basic set of acquisitional processes Intention reading and pattern finding Constructions may contain concrete language segments as well as idiosyncratic phrases and item-based constructions Constructions are templates pairing syntax with meaning Düsseldorf - June 2009

  9. Language Acquisition Not isolated association making and induction but integrated with other cognitive and social-cognitive skills. • These fall into two broad sets: • Intention reading • Ability to share attention with other persons to objects and events of mutual interest • The ability to follow the attention and gesturing of other persons to distal objects and events outside the immediate interaction • The ability to actively direct the attention of others • The ability to culturally learn the intentional actions of others • Categorisation • Ability to form perceptual and conceptual categories of “similar” objects and events • Ability to form sensory-motor schemas from recurrent patterns of perception and action • Ability to perform statistically based distribution analyses on various kinds of perceptual and behavioural sequences • The ability to create analogies across two or more complex wholes based on similar functional roles of these wholes Düsseldorf - June 2009

  10. Language Acquisition Not isolated association making and induction but integrated with other cognitive and social-cognitive skills. • These skills fall into two broad sets: • Intention reading • Ability to share attention with other persons to objects and events of mutual interest • The ability to follow the attention and gesturing of other persons to distal objects and events outside the immediate interaction • The ability to actively direct the attention of others • The ability to culturally learn the intentional actions of others • Categorisation • Ability to form perceptual and conceptual categories of “similar” objects and events • Ability to form sensory-motor schemas from recurrent patterns of perception and action • Ability to perform statistically based distribution analyses on various kinds of perceptual and behavioural sequences • The ability to create analogies across two or more complex wholes based on similar functional roles of these wholes Düsseldorf - June 2009

  11. Language Acquisition Not isolated association making and induction but integrated with other cognitive and social-cognitive skills. • These skills fall into two broad sets: • Intention reading • Ability to share attention with other persons to objects and events of mutual interest • The ability to follow the attention and gesturing of other persons to distal objects and events outside the immediate interaction • The ability to actively direct the attention of others • The ability to culturally learn the intentional actions of others • Categorisation • Ability to form perceptual and conceptual categories of “similar” objects and events • Ability to form sensory-motor schemas from recurrent patterns of perception and action • Ability to perform statistically based distribution analyses on various kinds of perceptual and behavioural sequences • The ability to create analogies across two or more complex wholes based on similar functional roles of these wholes Düsseldorf - June 2009

  12. WP 3.1 and WP 3.5 3.1 An empirically motivated ‘constructicon’ for HRI and communication experiments 3.5 Expectations in HRI of robotic linguistic capability Düsseldorf - June 2009

  13. WP3 Summary Progress and Integration WP 3.1 - An empirically motivated ‘constructicon’ for HRI and communication experiments constructicon working paper ITALK consortium HRI Experiments: PLYM and P.Dominey WP 3.5 – Expectations in HRI of robotic linguistic capability HRI Experiments Düsseldorf - June 2009

  14. Objectives WP 3.1 An empirically motivated constructicon for HRI • Define an elementary constructicon (set of grammatical target constructions) for robotic language learning experiments • Identify facilitative cues to distributional learning of these patterns in naturally occurring child-directed speech (CDS) through qualitative & quantitative corpus-linguistic analysis • Incorporate extracted features into stimuli to be used for robotic language learning/generalisation experiments (→ WP 4) Düsseldorf - June 2009

  15. Data WP 3.1 Dependency-parsed CDS from CHILDES Sample transcript Feature Structure/Parse tree Sagae, K. et al. 2004. Automatic parsing of parental verbal input. Behaviour Research Methods, Instruments, and Computers 36, 1, 113-126. Düsseldorf - June 2009

  16. Parameters WP 3.1 Investigated properties • Lexical frequencies • Relative constructional frequencies • Lexical type frequency per constructional slot • Significant associations between particular lexical items and constructional slots • Amount of lexical overlap between constructions • Occurrence in variation sets • → flexibly extendable depending on experimental interests Düsseldorf - June 2009

  17. WP 3.1 - Summary Objective: -creation of an empirically motivated set of linguistic constructions for grammar and language learning experiments in robots Progress: - review of key literature - generation of constructivist classifications of input constructions to be employed in ITALK consortium experiments and - working paper on criteria and experiments distributed to ITALK partners Düsseldorf - June 2009

  18. WP 3.5 3.5 Expectations in HRI of robotic linguistic capability Düsseldorf - June 2009

  19. Corpus Elicitation WP 3.5 • Cooperation BIEL – USD (Vollmer, Lohan et al. 2009) • 30 human-robot interactions • simulated robot • attention to • movements • colours • skin • tasks • explain lamp • show bell • show putting on salt • block world task • putting rings in box • stacking cups Düsseldorf - June 2009

  20. Corpus Elicitation WP 3.5 Two parts • replication of corpus of child-directed speech previously elicited by BIEL (Rohlfing et al. 2006; Vollmer, Lohan et al. 2009) • investigation of differences in linguistic and non-verbal strategies in tutoring scenarios • explication of the meanings of a set of sentences • investigation of constructional relationships • 'construction conspiracy' • presupposed concepts and constructions • investigation of construction-specific versus verb-specific instruction modes Düsseldorf - June 2009

  21. Comparison CDS - HRI WP 3.5 Differences regarding Concept of Communication Partner • 20 of 30 participants requested additional information in HRI, but none in CDS • estimated age of robot: range 6 months – 10 years, mean 3.7, SD 2.7 • high variability in HRI Düsseldorf - June 2009

  22. Comparison CDS - HRI WP 3.5 Differences in Linguistic Complexity • number of relative clauses CDS: M = 0.04 HRI: M= 0.22 ( F(1,340) =16,0699, p < .001) • number of direct, indirect objects and object complements CDS: M = 1.38 HRI: M = 3.68 (F(1,340) = 73,88916, p< .001) • number of abstract terms CDS: M = 0.0 HRI: M = 0.37 (F(1,340) = 23,89889), p< .001) Düsseldorf - June 2009

  23. Comparison CDS - HRI WP 3.5 Parent to Child: 'Look that's how you put salt on' Human to Robot: 'this is an uh container in which uh salt crystals are uh contained' Düsseldorf - June 2009

  24. Comparison CDS - HRI WP 3.5 • Differences in verbosity • number of words CDS: M = 45.5 HRI: M = 60.5 (F(1,340) = 14,54830, p < .001) • number of different words CDS: M = 24.7 HRI: M = 34.1 (F(1,340) = 33,66639, p < .001) • input for learning experiments from CDS • ‘shaping’ strategies necessary for HRI learning experiments Düsseldorf - June 2009

  25. WP 3.5 - Summary Objective: -study of the role of user expectations in human-robot interaction to inform future HRI episodes Progress: -empirical analysis of HRI data (cooperation BIEL-USD) -contrastive analyses of the data Vollmer A. L., Lohan K. S., Fischer K., Nagai Y., Pitsch K., Fritsch J., Rohlfing K. J. & Wrede B. (2009). People modify their tutoring behaviour in Robot-Directed Interaction for Action Learning. Proceeding of International Conference on Development and Learning (ICDL). Düsseldorf - June 2009

  26. WP 3.2 and WP 3.3 WP 3.2 – HRI experiments on deixis, gesture and reference WP 3.3 – HRI experiments on negation Düsseldorf - June 2009

  27. WP3 Summary Progress and Integration WP 3.1 - An empirically motivated ‘constructicon’ for HRI and communication experiments constructicon working paper ITALK consortium HRI Experiments: PLYM and P.Dominey WP 3.5 – Expectations in HRI of robotic linguistic capability HRI Experiments Düsseldorf - June 2009

  28. WP3 Summary Progress and Integration WP 3.1 - An empirically motivated ‘constructicon’ for HRI and communication experiments constructicon working paper WP 3.2 – HRI experiments on deixis, gesture and reference ITALK consortium HRI Experiments: PLYM and P.Dominey HRI Experiments Socially Acquired Language iCub/Kaspar Learning Architecture WP 3.3 – HRI experiments on negation WP 3.5 – Expectations in HRI of robotic linguistic capability HRI Experiments Düsseldorf - June 2009

  29. Objectives WP 3.2 to model the positive feedback cycle observed in infants whereby gradual acquisition of linguistic competence occurs • transfer and extension of learning architecture to humanoid platform (k2)+ yarp integration for language games • planning and start of experiments on socially acquired linguistic knowledge • started work on ideas on grammar induction coordinated with experiments above Düsseldorf - June 2009

  30. Outline WP 3.2 WP 3.2 Word and Phrase Acquisition Without Meaning Acquisition of Meaning via Mediated Physical Interaction Semantic based grammar induction Corpus of mapped phoneme and robot sensory/motor proprioceptions segmented utterances words, holophrases Induced syntax object/action ‘components’ available for scaffolding Language constructions WP 3.1/3.5 constructions Düsseldorf - June 2009

  31. Outline WP 3.2 WP 3.2 Word and Phrase Acquisition Without Meaning Acquisition of Meaning via Mediated Physical Interaction Semantic based grammar induction Corpus of mapped phoneme and robot sensory/motor proprioceptions segmented utterances words, holophrases Induced syntax object/action ‘components’ available for scaffolding Language constructions WP 3.1/3.5 constructions Düsseldorf - June 2009

  32. WP 3.2 From Babbling to the Acquisition of Words and Phrases Without Meaning Aim: To show how a robot might learn to segment an utterance, an acoustic stream of sounds, as the basis for extracting meaningful elements (Tomasello (2003)). “new research shows how children attend to patterns in speech in their first year, and listen for meaning in the second” – Fernald, A. & Marchman, V. A. (2006). Language learning in infancy. In Handbook of Psycholinguistics, Ed Traxler and Gernsbacher Approach 1. Develop a simulation of early learning processes – under way 2. Use this as a test bed to investigate the role of various mechanisms Düsseldorf - June 2009

  33. Examples of input and output at intermediate learning stage WP 3.2 Input: Actual speech from “teachers” to robot Output: First words and holophrases are learnt, among babble Utterance syllable type CV, repeated t u t u t u syllable type Ve syllable type CVCbl o d Syllable type CVCbl a k syllable recognised as word “black” by teacher syllable type CV, repeated p e p e Düsseldorf - June 2009

  34. Outline WP 3.2 WP 3.2 Word and Phrase Acquisition Without Meaning Acquisition of Meaning via Mediated Physical Interaction Semantic based grammar induction Corpus of mapped phoneme and robot sensory/motor proprioceptions segmented utterances words, holophrases Induced syntax object/action ‘components’ available for scaffolding Language constructions WP 3.1/3.5 constructions Düsseldorf - June 2009

  35. Outline WP 3.2 WP 3.2 Word and Phrase Acquisition Without Meaning Acquisition of Meaning via Mediated Physical Interaction Semantic based grammar induction Corpus of mapped phoneme and robot sensory/motor proprioceptions segmented utterances words, holophrases Induced syntax object/action ‘components’ available for scaffolding Language constructions WP 3.1/3.5 constructions Düsseldorf - June 2009

  36. Background WP 3.2 Scaffolded Teaching via Assisted Imitation • How can a child discover affordances and effectivities for action?: • by trial-and-error – may be slow • by observational imitation – but correspondence problem exists • by assisted/self-imitation • Caregivers invite infants to imitate – by putting infants on both sides of the correspondence problem • Assisted/Self-Imitation • Putting through of infants movements • Acting in tandem • Linking effectivities of the infants body with affordances for action • Scaffolding taught competencies (* Affordances, Effectivities and Assisted Imitation - Patricia Zukow-Goldring, Michael Arbib (2007).) Düsseldorf - June 2009

  37. ROSSUM Behavioural Component / Memory Model WP 3.2 Current State ‘Ideas of Movement’ From Previous Learning Effects of Movement Perceived State Action State Resulting From Action Execution Perceived State Action State Resulting From Action Execution Perceived State Action State Resulting From Action Execution Similarity Matching Similarity Matching ... Perceived State Action State Resulting From Action Execution Model State Acts as a Forward Model Acts as an Inverse Model

  38. WP 3.2 State Behaviour Sequence Goal-directed Task Primitive Goal-directed Task Behaviour Primitive Primitive Primitive Primitive Primitive Primitive

  39. WP 3.2 Acquisition of Meaning via Mediated Physical Interaction Approach • ‘Usage-based’ approach to infant language acquisition • – Tomasello (2003), Bloom (2002) • Motivated by Previous Research on: • Linguistic Emergence - Steels (1998) • Linkage of speech with object and action perception – Roy (2005), Roy and Pentland (2002), Yu and Ballard (2004) Research Questions • Are associative clusters formed between human speech and gestural actions and a robot’s action, visual and proprioceptive perceptions? • If present are the clusters scaffolded hierarchically? • Is it possible to exploit existing learned sensory/motor circuits and behaviours and to associate these at appropriate points on the scaffolded language usage hierarchy?

  40. WP 3.2 ROSSUM Migrated to Humanoid Platform Extended for Use in Language Games and for future use with iCub Düsseldorf - June 2009

  41. Acquisition of Meaning via Mediated Physical InteractionApproach so far... WP 3.2 Meaning –> where an utterance is grounded in the robot’s sensory/motor history from acting in the world Grounded meanings can be scaffolded –> via regularities in the recognised phoneme/sensory motor stream Referential intent? -> ‘appearance’ of shared attention What’s important in the sensory/motor stream? -> apply mutual information Reaction and prediction -> learnt words considered as actions – same ‘unit’ used for perception and prediction Heuristics for biased learning -> Words are strings of phonemes End of utterance word placement Word duration Temporal mismatch -> apply selected word backwards over all of previous utterance Düsseldorf - June 2009

  42. Experiment 2 WP 3.2 • Teach the robot about 6 ‘shapes’ • ‘as if’ the robot were a child 1-2 years old • 8 Participants • 5 sessions with the robot of 2 minutes (total of 40 robot interaction sessions) • Unrestricted speech • Robot speech feedback to 2 participants semi-random (based on their interaction but not on correct match) table human robot 6 pre-trained AR ToolKit patterns cam1 Cam 2 Düsseldorf - June 2009

  43. Experiment 2 WP 3.2 Session 1 Kaspar has no Language Associations Session 3 Kaspar has some Language Associations Düsseldorf - June 2009

  44. Experiment 2 - Mechanics WP 3.2 Robot Sensory/motor stream Phonemes & timings Human Learning

  45. Experiment 2 - Mechanics WP 3.2 Robot Current Sensory Motor State K Nearest Neighbour Weighted with Mutual information between attribute/word Expression Threshold (winner takes all) Human Executing Action taken phonemes ‘spoken’

  46. Preliminary Results WP 3.2 Session 5 Kaspar has more Language Associations Düsseldorf - June 2009

  47. Outline WP 3.2 WP 3.2 Word and Phrase Acquisition Without Meaning Acquisition of Meaning via Mediated Physical Interaction Semantic based grammar induction Corpus of mapped phoneme and robot sensory/motor proprioceptions segmented utterances words, holophrases Induced syntax object/action ‘components’ available for scaffolding Language constructions WP 3.1/3.5 constructions Düsseldorf - June 2009

  48. Outline WP 3.2 WP 3.2 Word and Phrase Acquisition Without Meaning Acquisition of Meaning via Mediated Physical Interaction Semantic based grammar induction Corpus of mapped phoneme and robot sensory/motor proprioceptions segmented utterances words, holophrases Induced syntax object/action ‘components’ available for scaffolding Language constructions WP 3.1/3.5 constructions Düsseldorf - June 2009

  49. Plan for Grammar InductionAim WP 3.2 Düsseldorf - June 2009

  50. Plan for Grammar InductionSemantics to Syntax WP 3.2 Experimentally grounded semantic bootstrapping + predictive grammar induction with partial information in acquiring linguistic constructions via social interaction Düsseldorf - June 2009

More Related