When the Shoe Fits: Cross-Situational Learning in Realistic Learning Environments

1. When the Shoe Fits:Cross-Situational Learning inRealistic Learning Environments Tamara N. Medina1 John Trueswell1 Jesse Snedeker2 Lila Gleitman1 1Institute for Research in Cognitive Science, University of Pennsylvania 2Department of Psychology, Harvard University

3. “A baby hears a word like “shoes”, for example, over and over again in daily life as the one constant sound in a large variety of statements. In one day you may say to him:” 1998, Third Edition, Completely Revised “That word “shoes” is the one sound which occurs in all those sentences and it is always associated with those things that go on his feet. Eventually he will associate the spoken sound with the objects and when he has made that association, he will have learned what the word “shoes” means.” • “Where are your shoes?” • “Oh, what dirty shoes!” • “Let’s take your shoes off.” • “I’ll put your shoes on.” • “Look what nice new shoes.”

4. Cross-Situational Learning “Find a set of possible meanings in each situation and intersect those sets across all situations in which a word occurs to determine the meaning for that word.” Siskind, J.M. (1996, Cognition) “It’s not so easy!” • Augustine, Locke, Quine, Gleitman, Fodor, Siskind, etc. Frame / Level of Description Animal? Dog? Terrier? Fido? Friendly? Referential Uncertainty Which referent?

5. Frame Problem Solved? • Xu & Tenenbaum (2007): learn appropriate extensions of a word via Bayesian inference (note “suspicious coincidences”) “VASH” “VASH” “VASH”

6. Reference Problem Solved? • Yu & Smith (2007): learn word-object associations in spite of “referential uncertainty” “DOON” … “VASH” … “MIPEN” … “ZANT”

7. “VASH” ??

8. “VASH” ! Goal: Explore cross-situational word learning using naturalistic settings: both the cluttered and potentially uninformative or misleading environments and these somewhat more transparent ones.

9. Overview • Adaptation of the Human Simulation Paradigm (Gillette et al., 1999) • Norming Study • Current Study • 2 measures to evaluate word learning

10. Adaptation of Human Simulation Paradigm (Gillette et al., 1999) • Selected stimuli based on results of earlier norming study: Gertner, Y., Fisher, C., Gleitman, G., Joshi, A., & Snedeker, J. (In progress). Machine implementation of a verb learning algorithm. • Large video corpus of parent-child interactions in natural settings (home, outdoors, etc.): Snedeker, J. (2001). Interactions between infants (12-15 months) and their parents in four settings. Unpublished corpus.

11. Norming Study • Identified 48 most frequently occurring content words. • Randomly selected six instances of each word. • Each instance was edited into a 40-second “vignette”. • Sound turned off. • Visual context only cue to word meaning, placing viewers in the situation of the early word learner. • Utterance of target word indicated by a BEEP. • Gertner, Y., Fisher, C., Gleitman, G., Joshi, A., & Snedeker, J. (In progress). Machine implementation of a verb learning algorithm.

12. (silence) (silence) <BEEP> 30 sec (silence) 10 sec Drawings courtesy of Emily Trueswell

13. No opportunity for cross-situational learning in norming study 8% correct 83% correct 0% correct Low Informative (<33% correct) High Informative (>50% correct) … Subject Guesses (Target Word = Shoe) Subject Guesses (Target Word = Horse) Subject Guesses (Target Word = Shoe) 90% of Vignettes = Low Informative (typical) 7% of Vignettes = High Informative (atypical) • Gertner, Y., Fisher, C., Gleitman, G., Joshi, A., & Snedeker, J. (In progress). Machine implementation of a verb learning algorithm.

14. Questions • Does the observation of multiple naturalistic learning instances generate a gradually increasing learning curve? • With regard to informativeness, given only the Low Informative vignettes, is cross-situational learning successful? Or is a High Informative instance necessary? • If learners are building an interpretation across instances, does it matter when the High Informative instance occurs?

15. Current Study Similar to norming study, BUT allows for cross-situational learning

16. (silence) Current Study Allows for Cross-Situational Learning (silence) “VASH” 30 sec (silence) 10 sec

17. Opportunity for cross-situational learning … “VASH” (Target Word = Shoe) Subject makes guess Subject rates Confidence (1 to 5) “MIPEN” (Target Word = Horse) Subject makes guess Subject rates Confidence (1 to 5) “VASH” (Target Word = Shoe) Subject makes guess Subject rates Confidence (1 to 5) Final Conjectures and Confidence Ratings for each word

18. Manipulated the distribution of informative events • For each of 8 Target nouns, there were: • 1 “High Informative” vignette (>50% of participants correct in norming study) • 4 “Low Informative” vignettes (<33% of participants correct in norming study) • 4 Filler nouns • 5 “Low Informative” vignettes • Participants assigned to one of 4 orders: • High Informative First: H-L-L-L-L • High Informative Middle: L-L-H-L-L • High Informative Last: L-L-L-L-H • High Informative Absent: L-L-L-L-L (fifth vignette is a repeat of the first)

19. Accuracy Across Vignettes

20. Accuracy Across Vignettes

21. Accuracy Across Vignettes

22. Accuracy Across Vignettes

23. Accuracy Across Vignettes

24. Interim Summary: What have we learned about learning? • Gradual learning from partially informative instances is small to nonexistent. • Successful learning depends on the presence of a High Informative instance. (Epiphany!) • Low Informative instances have a corrupting influence on later-occurring High Informative instances. (Cross-situational learning of the bad sort.)

25. Epiphany! Successful learning depends on the presence of a High Informative instance. • Explicit and immediate insight? • After using evidence from later instances? • High Informative instance provides key for interpreting later instances.

26. Confidence on Correct Guesses across Vignettes

27. Confidence on Correct Guesses across Vignettes

28. Confidence on Correct Guesses across Vignettes

29. Confidence on Correct Guesses across Vignettes

30. Implications • Shape of the word learning curve may be very different than what cross-situational learning models (e.g., Yu & Smith, 2007) have suggested: Rapid Incremental

31. Implications • Successful word learning from cross-situational observation requires the occurrence of a highly informative instance. • But must it occur first? • Greater delay between instances of a novel word: Every day is a new day. • Multiple High Informative learning instances. • Previous studies which show striking rapid word learning are such cases. • Less weight on interpretations of Low Informative instances. Logically, no!

32. Implications • A High Informative instance is the first step in successful cross-situational word learning. • Prior Low Informative instances might not be remembered over time. • Later Low Informative instances become useful (confirmatory evidence?) • Supported by rising confidence levels after a High Informative vignette.

33. Cross-situational learning does work,but only when the shoe fits.

34. Accuracy Confidence

35. Perseverance of First Guess