A Dynamic Learning Model For Categorizing Words Using Frames

1. A Dynamic Learning Model For Categorizing Words Using Frames Hao Wang, Toben Mintz Department of Psychology University of Southern California

2. The Problem of Learning Syntactical Categories • Grammar includes manipulations of lexical items based on their syntactical categories. • Learning syntactical categories are fundamental to the acquisition of language.

3. The Problem of Learning Syntactical Categories • Nativist approach • Children are innately endowed with the possible syntactical categories. • How to map a lexical item to its syntactical category or categories? • Empirical approach • Children have to figure out the syntactical categories in their target language, and assign categories to lexical items. • There is no or little help from syntactical constraints.

4. Approaches Based on Semantic Categories • Grammatical Categories correspond to Semantic/Conceptual Categories(Macnamara, 1972; Bowerman, 1973; Bates & MacWhinney, 1979; Pinker, 1984)object  noun action  verb • But what about • action, noise, love • to think, to know(Maratsos & Chalkley, 1980)

5. Grammatical Categories from Distributional Analyses • Structural LinguisticsGrammatical categories defined by similarities of word patterning (Bloomfield , 1933; Harris, 1951) • Maratsos & Chalkley (1980): Distributional learning theory • lexical co-occurrence patterns • (and morphology and semantics) • the cat is on the mat • cat, mat

6. Grammatical Categories from Distributional Analyses • Patterns across whole utterances(Cartwright & Brent, 1997) • My cat meowed. • Your dog slept. • Det N X/Y. • Bigram co-occurrence patterns(Mintz, Newport, & Bever, 1995, 2002; Redington, Chater & Finch, 1998) • the cat is on the mat

7. Frequent Frames (Mintz, 2003) • Frames are defined as “two jointly occurring words with one word intervening”. • “would you put the cans back ?” • “you get the nuts .” • “you take the chair back . • “you read the story to Mommy .” • Frame: you_X_the

8. Sensitivity to Frame-like Units • Frames lead to categorization in adults (Mintz, 2002) • Fifteen-month-olds are sensitive to frame-like sequences (Gómez & Maye, 2005)

9. Distributional Analyses Using Frequent Frames (Mintz, 2003) • Six corpora from CHILDES (MacWhinney, 2000). • Analyzed utterances to children under 2;6. • Accuracy results averaged overall corpora.

10. Limitation of the Frequent Frame Analyses • Requires two passes through the corpus • Step 1, identify the frequent frames by tallying the frame frequency. • Step 2, categorizing words using those frames. • Tracks the frequency of all frames • E.g., approximately 15000 frame types in one of the corpora in Mintz (2003).

11. Goal of current study • Provides a psychological plausible model of word categorization • Children possesses limited memory and cognitive capacity. • Human memory is imperfect. • Children may not be able to track all the frames he/she has encountered.

12. Features of current model • It processes input and updates the categorization frames dynamically. • Frame is associated with and ranked by a activation value. • It has a limited memory buffer for frames. • Only stores the most activated 150 frames. • It implements a forgetting function on the memory. • After processed a new frame, the activation of all frames in the memory decreased by 0.0075.

13. Child Input Corpora • Six corpora from CHILDES (MacWhinney, 2000). • Analyzed utterances to children under 2;6. • Peter (Bloom, Hood, Lightbown, 1974; Bloom, Lightbown, Hood, 1975)Eve(Brown, 1973)Nina (Suppes, 1974)Naomi(Sachs, 1983)Anne(Theakston, Lieven, Pine, Rowland, 2001)Aran(Theakston et al., 2001) • Mean Utterance/Child: ~17,200 • MIN: 6,950 ; MAX: 20,857

14. Procedure • The child-directed utterances from each corpus was processed individually • Utterances were presented to the model in the order of appearance in the corpus • Each utterance was segmented into frames • “you read the story to Mommy” • you read the • read the story • the story to • story to Mommy

15. Procedure continued… • you read the • read the story • the story to • story to Mommy

16. Procedure continued… • The memory buffer only stores most activated 150 frames. • It becomes full very quickly after processing several utterances.

17. Procedure continued… • “you put the” • Frame: you_X_the • Look up you_X_the frame in the memory • Increase the activation of you_X_the frame by 1 • Re-rank the memory by activation

18. Procedure continued… • “you have a” • Frame: you_X_a • Look up you_X_a frame in the memory • story_X_Mommy < 1 • Removestory_X_Mommy • Add you_X_a to memory, set the activation to 1 • Re-rank the memory by activation

19. Procedure continued… • A new frame not in memory • The activation of all frames in memory are greater than 1 • There is no change to the memory.

20. Evaluating Model Performance • Hit: two words from the same linguistic category grouped together • False Alarm: two words from different linguistic categories grouped together • Upper bound of 1

21. V V V ADV V V Hits: 10 False Alarms: 5 Accuracy: Accuracy Example

22. Noun, pronoun Verb, Aux., Copula Adjective Preposition Adverb Determiner Wh-word Negation -- “not” Conjunction Interjection Ten Categories for Accuracy

23. Averaged accuracy across 6 corpora

24. The Development of Accuracy • Accuracy are very high and stable in the entire process

25. Compare to Frequent Frames • After processing about half of the corpus, 70% of frequent frames are in the most activated 45 frames in memory.

26. Memory of Final Step of Eve Corpus

27. Stability of Frames in Memory • Big changes of frames in memory in early stage, but become stable after processing 10% of the corpus

28. Summary • After processed the entire corpus, the learning algorithm has identified almost all of the frequent frames by highest activation. • Consequently, high accuracy of word categorization is achieved. • After processing fewer than half of the utterances, the 45 most activated frames included approximately 70% of frequent frames.

29. Summary • Frames are a robust cue for categorizing words. • With limited and imperfect memory, the learning algorithm can identify most frequent frames after processing a relatively small number of utterances. Thus yield a high accuracy of word categorization.