Semantic Cognition: A Parallel Distributed Processing Approach

1. Semantic Cognition:A Parallel Distributed ProcessingApproach James L. McClelland Center for the Neural Basis of CognitionandDepartments of Psychology and Computer Science, Carnegie Mellon Timothy T. Rogers Center for the Neural Basis of Cognition and nowMRC Cognition and Brain Sciences Unit, UK CNBC A Joint Project of Carnegie Mellon and the University of Pittsburgh

2. Approaches to Semantic Cognition • Concepts and their Properties • Is Socrates Mortal? • Hierarchical Propositional Models • Quillian, 1968; Collins and Quillian, 1969 • Theory-Theory and Related Approaches • Murphy and Medin, 1985; Gopnik and Wellman, 1994; Keil, 1991; Carey, 1985 • Parallel Distributed Processing • Hinton, 1981; Rumelhart and Todd, 1993; McRae, De Sa, and Seidenberg, 1997

3. Plan for This Talk • Compare a distributed, connectionist model that learns from exposure to information about the relations between concepts and their properties to the ‘classical’ Hierarchical Propositional Approach. • Show how the model accounts for a set of phenomena that have been introduced in support of ‘Theory Theory’ • Conclude with a brief consideration of where we are in the development of a theory of semantic cognition.

4. Initial Motivations for the Model • Provide a connectionist alternative to traditional hierarchical propositional models of conceptual knowledge representation. • Account for development of conceptual knowledge as a gradual process involving progressive differentiation.

5. Quillian’sHierarchicalPropositional Model

6. The Parallel Distributed Processing Approach • Processing occurs via propagation of activation among simple processing units. • Knowledge is stored in the weights on connections between the simple processing units. • Propositions are not stored directly. • The ability to produce complete propositions from partial probes arises through the activation process, based on the knowledge stored in the weights. • Learning occurs via adjustment of the connections. • Semantic knowledge is gradually acquired through repeated exposure, mirroring the gradual nature of cognitive development.

7. Activation The Rumelhart Model

8. The Training Data: All propositions true of items at the bottom levelof the tree, e.g.: Robin can {grow, move, fly}

9. Error

10. Any Questions?

11. Differentiation in Development

13. The Rumelhart Model

16. Trajectories of Representations Through State Space over Time

17. Any Questions?

18. Tenets of Theory Theory • Intuitive domain knowledge of relations between items and their properties is use to decide: • which categories are ‘good’ ones and which properties are central to particular concepts • how properties should be generalized from one category to another • Many proponents suggest that some theory-like knowledge (or constraints on acquiring such knowledge) must be available ‘initially’. • Others emphasize reorganization of knowledge through experience, but provide very little discussion of how experience leads to reorganization.

19. Three Phenomena Supporting Theory Theory • Category goodness and feature importance. • Differential importance of properties in different concepts. • Reorganization of conceptual knowledge.

20. Effects of Coherent Variation of Properties on Learning • Attributes that vary together create the concepts that populate the taxonomic hierarchy, and determine which properties are central to a given concept. • Where sets of attributes vary together, they exert a strong effect on learning. • Items with co-varying properties stay together through semantic space and form the clusters corresponding to super-ordinate concepts. • Arbitrary properties (those that do not co-vary with others) are very difficult to learn, even when frequency is controlled. • They control a late stage of differentiation in which individual items within clusters become conceptually distinct.

21. CoherenceTrainingEnvironment Properties Coherent Incoherent 12345678910111213141516 Items No Category Labels are Provided!

22. Effect of Coherence on Learning

23. Effect of Coherence on Representation

24. Extended modelfor remaining simulations

25. Progressive Differentiation of Category Structure Without Names 300 Epochs 1200 Epochs plants | animals plants | animals

26. Any Questions?

27. Three Phenomena Supporting Theory Theory • Category goodness and feature importance. • Differential importance of properties in different concepts. • Reorganization of conceptual knowledge.

28. Differential Importance (Marcario, 1991) • 3-4 yr old children see a puppet and are told he likes to eat, or play with, a certain object (e.g., top object at right) • Children then must choose another one that will “be the same kind of thing to eat” or that will be “the same kind of thing to play with”. • In the first case they tend to choose the object with the same color. • In the second case they will tend to choose the object with the same shape.

29. Adjustments to Training Environment • To address this we added some new property units and created clear cases of feature-dependencies in the model: • Among the plants: • All trees are large • All flowers are small • Either can be bright or dull • Among the animals: • All birds are bright • All fish are dull • Either can be small or large • Though partially counter-factual, these assignments allow us to explore domain specificity of feature dependencies in the model.

30. Testing Feature Importance • After partial learning, model is shown eight test objects: • Four “Animals”: • All have skin • All combinations of bright/dull and large/small • Four “Plants”: • All have roots • All combinations of bright/dull and large/small • Representations are generated by usingback-propagation, training the item-to-representation weights only. • Representations are then compared to see which animals are treated as most similar, and also which plants are treated as most similar.

31. (One unit is addedfor each test object)

32. Similarities of Obtained Representations Brightness is relevant for Animals Size is relevant for Plants

33. Differential Feature Importance • The simulation suggests that domain-general learning mechanisms can learn that different features are important for different concepts. • The network has acquired domain-specific knowledge of just the sort theory theorists claim children know about concepts. • It does so from the distributions of properties of concepts, without the aid of initial domain knowledge.

34. Phenomena Supporting Theory Theory • Category goodness and feature importance. • Differential importance of properties in different concepts. • Reorganization of conceptual knowledge.

35. Conceptual Reorganization (Carey, 1985) • Carey demonstrates that young children ‘discover’ the unity of plants and animals as living things only around the age of 10. • She suggests that the emergence of the concept of living thing coalesces from assimilation of different kinds of information, including: • Need for nutrients • What it means to be dead vs. alive • Reproductive properties

36. Conceptual Reorganization in the Model • Our simulation model provides a vehicle for exploring how conceptual reorganization can occur. • The model is capable of forming initial representations based on superficial appearances • Later, it can discover shared structure that cuts across several different relational contexts, and use the emergent common structure as a basis for a deeper organization.

37. Reorganization Simulation • We consider the coalescence of the superordinate categories plant and animal, in a situation where the training data initially supports a superficial organization based on appearance properties. • In each training pattern, the input is an item and one of the three relations: ISA, HAS, or CAN. • The target includes all of the superficial appearance properties (IS properties) plus the properties appropriate for the relation. • The model quickly learns representations that capture the superficial IS properties. • Later, it reorganizes these representations as it learns the relation-dependent properties.

38. Organization of Conceptual Knowledge at Different Points in Development

39. Phenomena Supporting Theory Theory • Category goodness and feature importance. • Differential importance of properties in different concepts. • Reorganization of conceptual knowledge.

40. Summary • The model exhibits several characteristics of human cognition that motivated the appeal to naïve domain theories. • The model does these things simply by adjusting the weights on connections among simple processing units, and by propagating signals backward and forward through these weighted connections.

41. Relationship between the Model and Theory Theory • There is a sense in which the knowledge in the connections plays the role of the informal domain theories advocated by theory theorists, and one might be tempted to suggest that the model is ‘merely an implementation’ of the theory theory. • However, it differs from the theory theory in several very important ways: • It provides explicit mechanisms indicating how domain knowledge influences semantic cognition. • The PDP model avoids bringing in unwanted aspects of what we generally mean by ‘theory’ • It offers a learning process that provides a means for the acquisition of such knowledge. • It demonstrates that some of the sorts of constraints theory-theorists have suggested might be innate can in fact be acquired from experience.

42. Conclusions • In our view the ‘theory theory’ should be viewed as more of a pre-theoretical heuristic than an actual theory of semantic cognition. • Our own proposals, built on Hinton’s and Rumelhart’s, are far from the final word, and do not constitute a complete theory at this point. • Our hope is that they will contribute, along with the work of many others, to the ongoing development of an adequate and complete theory of semantic cognition.