“Perceptual Symbol Systems”

1. “Perceptual Symbol Systems” L. W. Barsalou (1999), Brain and Behavior Science, 22, 577-660. An overview of Barsalou’s Perceptual Symbol System theory James Sulzen June 6, 2001 Psychology 264 Gordon Bower Spring 2001

2. Terminology • P-state - “Perceptual state” • A combination of multimodal percepts constituting an experience or some aspect of perception of the real world. • P-sym (or p-symbol) - “Perceptual symbol” • Recalled or constructed subset of p-states which symbolically stands for a referent of some sort • PSS - Perceptual Symbol System • System of symbols and processes which operate on them to produce cognitive processes. • Frame • Simulation

3. Amodal Symbol Systems • Amodal systems transduce p-states into amodal equivalents.

4. Amodal vs. Modal Issues • Evidence: Little/no direct evidence for amodal systems • Neuroscience & Psychology: Much evidence for modal processing • Awkwardness: Certain computations are very amodally awkward (i.e. spatio-temporal) • Transduction and symbol grounding: Amodal systems are essentially arbitrary. (Grounding/associating amodal representations back to perceptual ones just ultimately rates to make the amodal systems redundant.) • Too much power: Amodal systems ultimately are too powerful: They can explain anything. Modal systems have a priori limitations (modal ones do not). Modal systems are open to falsifiability and provocative hypothesis

5. PSS Core Properties • P-syms - Stand in as referents • Frames - These combine p-syms & frames with relationships • Simulators - dynamically bind other elements • Language - Can stand for or drive other elements

6. PSS Core Properties • P-syms • Have a neural representation / substrate • Are schematic representations of p-states (see Fig. 1) • Arise from repeated exposure [neuronal recruitment / exclusion] • Are inherently multimodal (including proprioception and introspection) • Selective attention operates to extract p-sym attributes • Frames • Organize p-syms (i.e., define primitive relationships - up/down, in/out) • Combine together: • Predicates • Attribute-value bindings • Constraints • Recursion

7. Multimodal Organization of Knowledge Sensory registers V- vision G - Gustatory H - Haptic K - Kinesthetic & proprioceptive O - Olfactory A - Auditory L - Language S - Spatial E - Emotional Other systems certainly exist - Each modality can be thought of as a representational system - Each provides certain affordances

8. Frames Establishing an initial frame for car after processing a first instance. Evolution after processing a second instance. Creating a simulation of second instance from frame in B

9. Simulators • Simulators • [This is an area requiring more development in the theory] • Frames a set of p-syms, frames, & simulators to create higher-level structures • Temporally dynamic integration of other elements • Can be dynamically constructed, modified, componentized, and executed • Comparable to mental models, schema, concepts, etc. • Simulation • This is the execution of a simulator • Are always sketchy and incomplete - are never veridical • Idealization occurs (i.e., Gestaltist principles apply) • Example: Categorization - “if a category simulator can produce a satisfactory simulation of a perceived entity, then the entity belongs in the category” (p. 587) • Derived properties of simulators • Productivity: Can be combinatorially and recursively combined • Propositions: Can be bound to individual entities [framing?] • Variability: Implement variable embodiment • Abstraction: Can combine physical and introspective events to represent abstractions

10. Simulator Productivity Object categories Spatial relationships Combinatorialproductivity Recursiveproductivity

11. Prop- ositions Proposition representation (“the balloon is above the cloud”) Complex hierarchicalproposition (balloon above cloud) Alternative proposition (balloon below cloud) simulators simulations Perceived situations

12. Language • Can be linked any other element (p-sym, frame, simulator) • Can be used to construct and control simulators • [Language is probably the amodal basis of cognition and memory?]

13. PSS Symbol Manipulation • Barsalou essentially shows that p-syms, frames, simulators, etc can: • Be composed, associated, combined, subtracted, and so on • They constitute a symbol manipulation system • He does not demonstrate a formal correspondence between his PSS and an amodal symbol system(such as a semantic knowledge network) • Such a demonstration would constitute a form of completeness proof • Language as a representational system [and can serve the role of an amodal symbol system]

14. Abstractions • Metaphor (anger => liquid exploding out of a container) • Not adequate in & of itself to represent all abstractions • Abstraction are constructed from three main elements • Framing: Are framed against the background of a simulation • Selectivity: Selective attention highlights the core content of the concept in the context of the simulation • Introspective symbols (i-syms): These are central to representing abstract concepts (i-syms are the internal sense of an experience or internal perception) • Methodology for identifying abstractions • Find a frame that characterizes the abstraction • Identify seeming p-syms and i-syms • Identify the focal element that characterizes the abstraction

15. Truth & Falsity Mapping succeeds:“truth” • Simulated event seq. frames the concept • The abstraction is only a focal part of the simulation (i.e., the outcome that construes the concept) • Introspective symbols are central to the construing of the meaning of the concept • After many mappings, a simulator develops for truth Mapping fails:“falsity” After many mappings abstractionsimulatorbecomes established

16. Anger • Anger involves • Appraisal of an initiating event, and that the agent’s goal is blocked by the event • Intense affective states • Behavioral responses (i.e., disapproval, revenge, and redirecting goals) • Core component: Blocked goal • A goal is a simulated state that the agent desires to achieve • A blocked goal is a failed mapping when it is expected to successfully map • Similar to falsity • A simulated situation fails to map to a perceived situation • Except that affective states and behavioral responses are also associated with anger • Lie • A statement induces a simulation purported to be true that is actually not(i.e., simulation is negative in liar’s simulation, but false in the deceived’s) (P. 602)

17. Disjunction Attempted reconstruction Partial recall Original event

18. Transformations

19. Ad hocCategories