1 / 26

The Time Dimension for Scene Analysis

The Time Dimension for Scene Analysis. DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA. Presentation outline. Introduction Scene analysis and temporal correlation theory Oscillatory Correlation LEGION network Oscillatory Correlation Approach to Scene Analysis

garland
Download Presentation

The Time Dimension for Scene Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA

  2. Presentation outline • Introduction • Scene analysis and temporal correlation theory • Oscillatory Correlation • LEGION network • Oscillatory Correlation Approach to Scene Analysis • Image segmentation • Object selection • Cocktail party problem • Concluding remarks

  3. Scene analysis problem

  4. Binding problem • Feature binding (integration) is a fundamental problem in neuroscience and perception (and perceptrons) Binding problem in Rosenblatt’s perceptrons

  5. Temporal correlation theory • Temporal correlation theory proposes a solution to the nervous integration problem (von der Malsburg’81; also Milnor’74) • Application to cocktail party processing (von der Malsburg & Schneider’86)

  6. Physiological evidence (Gray et al.’89)

  7. Oscillatory correlation theory • Oscillators represent feature detectors • Binding is encoded by synchrony within an oscillator assembly and desynchrony between different assemblies

  8. Computational requirements • Need to synchronize locally coupled oscillator population • Need to desynchronize different populations, when facing multiple objects • Synchrony and desynchrony mustbe achieved rapidly

  9. LEGION architecture • LEGION - Locally Excitatory Globally Inhibitory Oscillator Network (Terman & Wang’95)

  10. Relaxation oscillator as building block With stimulus Without stimulus Typical x trace (membrane potential)

  11. Analytical results • Theorem 1. (Synchronization). The oscillators in a connected block synchronize at an exponential rate • Theorem 2. (Multiple patterns) If at the beginning all the oscillators of the same block synchronize and different blocks desynchronize, then synchrony within each block and the ordering of activations among different blocks are maintained • Theorem 3. (Desynchronization) If at the beginning all the oscillators of the system lie not too far away from each other, then the condition of Theorem 2 will be satisfied after some time. Moreover, the time it takes to satisfy the condition is no greater than N cycles, where N is the number of blocks

  12. Connectedness problem • Minsky-Papert connectedness problem is a long-standing problem in perceptron learning • The problem exposes fundamental limitations of supervised learning, and illustrates the importance of proper representations

  13. Connectedness problem: LEGION solution • Basic idea: Synchronization within a connected pattern and desynchronization between different ones

  14. Presentation outline • Introduction • Scene analysis and temporal correlation theory • Oscillatory Correlation • LEGION network • Oscillatory Correlation Approach to Scene Analysis • Image segmentation • Object selection • Cocktail party problem • Concluding remarks

  15. Oscillatory correlation approach to scene segmentation • Feature extraction first takes place • An visual feature can be pixel intensity, depth, local image patch, texture element, optic flow, etc. • An auditory feature can be a pure tone, amplitude and frequency modulation, onset, harmonicity, etc. • Connection weights between neighboring oscillators are set to be proportional to feature similarity • Global inhibitor controls granularity of segmentation • Larger inhibition results in more and smaller regions • Segments pop out from LEGION in time

  16. Image segmentation example: Demo Input image

  17. Image segmentation example Input image Segmentation result

  18. Object selection • The slow inhibitor keeps trace of each pattern, which can be overcome by only more salient (larger) patterns • Unlike traditional winner-take-all dynamics, selection (competition) takes place at the object level • Consistent with object-based attention theory • Binding precedes attention, rather than attention precedes binding (Treisman & Gelade’80)

  19. Results of object selection LEGION output Selection output Input image Input LEGION segmentation Selection

  20. Cocktail party problem • In a natural environment, target speech is usually corrupted by acoustic interference, creating a speech segregation problem • Popularly known as cocktail-party problem (Cherry’53);alsoball-room problem (Helmholtz, 1863) • Human listeners organize sound in a perceptual process called auditory scene analysis (Bregman’90) • Auditory scene analysis (ASA) takes place in two conceptual stages: • Segmentation. Decompose the acoustic signal into ‘sensory elements’ (segments) • Grouping. Combine segments into groups, so that segments in the same group likely originate from the same sound source

  21. Oscillatory correlation for ASA (Wang & Brown’99) Frequency

  22. Auditory periphery: Cochleagram • Cochleagram representation of the utterance: “Why were you all weary?” mixed with phone ringing

  23. Grouping layer: Example • Two streams emerge from the group layer • Foreground: left (original mixture ) • Background: right • More recent results (Hu & Wang’04):

  24. Back to physiology • Chattering cells recorded by Gray & McCormick’96 • Burst oscillations are best modeled by relaxation oscillators

  25. Versatility and time dimension • The principle of universality: “Give me a concrete problem and I will devise a network that solves it.” (von der Malsburg’99) • It characterizes artificial intelligence • The principle of versatility: “Given the network, learn to cope with situations and problems as they arise.” (von der Malsburg’99) • It characterizes natural intelligence • Time dimension is necessary for versatility • Flexible and infinitely extensible • Irreplaceable by spatial organization

  26. Conclusion • Advances in dynamical analysis overcome computational obstacles of oscillatory correlation theory • Major progress is made towards solving the scene analysis problem • From Hebb’s cell assemblies to von der Malsburg’s correlation theory, time is an indispensable dimension for scene analysis

More Related