The Time Dimension for Scene Analysis

1. The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA

2. Presentation outline • Introduction • Scene analysis and temporal correlation theory • Oscillatory Correlation • LEGION network • Oscillatory Correlation Approach to Scene Analysis • Image segmentation • Object selection • Cocktail party problem • Concluding remarks

3. Scene analysis problem

4. Binding problem • Feature binding (integration) is a fundamental problem in neuroscience and perception (and perceptrons) Binding problem in Rosenblatt’s perceptrons

5. Temporal correlation theory • Temporal correlation theory proposes a solution to the nervous integration problem (von der Malsburg’81; also Milnor’74) • Application to cocktail party processing (von der Malsburg & Schneider’86)

6. Physiological evidence (Gray et al.’89)

7. Oscillatory correlation theory • Oscillators represent feature detectors • Binding is encoded by synchrony within an oscillator assembly and desynchrony between different assemblies

8. Computational requirements • Need to synchronize locally coupled oscillator population • Need to desynchronize different populations, when facing multiple objects • Synchrony and desynchrony mustbe achieved rapidly

9. LEGION architecture • LEGION - Locally Excitatory Globally Inhibitory Oscillator Network (Terman & Wang’95)

10. Relaxation oscillator as building block With stimulus Without stimulus Typical x trace (membrane potential)

11. Analytical results • Theorem 1. (Synchronization). The oscillators in a connected block synchronize at an exponential rate • Theorem 2. (Multiple patterns) If at the beginning all the oscillators of the same block synchronize and different blocks desynchronize, then synchrony within each block and the ordering of activations among different blocks are maintained • Theorem 3. (Desynchronization) If at the beginning all the oscillators of the system lie not too far away from each other, then the condition of Theorem 2 will be satisfied after some time. Moreover, the time it takes to satisfy the condition is no greater than N cycles, where N is the number of blocks

12. Connectedness problem • Minsky-Papert connectedness problem is a long-standing problem in perceptron learning • The problem exposes fundamental limitations of supervised learning, and illustrates the importance of proper representations

13. Connectedness problem: LEGION solution • Basic idea: Synchronization within a connected pattern and desynchronization between different ones

14. Presentation outline • Introduction • Scene analysis and temporal correlation theory • Oscillatory Correlation • LEGION network • Oscillatory Correlation Approach to Scene Analysis • Image segmentation • Object selection • Cocktail party problem • Concluding remarks

15. Oscillatory correlation approach to scene segmentation • Feature extraction first takes place • An visual feature can be pixel intensity, depth, local image patch, texture element, optic flow, etc. • An auditory feature can be a pure tone, amplitude and frequency modulation, onset, harmonicity, etc. • Connection weights between neighboring oscillators are set to be proportional to feature similarity • Global inhibitor controls granularity of segmentation • Larger inhibition results in more and smaller regions • Segments pop out from LEGION in time

16. Image segmentation example: Demo Input image

17. Image segmentation example Input image Segmentation result

18. Object selection • The slow inhibitor keeps trace of each pattern, which can be overcome by only more salient (larger) patterns • Unlike traditional winner-take-all dynamics, selection (competition) takes place at the object level • Consistent with object-based attention theory • Binding precedes attention, rather than attention precedes binding (Treisman & Gelade’80)

19. Results of object selection LEGION output Selection output Input image Input LEGION segmentation Selection

20. Cocktail party problem • In a natural environment, target speech is usually corrupted by acoustic interference, creating a speech segregation problem • Popularly known as cocktail-party problem (Cherry’53);alsoball-room problem (Helmholtz, 1863) • Human listeners organize sound in a perceptual process called auditory scene analysis (Bregman’90) • Auditory scene analysis (ASA) takes place in two conceptual stages: • Segmentation. Decompose the acoustic signal into ‘sensory elements’ (segments) • Grouping. Combine segments into groups, so that segments in the same group likely originate from the same sound source

21. Oscillatory correlation for ASA (Wang & Brown’99) Frequency

22. Auditory periphery: Cochleagram • Cochleagram representation of the utterance: “Why were you all weary?” mixed with phone ringing

23. Grouping layer: Example • Two streams emerge from the group layer • Foreground: left (original mixture ) • Background: right • More recent results (Hu & Wang’04):

24. Back to physiology • Chattering cells recorded by Gray & McCormick’96 • Burst oscillations are best modeled by relaxation oscillators

25. Versatility and time dimension • The principle of universality: “Give me a concrete problem and I will devise a network that solves it.” (von der Malsburg’99) • It characterizes artificial intelligence • The principle of versatility: “Given the network, learn to cope with situations and problems as they arise.” (von der Malsburg’99) • It characterizes natural intelligence • Time dimension is necessary for versatility • Flexible and infinitely extensible • Irreplaceable by spatial organization

26. Conclusion • Advances in dynamical analysis overcome computational obstacles of oscillatory correlation theory • Major progress is made towards solving the scene analysis problem • From Hebb’s cell assemblies to von der Malsburg’s correlation theory, time is an indispensable dimension for scene analysis