An Oscillatory Correlation Approach to Scene Segmentation

1. An Oscillatory Correlation Approach to Scene Segmentation DeLiang Wang The Ohio State University

2. Outline of Presentation • Introduction • Scene segmentation problem • Oscillatory Correlation Theory • LEGION network • Oscillatory Correlation Approach to Image Segmentation • Circuit Implementation of LEGION • Summary IPAM'02

3. Scene Analysis Problem IPAM'02

4. Simplified Scenario IPAM'02

5. Temporal Correlation Theory • Feature binding is a fundamental problemn • In neuroscience • In perception • Temporal correlation as a representation • Extra degree of freedom • A plausible mechanism • See von der Malsburg’81, Milner’74, Abeles’82 IPAM'02

6. Neurophysiological Evidence • Gray & Singer (1989) IPAM'02

7. Neurophysiological Evidence - continued • Gray et al. (1989) IPAM'02

8. Oscillatory Correlation Theory IPAM'02

9. Computational Requirements forOscillatory Correlation • Need to synchronize locally coupled oscillator population • Extensive literature in theoretical physics and applied mathematics • Need to desynchronize different populations, when facing multiple objects • Critically, the above functions must be achieved very rapidly IPAM'02

10. Three Possible Ways to Reach Synchrony • Comparator model • Globally connected • Locally connected IPAM'02

11. LEGION Architecture • LEGION - Locally Excitatory Globally Inhibitory Oscillator Network (Terman & Wang’95) IPAM'02

12. Single Relaxation Oscillator With stimulus Without stimulus Typical x trace (membrane potential) IPAM'02

13. Model of a Relaxation Oscillator • Model definition • Coupling between oscillators • Where N(i) is the set of neighboring oscillators that connect to oscillator i IPAM'02

14. Model of a Relaxation Oscillator - continued • Global inhibitor IPAM'02

15. Fast Threshold Modulation for Two Oscillators Somers & Kopell’93 LBE LB RBE RB IPAM'02

16. Selective Gating for Two Oscillators LB LBI RB IPAM'02

17. Summary of Analytical Results • Definitions: A pattern is a connected region, and a block a subset of oscillators stimulated by a given pattern. The following results are established for e > 0 sufficiently small • Theorem 1. (Synchronization). The parameters of the system can be chosen so that all of the oscillators in a block synchronize. Moreover, the rate of synchronization is exponential IPAM'02

18. Summary of Analytical Results - continued • Theorem 2. (Multiple patterns) If at the beginning all the oscillators of the same block synchronize with each other and the distance between any two oscillators belonging to two different blocks is greater than some constant, then: • Synchronization within each block is maintained • The ordering of the activation among different blocks is fixed • Given a certain period of time, at least one block is in its active phase • At most one block is in its active phase at any time IPAM'02

19. Summary of Analytical Results - continued • Theorem 3. (Desynchronization) If at the beginning all the oscillators of the system lie not too far away from each other, then the condition of Theorem 2 will be satisfied after some time. Moreover, the time it takes to satisfy the condition is no greater than N cycles, where N is the number of patterns. • The entire mechanism is called Selective Gating (Terman & Wang’95) IPAM'02

20. LEGION Example Input image Successive snapshots IPAM'02

21. LEGION Example - Temporal Traces IPAM'02

22. LEGION Example: Segmentation Capacity Input image IPAM'02

23. Oscillatory Correlation Approach to Image Segmentation • Feature extraction first takes place • An image feature can be pixel intensity, depth, local image patch, texture element, optic flow, etc. • Connection weights between two neighboring oscillators are set to be proportional to feature similarity • Global inhibitor controls granularity of segmentation • Larger inhibition results in more and smaller regions • Segments pop out from LEGION in time IPAM'02

24. Image Segmentation Example Input image Segmentation result IPAM'02

25. Image Segmentation Example Input image Segmentation result IPAM'02

26. 3D MRI Image Segmentation Left: LEGION results Right: Manual results IPAM'02

27. 3D MRI Image Segmentation: a 2D View Left: Input Middle: Segmentation results Right:Further segmentation into white and gray matter IPAM'02

28. Aerial Image Analysis Extraction of hydrographic objects IPAM'02

29. Texture Segmentation Upper: input; Lower: segmentation result IPAM'02

30. Motion Segmentation IPAM'02

31. Circuit Implementation of LEGION • Circuit implementation of LEGION using PWM/PPM • PWM: Pulse-width modulationPPM: Pulse-phase modulation • By Ando et al. (2000) Pulse modulation IPAM'02

32. Circuit Implementation of a Single Oscillator Circuit of a single relaxation oscillator SPICE simulation result IPAM'02

33. Circuit Implementation of a LEGION Network Connection circuit Global inhibitor circuit IPAM'02

34. Circuit Implementation of LEGION - continued • Application to gray-level image segmentation Input image Output at two different times IPAM'02

35. LEGION on a Chip (Cosp 2000) The chip area is 6.7 mm2 (Core 3 mm2) and implements a 16x16 LEGION network IPAM'02

36. Back to Biology Gray & McCormick’96 IPAM'02

37. Summary • Mathematical analysis of the selective gating mechanism and LEGION dynamics • Both synchronous oscillations and the structure of the model are neurally plausible • Oscillatory correlation approach to scene segmentation • VLSI implementation • A neural theory for perceptual organization IPAM'02

38. Collaborators • David Terman • Erdogan Cesmeli • Ke Chen • Xiuwen Liu • Naeem Shareef IPAM'02