Ad multos annos Joos Vandewalle Frameless Wave Computing

1. Ad multos annosJoos VandewalleFrameless Wave Computing Tamás Roska András Horváth and Miklós Koller Pázmány P. Catholic University, Budapest

2. Outline Cellular Wave Computing Frameless spatial-temporal computing Activation controlled frameless computing Delayed template frameless computing Outlook

3. Cellular Wave Computing Spatial-temporal waves combined: Input wave Self wave Activation wave (e.g. stroboscopic effect) boundary wave in a CNN Universal Machine with non-standard CNN dynamics

4. The computational model Wehavethreewavedynamicsevolvingtogether: • thedynamics of thespatial-temporal input flow (u) • theself-dynamics of thecomputingcellulararray (xdefinedbyF) • thedynamics of theactivelight-sources (vdefinedbyG1G2) Weareinterestedintheirinteractionintwocases: ‘independentactivation’ case ‘adaptiveactivation’ case u: two-dimensional input-flow x: two-dimensional computation-flow (inner state of the cells) v: two-dimensional flow defining the activation strength of the light-sources

5. Frameless spatial-temporal computing A.Solving an NP hardproblemwith a CellularWave Computer withsparsenonlocalconnection inonesiglewave B. Detectingspatial-temporalevents For A: M. Ercsey-Ravasz, T. Roska, Z. Néda, „Cellular Neural Networks for NP-hard optimization”,EURASIPJournal on Advances in Signal Processing, Special issue: CNN Technology for Spatio-temporal Signal Processing, doi: 10.1155/2009/646975, 2009. M. Ercsey-Ravasz, Z. Toroczkai, "Optimization Hardness as Transient Chaos in an Analog Approach to Constraint Satisfaction", Nature Physics 7, 966 (2011) arxiv:1208.0526 B. Molnár, Z. Toroczkai, M. Ercsey-Ravasz, "Continuous-time Neural Networks Without Local Traps for Solving Boolean Satisfiability", CNNA 2012, Torino, Italy (2012) doi:10.1109/CNNA.2012.6331411

6. Problem statement for an NP complete problem Solution of the K-SAT problem The KSAT problem is NP complete and (widely used in the field of optimization) For our prototype problem we have 10 state variables (xi) and 35 constraints (Ci) each of them containing three state varaibles. A constraint can be writen in the following form: The problem is solved if each of the constarints are satisfied in the formula.

7. Problem statement The example problem we have investigated can be written in the following form:

8. The Dynamics This heterogenous CNN network contains two type of cells (one for the state and one for the constraints) with state variables s(t) and a(t) Where:

9. The architecture of the network

10. Transient behaviour The only fixed point of this system is the solution of the logical formula The system converges to the solution from every initial state

11. Transient behaviour The only fixed point of this system is the solution of the logical formula The system converges to the solution from every initial state

12. Transient behaviour The state transition of all 10 state variables 1.5 means true and -1.5 means false

13. B. Spatial-temporal event detection No framesinbiology – multichannelvisual „computing” – starting inthe retina Dynamicspatial-temporalmotifs Examples:looming, horizontal and verticalspeed „calculatedalreadyinthe retina, like an optical flow Combining a fewwavechannels Registration of threemodalitiesinsuperiorcolliculus (vision, audio, touch)

14. Activation controlled frameless computing Use an unstable spatial-temporal self wave Use a constant activation dynamics Apply the reflected wave as an input The output dynamics becomes stable in time and codes the terrain property

15. The general scope: • the aim: to detect spatio-temporal features or events • the computational environment:a Cellular Wave Computer architecture, where the computations are done by locally propagating waves. The active light of the sensors can be adaptively tuned in spatial-temporal rule. • system setup:computational method: software simulationhardware framework: infrared lighting and sensorarray • spatia-temporal algorithms • measurement and simulation results

16. System setup • Sensor array: • to collect the input-data from the scene • A) 8x8 active LED array with receiver photo sensors • B) control- and readout-circuits • Simulator: • to process the raw measurement data in the afore mentioned computational model • state-equations: both explicit Euler and RK-45 methods to approximate • software framework: c++, MATLAB

17. The particular example: The task: to detect a specific terrain feature (a bump or a valley) which has bigger size than the sensorarray itself. The key step: to apply the whole image flow on the input, instead of the separately captured frames (frameless detection).

18. The computational model Wehavethreewavedynamicsevolvingtogether: • thedynamics of thespatial-temporal input flow (u) • theself-dynamics of thecomputingcellulararray (xdefinedbyF) • thedynamics of theactivelight-sources (vdefinedbyG1G2) We areinterestedintheirinteractionintwocases: ‘independentactivation’ case ‘adaptiveactivation’ case u: two-dimensional input-flow x: two-dimensional computation-flow (inner state of the cells) v: two-dimensional flow defining the activation strength of the light-sources

19. Template-program of the computing array An asymmetric template with few non-zero element: • boundary condition: zero-flux • size of the computational array: 8 x 8 cells • computational model: Chua-Yang • Please consider the qualitative effect of the vertical coupling (from I. Petrás; size: 41 x 23; FSR-model):

21. Delayed template frameless computing Motivation Delays-timeconstantdifferencesinsinglesynapses Drasticdelaydifferencesbetweenelectrical and chemicalsynapses Delaydifferencesbetweenchannels

22. The CNN Universal Machine architecture is capable of detecting structures (spatial characteristic) by simple templates (operations) in a simple and elegant way Detection of different spatial frequencies Grayscale input image Binary output image representing the different structures

23. Detection of spatial frequencies in practice

24. Frameless detection Nyquist-Shanon sampling theorem: If a function x(t) contains no frequencies higher than B hertz, it is completely determined by giving its ordinates at a series of points spaced 1/(2B) seconds apart. Temporal detection: almost always frame based temporal changes are the differences between the frames, not the real dynamics. The detection of a spatial-temporal event can be easier in the continuous time-domain if the criteria above are not fulfilled.

25. Frameless detection It is difficult to identify the highest frequency in some dynamics: Tsunami If the event is fast the (sampling and processing) detection has to be two times faster.

26. Spatial-temporal detection in the retina Continuous analogue processing in the retina Our retina (brain) handles dynamics, not image sequences: Low frame-rate movies, animations

27. -Complex task -Computationally expensive with regular architectures -Simply done in the retina - Done in an analogue, continuous way Example: Looming detection

28. Modeling the response of the ganglion cells with a CNN chip Looming

29. Delay type CNN template Not only the coupling strengths, but also coupling delays are defined. Extension of regular CNN dynamics, the delay is defined as the delay between the elements CNN with implicit memory B and W templates design

30. Diagonal movement detection Input video Output video

31. Diagonal movement detection Excites the cells temporarily: the time of excitation is controlled by the template

32. Diagonal movement detection The excited cells remain excited (in this case black). Detects the trajectory of an object.

33. Detection of a given trajectory The aim is to identify the object moving up in the input-flow This task can be solved by a single delayed-cnn template Input Video Output video

34. Detection of a given trajectory The previous result can be extended to identify objects moving along a given trajectory with a given speed Input Video Output video

35. Delayed edge detection: Identification of movement speed and direction The dark edge will appear where we can detect an edge on the current input flow, while the bright edge will appearwhere the edge was τe time ago. This can be used to detect the speed and the direction of the moving object. Input Video Output video

36. Outlook Develop a design methodology for spatial-temporal computing without frames Develop a special physical mplementation framework Towards a 3-layer vertically integrated system Learning from neurobiological prototypes