1 / 31

On Implementing Reservoir Computing

Benjamin Schrauwen Electronics and Information Systems Department Ghent University – Belgium December 9 2006 - NIPS 2006. On Implementing Reservoir Computing. Outline. Introduction Software: Reservoir Computing Toolbox Hardware: Digital spiking neurons Future hardware Conclusions.

bewald
Download Presentation

On Implementing Reservoir Computing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Benjamin Schrauwen Electronics and Information Systems Department Ghent University – Belgium December 9 2006 - NIPS 2006 On Implementing Reservoir Computing

  2. Outline • Introduction • Software: Reservoir Computing Toolbox • Hardware: Digital spiking neurons • Future hardware • Conclusions

  3. Introduction • LSM, ESN, BPDC, SDN, … are all the same concept, just use different nodes and topologies: Reservoir Computing • How to evaluate RC performance across node types? • Opensource MATLAB toolbox for reservoir computing research • A box of tools + examples + a large scale explorer • Because all techniques in single flow: able to focus on specific influence of: • Topology • Node type • Reservoir adaptation

  4. Reservoir Computing Toolbox • Generic way to construct topologies and weight scaling • Various node types supported: linear, TLG, tanh, fermi, spiking (LIF, synapse models, dynamic synapses) • Event based simulator for spiking neurons: ESSpiNN • Supports batching for large datasets • Currently focused on off-line training (on-line in construction) • Resampling and post-processing pipeline • Linear, ridge-regression, non-linear readout • Cross-validation, grid-search • Reservoir adaptation

  5. The RC Toolbox Input data generation Topology Adaptation ESSpiNN (CSIM) Simulation Readout pipeline Cross-val/grid

  6. The RC Toolbox: topology Connection structure Rewiring Assign weights Scaling

  7. The RC Toolbox: readout Spatial non-linearity Filtering/mean Sp./temp. non-linearity Scoring

  8. The RC Toolbox SOON http://www.elis.UGent.be/rct

  9. Hardware • Hardware advantages of RC: • Sparse/local connectivity is good • Random weights are allowed • (mild) node and network chaos can be taken advantage of • Weights are fixed or can only change locally with RA • Various HW implementations possible: • Spiking/analog/non-linear • Digital/aVLSI/…

  10. Digital spiking neurons • SNN: mathematically a more complex model than ANN • But: better implementable in hardware • No weight multiplications: table look-up • Filtering can be implemented using shifts and adds • Interconnection only single bit, and sparse communication • Asynchronous communication easily implementable

  11. Digital spiking neurons • Hardware can take advantage of parallelism • But area-speed trade-off: we don’t have to make the implementation faster than needed by the application • For trade-off: different implementations with other area-speed needed • Possible parallelisms: • Network parallelism • Neuron/synapse parallelism • Arithmetic parallelism • We implemented: • SPPA: network parallel, neuron serial, arithmetic parallel • PPSA: network parallel, neuron parallel, arithmetic serial • SPSA: network serial or parallel, neuron serial, arithmetic serial

  12. Digital spiking neurons: PPSA

  13. Digital spiking neurons: SPPA

  14. Digital spiking neurons: SPSA

  15. Results sppa spsa ppsa Number of inputs per neuron

  16. Area-speed trade-off for speech task • Speech task in hardware • LSM with 200 neurons • 12 kHz processing speed • Real-time requirement

  17. Digital spiking neurons and RCT • Topology can be exported from RCT to different HW models • Exploration in SW  export to HW for deployment • Basic HW simulation model in RCT

  18. Intermezzo: some science • Most valuable resource in hardware: long connections • Impact for RC: readout is hardest part • Solution: only do partial readout • What is performance penalty of this?

  19. Intermezzo: some science • Most valuable resource in hardware: long connections • Impact for RC: readout is hardest part • Solution: only do partial readout • What is performance penalty of this? Moore-Penrose pseudo inverse

  20. Effective parameters Intermezzo: some science • Most valuable resource in hardware: long connections • Impact for RC: readout is hardest part • Solution: only do partial readout • What is performance penalty of this? Ridge regression Tikhonov regularization

  21. Intermezzo: some science • Most valuable resource in hardware: long connections • Impact for RC: readout is hardest part • Solution: only do partial readout • What is performance penalty of this?

  22. Future: parallel event based

  23. Future: parallel event based

  24. Network communication needs to be minimized Best for networks with much local and few global connections High speed-up possible due to Event based Parallel Hardware implementation Future: parallel event based

  25. Future: CNN • Cellular Neural/Non-linear Network as reservoir • Outlook: • Very fast, analog non-linear network with only nearest-neighbor connections (128x128) • Analog computer: instruction flow possible that implements reservoir and full parallel read-out • Intrinsically random connections: corrections needed when deterministic computations on CNN • Parallel image input via CCD layer • With Samuel Xavier de Souza and Johan Suykens from KULeuven • On ACE16k_v2 chip from AnaFocus

  26. Future: photonic “Photonics is the science and technology of generating, controlling, and detecting photons, particularly in the visible light and near infra-redspectrum“ Wikipedia.org • Currently mainly focused on communication • Long standing photonicist dream: photonic computing • Problems: • Feature size at least order of wavelength (~1μm) • Implementing memory is complex • Change light with light only possible through medium: slow • Laser is intrinsically non-linear/chaotic • Problems with fabrication variances

  27. Future: photonic • Possible implementation of reservoir: photonic crystal • Semi-crystal fabricated on silicon to affect the path of light • Creates stop band where light of given bandwidth can’t exist • Light can be bend in any direction • Single crystal ‘flaw’ can be a laser

  28. Future: photonic • Idea: use photonics to implement a reservoir • Why: • Nodes (lasers) intrinsically non-linear/chaotic • Possibly very fast (ps timescale) • Full parallel readout and linear regression trivial • Random (but fixed) process variation is allowed/desired • Research project recently started together with Roel Baets and Peter Bienstman from photonics lab at Ghent University

  29. LCD LASER Future: photonic

  30. Future: photonic • Possible applications: • Full optical signal reconstruction in optical communication • Optical image processing • Very high speed signal processing • Questions/problems: • Harness laser chaos or use it to our advantage • Information in light in multiple physical properties: energy, polarisation, EM field, …

  31. Conclusions • The reservoir computing concept is very suited for hardware implementation • … or no … much hardware is very suited to be used as a reservoir

More Related