IEEE Santa Clara ComSoc/CAS Weekend Workshop – Event-based analog sensing

1. IEEE Santa Clara ComSoc/CAS Weekend Workshop – Event-based analog sensing Theodore Yu theodore.yu@ti.com Texas Instruments – Kilby Labs, Silicon Valley Labs September 29, 2012

2. Living in an analog world • The world is analog • Many different levels to sense • Sight, sound, touch, taste, smell • Analog interfaces are uniquely suited for each environment • Increasingly, we turn to machines to help interpret the world for us • Interface through sensors and actuators with computation being performed in digital machines • e.g. microprocessors, cellphones, CPUs, etc. • Digital computation is robust, easily configurable, and widespread

3. Analog-digital interface • The placement of the boundary between analog and digital is flexible • But transitions are expensive • All-digital approach: send raw sensor data to digital domain • Places the burden upon the analog-digital interconnect and digital processing power consumption • All-analog approach: all-analog signal processing • Often highly task specific which increases development time and reduces generalization to other applications • Mostly digital • Analog world is directly sampled into the digital domain • e.g. all-digital implementations • -Mostly analog • Analog world is processed and interpreted in analog • e.g. traditional analog implementations A D A D

4. Analog-digital interface – smart sensors • The placement of the boundary between analog and digital is flexible • But transitions are expensive • Smart sensors and actuators • Learning and interpretation of analog information • Adaptation in analog sensor and actuator operation • Mostly digital • Analog world is directly sampled into the digital domain • e.g. all-digital implementations • -Mostly analog • Analog world is processed and interpreted in analog • e.g. traditional analog implementations A D A D

5. Analog-digital interface • Since the transition from analog domain to digital domain is expensive, only transmit what is necessary. • Maximize information content of each digital bit • Minimize transfer of redundant information • Analog sensor interface • Objective • Operate analog circuits in high efficiency regime for low-power performance • Integrated local analog signal processing circuitry results in sparse data being transferred to the digital domain • Extract features of interest from sensors in the analog domain • Transmit as digital events to the digital domain meaning? 0 1 0 0 1 0 Analog to digital encoding

6. Event-based sensing approach Describes features of object as time- based digital events • Each digital event encodes a feature of interest from the sensor • Event encoding • Feature selection • Select what is and is not a feature from sensor data • Decide what feature information to transmit for each event (i.e. spatial position, temporal position, etc.) • Event decoding • Digital processor must now interpret and understand what each event means 0 1 0 0 1 0 Analog to digital encoding

7. Dynamic vision sensor (DVS) • Frame-free image (scene) processing • Only transmits individual pixel information when has a change in relative log intensity • Characteristics • Low bandwidth • Low power consumption • Low computational requirements • High sensor dynamic range • Technical specifications • 128x128 resolution, 120dB dynamic range, 23mW power consumption, 2.1% contrast threshold mismatch, 15us latency • http://www.youtube.com/embed/5NNoq1Gq4sc Lichtsteiner, et. al. (ISSCC 2006, JSSC 2008)

8. A silicon retina that reproduces signals in the optic nerve • Frame-free image (scene) processing • Only transmits individual pixel information when has a change in relative log intensity • Event decoding scheme • ON activity corresponds to bright pixels and OFF activity corresponds to dark pixels • Technical specifications • <100mW power consumption, 3.5mm x 3.3 mm Zaghloul, et. al. (J. Neural Eng. 2006)

9. Convolution chips for image processing • Event-based image processing • Frame-free event-based image processing of asynchronous events • On-the-fly processing of events results in 2-D filtered version of the input flow • Characteristics • Arbitrary kernel size and shape • Technical specifications • 32x32 pixel 2-D convolution event processor, 155ns event latency between output and input, 20Meps input rate, 45 Meps output rate, 350nm CMOS, 4.3x5.4mm2, 200mW at maximum kernel size and maximum input event rate Linares-Barranco, et. al. (TCAS 2011)

10. Silicon cochlea architecture Input sound • 2nd order LPF bank Seek to emulate cochlea performance and functionality by emulating cochlea biological architecture in silicon • Transform into analog signal • Transform into “digital” neural event signal Chan, et. al. (TCAS I 2007) Digital events • Each “event” is a data packet describing event source (LPF) and event time

11. Reconstructed silicon cochlea data Input sound PC reconstructs the output digital event information by sorting by channel (LPF) number and then aligning according to time stamp information. Silicon cochlea Digital events channel number PC time

12. Example data with pure tones (for one channel) 300 Hz pure tone 750 Hz pure tone • Simple real-time data processing procedure • Count the time difference between events (interspike interval, ISI) for each channel • Arrange the ISIs into a histogram • A peak in the ISI histogram indicates a resonant frequency response channel number channel number time time bin count bin count ISI ISI

13. Sound Discrimination Example “coo” sound “hiss” sound Wav file FFT ISI histogram

14. IFAT (Analog CMOS) DRAM HiAER (Digital CMOS) Top metal Top metal TSV I/O pad 5 mm 5 mm 5 mm 5 mm Hierarchical address-event routing (HiAER) 3-D integrated silicon neuromorphic processor Park, et. al. (ISCAS 2012) Receiver Sender • 65,000, two-compartment neurons • Conductance-based integrate and fire array transceiver (IFAT) • 65 million, 32-bit “virtual” synapses • Conductance-based dynamical synapses • Dynamic table-look in embedded memory (2Gb DRAM) • Locally dense, globally sparse synaptic interconnectivity • Hierarchical address-event routing (HiAER) • Dynamically reconfigurable • Asynchronous spike event I/O interface HiAER IFAT 0.13μm CMOS 0.13μm CMOS

15. Event-driven framework Provide background on motivation • Coincidence detection performs efficient spike-based computation • coincidence detection • two or more arriving events result in a stronger response than a single arriving event • applications • event-driven sensing • sensors are only “on” when something important happens • event-driven computation • information is sparsely represented with events Event-based approach relies upon temporal encoding to communicate signals. The time of the event is the key parameter, not the voltage value. Event-encoding is robust against additive noise. Yu, et. al. (EMBC 2012) Theodore Yu UCSD Integrated Systems Neuroengineering Lab

16. Temporal code and synchrony • At a local scale, neurons perform coincidence detection within temporal integration window. • At a network scale, the temporal delay information in events models the spatial distribution between neurons. • Each scene of interest can be encoded as a unique combination of features 5ms delay Coincidence? Yes or no? Input pattern 10ms delay 4ms delay

17. Temporal code and synchrony example Event at t = 7ms 5ms delay Coincidence? No! Event at t = 3ms 10ms delay Event at t = 8ms 4ms delay Event at t = 7ms 5ms delay Coincidence? Yes! Event at t = 2ms 10ms delay Event at t = 8ms 4ms delay

18. Summary • Analog event-based sensing • Since the transition from analog domain to digital domain is expensive, only transmit what is necessary. • Maximize information content of each digital event through encoding of features in analog domain • Minimize transfer of redundant information for sparse digital signal processing • Applications • Visual and acoustic sensors for event-encoding of features • Event-based processor performs event-decoding of features utilizing coincidence detection in neural synchrony

19. Thank you