Towards Sensate Media and Electronic Skins Testbeds for very high density sensor networks Joseph Paradiso, Joshua Lifton

1. Towards Sensate Media and Electronic Skins Testbeds for very high density sensor networks Joseph Paradiso, Joshua Lifton, Michael Broxton Responsive Environments Group MIT Media Laboratory

2. mm-cm Local Processor Sensors Building an Electronic Skin involves: • Massive quantity of sensor channels - Not feasible to wire each sensing element to central processor • Processing must be embedded into the “skin” with the sensors - Biological analogy with neurons reducing sensor complexity before routing to brain - Sensor/processor “soup” • Pushes the frontier of Ultradense, multimodal sensor/processor nets - Deep challenges in decentralized processing and estimation, fabrication, power - Enables things to immersively sense as we do - Revolutionary apps in robotics, telepresence, medicine

3. Nervous system processes (inhibits, enhances) signals from skin receptors enroute to brain

4. The Pushpin Computing Testbed Completely Configurable Topology • Pushpin nature lets one place nodes wherever desired • Dynamic density • Easily block or shield parts of network • Easy to access each node directly • Easy to collectively stimulate sensors on groups of nodes • Layered Circuitry • Communications Layer (currently IR, capacitive prototype) • Processor Layer (currently 22 MIPs) • Sensor layer (photo sensor with LED outputs) • Layers easily swapped for customization, upgrades • Toolkit – all over ML, MIT • Over 100 constructed (Currently IR communication) • Capacitive (low-power RF) layer coming – more isotrophic

7. Pushpin Structure

8. Distributed Pattern Recogniton Light sensor & LED top layer New top layer under development that adds fast photonics and ultrasound pickup for synchronization/location

9. Current Pushpin Specs • Current processor is the Cygnal C8051F016 • 22 MHz clock • 1-2 cycles per instruction typical • 92 kbits per second (slow due to flash write speed when updating OS) • ~20mA @ 3V per Pushpin running with all analog systems & comm. • PWMs, comparators, 8 10-bit ADC lines, timers, external interrupts… • Power, processing, communication, expansion layers all modular • Transmit is fanned out from a common line to all IR LED’s • Receive is from one of four ports – each port separately detected • Can determine pulse width (~ signal strength) from external interrupts • Substrate is polyurethane and aluminum, ~1.2 m2 • Comm protocol is random back-off • >100 nodes have been built and used together

10. Testbed #2 – The Trible Tactile Reactive Interface Based on Linked Elements • First step at a multimodal electronic skin • 32 networked elements • Each measuring up to 12 channels of touch via tactile whiskers, distributed pressure at 3 points, local temperature, local illumination, local sound • Each with local speaker, pager motor, RGB LED • Elements talk to neighbors • No central processor • Behavior from decentralized algorithms • Peer-peer signals routed directly through Trible frame • Research platform for distributed sensing, processing, and control

11. Trible Construction

12. Completed Hairless Ball Single Cell Element

13. Public Exhibition – Artbots (NYC), 7/03

14. Trible Specs • 20 hexagons, 12 pentagons • Hex: 12 whiskers, 3 pressure, 1 light, 1 microphone, 1 temperature, 1 speaker, 1 pager motor, 1 PWM’ed RGB LED. • Pent: same as hex, but only 7 whiskers • Frame is laser cut ABS plastic joined with aluminum, all custom • Shell is lasercut, heat-formed, sand-blasted polyethylene • Whiskers are paintbrush hairs glued into piezoelectric foil picku[ • Same processing layer used as with Pushpins • Rechargeable battery power provides central power, can also take power from an external wired supply • Communication is neighbor-to-neighbor through aluminum joiners in frame • Central comm bus is only used for debugging and code updates • Communication is at 115200bps between panels • Maintains windowed average of each sensor channel • Plays 8-bit additive synthesis and sampled sounds • 17.2 kHz total sampling bandwidth spread among all channels... can go up • 72 kHz total sampling bandwidth by sacrificing processing time. • Output also updated at 17.2kHz, spread out depending on mapping of inputs to outputs • All input channels are sampled at 10 bits, but only 8 are used • Total of 516 sensor channels • Assembled Tribble is about 13" in diameter.

15. - A sensate floor as a sensor net • Self-tiling elements w. a processor and a dozen 2.5 cm pressure sensors • Network formed as tiles snap together • - Tiles talk to neighbors to parameterize footstep • Route results peer-peer to outside connection • Applications in entertainment, health care, smart homes/UbiComp… Z-Tiles (collaboration with U. Limerick)