Multimodal sensors & digital interfaces

1. Multimodal sensors & digital interfaces

2. Credits • The original Multimodal Project developed by and credit for: • Zhigang Zhu and Weihong Li (Integration of Laser Vibrometry with Infrared Video for Multimedia Surveillance Display)

3. Outline • Multimodal System Overview • Multimedia Sensors • Infrared camera • The LDV sensor • PTZ camera • Multimodal System Components • System Design Concept • Design Issues • Integration Issues

4. Multimodal System Overview • The object of this system is to provide a multimodal integration of audio, visible, thermal for human signature detection. • The goal is to use the sensing technologies for perimeter surveillance. • Sensors, alarm, response. • Multimodal system interface • The environment, the sensors, and the events.

5. Multimedia Sensors – Infrared Camera • Infrared camera • FLIR ThermoVision A40M IR camera • Temp Range of -20º to 500ºC, accuracy (% of reading) ± 2ºC or ± 2% • 320x240 Focal Plane Array • 24º FOV Lens • Firewire Output IEEE 1394 • Video output – RS170 EIA/NTSC or CCIR/PAL composite video for monitoring on a TV screen • ThermoVision System Developers Kit (C++) • Each thermal image is built from 76,800 individual picture elements that are sampled 60 times per second by the camera's on-board electronics.

6. Multimedia Sensors – Infrared Camera • Samples Figure 1. A person sitting in dark room can be clearly seen in the IR image. The temperature at Sp1 on the face is 33.1ºC Figure 2. Two IR images before and after a person standing at about 200 feet. The reading of the temperature at Sp1 changes from 11C to 27C.

7. Multimedia Sensors – Infrared Camera • Thermographic measurement techniques • An infrared camera measures and images the emitted infrared radiation from an object. • The radiation measured by the camera does not only depend on the temperature of the objects but is also a function of the emissivity. • Radiation also originates from the surroundings and is reflected in the object • Radiation from the object and reflected radiation will also be influenced by the absorption of the atmosphere • Parameters need to take care: • The emissivity of the object • The reflected temperature • The distance between the object and the camera • The relative humidity

8. Multimedia Sensors – Infrared Camera • Emissivity: • How much radiation is emitted from the object • Object materials and surface treatments exhibit emissivity ranging from approximately 0.1 – 0.95 • Highly polished (mirror) surface < 0.1 • Human skin exhibits an emissivity close to 1 • Metal: low, increase with temperature • Non-metal: high, decrease with temperature

9. Multimedia Sensors – Infrared Camera • Reflected ambient temperature: • To compensate for the radiation reflected in the object and the radiation emitted from the atmosphere between the camera and the object. • If the emissivity is low, the distance very long and the object temperature relatively close to that of the ambient it will be important to set and compensated for the ambient. • Distance • The distance between object and the front lens of the camera. • Relative Humidity • Normally, default 50%

10. Multimedia Sensors – Infrared Camera • History of Infrared Technology • Sir William Herschel (1738-1822) • Discover of infrared spectrum • Marsilio Landriani (1746-1815) • As the blackened thermometer was moved slowly along the colors of the spectrum, the temperature readings showed a steady increase from the violet end to the red end. • Macedonio Melloni (1798-1854) • Rock salt (NaCl) (to be made into lenses and prisms) is remarkably transparent to the infrared. • Sir John Herschel • The first ‘heat-picture’ in 1840, thermograph • Samuel P. Langley (1834-1906) • Inventor of the bolometer (1880)

11. Multimedia Sensors – LDV sensor • Vibrometer types: • Single Point Vibrometers: • Measure the vibration of an object in the direction of laser beam • Differential Vibrometers: (dual beam) • Allow vibration measurement between two points vibrating relative to each other. • Rotational Vibrometers: • Measure angular vibrations on rotating structures. • In-plane Vibrometers: • measure continuous (DC) velocity and superimposed variable (AC) components perpendicular to the central axis of two converging laser beams. • 3D Vibrometers

12. Multimedia Sensors – LDV sensor • Laser Doppler Vibrometer (LDV) • Optical instruments for accurately measuring velocity and displacement of vibrating structures completely without contact. • Sensor head OFV-505 • HeNe laser,  = 633.8 nm. • OFV-SLR lens (f=30mm) 1.8m – 200+m, auto focus • Controller OFV-5000 with a digital velocity decode card VD-6 • RS232 interface for computer control • Telescope VIB-A-P05 • ±1º vertical tilt and ± 1.5º horizontal tilt

13. Multimedia Sensors – LDV sensor • Measurement Principle S is the light source f is frequency P is the moving with velocity v and reflects the light O is the receiver (f + fD) Resultant frequency shift For vibrometers: S=O ("backscatter") 1= - 2, therefore,

14. Multimedia Sensors – LDV sensor • LDV schema • Velocity is directly obtained by demodulation:  = 2fm • Voice frequency f: 300Hz ~ 3000Hz • LDV can detect vibration at a magnitude as low as m = /2f = 1/(2*3.14*300) = 0.5µm

15. Multimedia Sensors – PTZ camera • Pan/tilt/zoom (PTZ) camera • Human and other target detection at a large distance • Canon PTZ • 26X optical zoom lens & 12X digital zoom • Pan: ±100º, Tilt: +90º/-30º • Built-in IR light (effective up to 9 feet) • BNC video output • RS-232 computer control interface

16. Multimedia Sensors – PTZ camera • PTZ Samples Two images of a person at a distance of about 200 feet, captured by changing the zoom factors of the PTZ camera.

17. Multimodal System Components • Three components • The IR/EO imaging video surveillance component • Human motion tracking, human face detection • Thermal Camera for daytime and nighttime • Visible camcorder (sony), Web cam (logitech) • The LDV audio surveillance component • Audio signal capture, voice recognition • The human-computer interaction component • Cognitive understanding of the environment, the sensors, and the events

18. System Design Concept • The overall goal the project is to design a human computer interface for human-centered multimodal (MM) surveillance.

19. Design Issues • Issues need to be considered • how to use EO camera tracking human motion; • how to incorporate IR imaging with existing EO captured image; • how to use IR imaging to help the laser Doppler vibrometer to select the appropriate targets; • how to select optimal viewpoint from audio detection.

20. Integration Issues • Target detection and localization via IR/EO imaging • Set up an IR/EO imaging system with an IR camera and a PTZ camera for finding vibration targets for LDV listening • Registration between the IR/EO imaging system and the LDV system. • Two types of sensors need to be precisely aligned so that we can point the laser beam of the LDV to the target that the IR/EO imaging system has detected • Future research on automated targeting and focusing.

21. References • Main multimodal system technical report • http://www-cs.ccny.cuny.edu/~zhu/LDV/FinalReportsHTML/CCNY-LDV-Tech-Report-html.htm • Polytec Laser Vibrometer • http://www.polytec.com/ • FLIR Systems Security ThermoVision Cameras • http://www.flir.com/ • Paper: • Z Zhu, W Li, “Integrating LDV Audio and IR Video for Remote Multimodal Surveillance” • Others…

22. (END)