1 / 19

Multimodal sensing-based camera applications

Multimodal sensing-based camera applications. Jari Hannuksela, Olli Silvén Machine Vision Group, Infotech Oulu Department of Electrical and Information Engineeering University of Oulu, Finland. Miguel Bordallo 1 , Jari Hannuksela 1 , Olli Silvén 1 and Markku Vehviläinen 2

mary
Download Presentation

Multimodal sensing-based camera applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multimodal sensing-based camera applications Jari Hannuksela, Olli Silvén Machine Vision Group, Infotech Oulu Department of Electrical and Information Engineeering University of Oulu, Finland Miguel Bordallo1, Jari Hannuksela1, Olli Silvén1 and Markku Vehviläinen2 1 University of Oulu, Finland 2 Nokia Research Center, Tampere, Finland

  2. Outline Introduction • Modern movile device with multiple sensors Vision-based User Interfaces Sensor data fusion system Application case implementations • Motion-based image browser • Motion sensor assisted panorama imaging Conclusions/Summary

  3. Introduction • More and more applications and features are being crammed into handhelds • Causes usability complications given the constraints of current mobile UIs • Increased computing power not harnessed for UIs • Keypad and pointer based UIs and/or touchscreens in current devices • User’s hand obstructs the view • Require two handed operation

  4. Modern mobile device with multiple sensors • The phone includes touch screen, GPS, accelerometers, light sensor, proximity sensor • Two cameras: low resolution for video calls and high resolution for photography and video capture • Newer phones will include magnetometers, gyroscopes

  5. Motivation for vision based user interfaces Allow recognition of the context - Detect user’s actions - Recognize environment Allow 3D information Provide interactivity - Real-time feedback - Single hand use

  6. Limitations of vision based UIs Fast movements Low light Difficult conditions +

  7. The solution: sensor data fusion Fusing the data obtained by several sensors • Ambience light sensor determines illumination conditions • Video analysis detects ego-movements and analyzes the context • Accelerometers provide complementary motion data

  8. Video analysis • Every frame divided into regions • Selection of feature blocks • Estimation of block displacements • Analysis of uncertainty • Results: 4-paramenter model • X, Y, Z, r

  9. Sensor data fusion: Accelerometers

  10. Sensor data fusion Model the device movement with the folowing Define a state vector: position, speed, acceleration Define a measurement model Apply Kalman filtering adding accelerometer values: State prediction + state correction

  11. Application cases • Sensor data fusion method applied in two applications • Implemented on a Nokia N900 mobile phone • Motion based image browser • Allows browsing large images and maps with one hand operation • Works under different light conditions • Sensor assisted panorama imaging • Stitches panorama images in real time from video frames • Increased robustness against fast movements and no-texture frames

  12. Motion based image browser • Uses fusion model from accelerometers + video analysis to generate commands • Scroll up/down/left/right • Zoom in/out • Light sensor decides: • if camera should be turned on • weighting factors and uncertainties • 3 modes defined: • Good image quality (video analysis + accelerometer correction) • Bad image (accelerometers have increased contribution) • No image (only accelerometers are used)

  13. Motion based image browser II

  14. Sensor assisted panorama Imaging • Based on video analysis • Guides the user with instructions • >360 degrees panoramas • Real-time registration • Real-time frame evaluation and selection • Real-time frame correction • Increased robustness via sensor-data integration

  15. Panorama imaging: Sensor uses Registration • Uses sensor fusion model to compute camera motion • Increased robustness against fast movements and frames with low/smooth texture

  16. Panorama: Sensor uses II Selection • Uses accelerometer data to detect blur • Detects unwanted shake/tilt • Integrated in scoring system

  17. Summary • Vision based interfaces offer high interactivity with one hand operation • They present several limitations • Sensor fusion improves motion estimation adding robusness against fast movements and dark conditions • The framework can be included in several applications (e.g. as a part of Motion Estimation API)

  18. Conclusions • We have presented a sensor fusion framework that fuses vide analysis with motion sensors (acelerometers+magnetometers+gyroscopes) • We have presented two applications cases that make use of sensor data fusion and integration • The applications presented are by no means the only ways to apply vision or multiple sensors, and one may find new interesting possibilities in further research

  19. Thankyou! Any question???

More Related