1 / 52

Designing for energy-efficient vision-based interactivity on mobile devices

Designing for energy-efficient vision-based interactivity on mobile devices. Miguel Bordallo Center for Machine Vision Research. Objective of the research. To gain understanding on how to build the future mobile plaftorms in order to satisfy their interactivity requirements

cyndi
Download Presentation

Designing for energy-efficient vision-based interactivity on mobile devices

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Designing for energy-efficient vision-based interactivity on mobile devices Miguel Bordallo Center for Machine Vision Research

  2. Objective of the research • To gain understanding on how to build the future mobile plaftorms in order to satisfy their interactivity requirements • To provide insight into the computing needs and characteristics of the future camera-based applications

  3. Smartphones are not smart • Current mobile devices lack interactivity • Unable to detect if you hold them • Unable to detect if you are looking • Unable to predict your intentions • Mobile devices don’t ”watch you” (or listen) • You need to actively indicate what you want • Application launch has VERY high latency

  4. Typical UIs and interaction methods • Buttons • Reduced functionality • Touch screens • Needs (often) two hands operations • Motion sensors (+ proximity, light, etc) • Mostly used when the user is ”active”

  5. Vision-based interactivity • Using cameras as an Input modality • Enables recognizing the context in real time (and to see the user and environment) • Current mobile devices integrate touch screen, sensors and several cameras • But UI s don’t use them together !! • The small size of handheld devices and their multiple cameras and sensors are under-exploited assets

  6. Vision-based UI

  7. Vision-based Interaction methods Head movement triggers Interactive image capture Automatic start of applications

  8. Why don’t* we have these kind of methods on our mobile devices? *(some of them are coming)

  9. Challenges/needs of vision-based interactivity

  10. Challenges/needs of vision-based interactivity • Very low latency (below 100 ms.)

  11. Challenges/needs of vision-based interactivity • Very low latency (below 100 ms.) • Computationally costly algorithms

  12. Challenges/needs of vision-based interactivity • Very low latency (below 100 ms.) • Computationally costly algorithms • Sensors (cameras) ”always” on

  13. Challenges/needs of vision-based interactivity • Very low latency (below 100 ms.) • Computationally costly algorithms • Sensors (cameras) ”always” on • Energy-efficient solutions

  14. Challenges/needs of vision-based interactivity • Very low latency (below 100 ms.) • Computationally costly algorithms • Sensors (cameras) ”always” on • Energy-efficient solutions

  15. Are mobile platforms energy-efficient?

  16. Energy-efficiency on mobile devices • Battery life is a critical mobile device feature • App. performance is constrained by battery life • Energy efficiency is managed by switching off complete subsystems • Cameras, motion sensors, GPS, CPU cores, ... • Only ”important” subsystems are always on and responsive (standby mode) • GSM/3G modem, buttons

  17. Battery capacity

  18. Battery vs. CPU frequency

  19. Battery vs. CPU power

  20. Battery vs. talk time

  21. Battery vs. ”active use”* time * Don’t trust these numbers 

  22. ”Active use”* time * Don’t trust these numbers 

  23. Active use vs processor power

  24. Current platforms

  25. How can we improve the energy efficiency of Vision-based interactive applications and UI s?

  26. Offering Computer Vision algorithms and apps as a part of a Multimedia/CV Framework - Filtering, feature detection, robust estimators, classifiers, block matching, - Face detection, motion estimation, blending Avoid the use of the application processor for ”sensing” tasks

  27. Asymmetric multiprocessing(Heterogenous computing) • Concurrently use different processors on a mobile device to perform suitable tasks • Processors identical (multicore) or heterogenous (CPU+GPU+DSP+CoDec)

  28. GP-GPU-based interaction acceleration • GPUs are present in most modern mobile devices • GP-GPU exploits GPUs for general purpose algorithms • Mobile GPUs have architectural advantages • Computer Vision on GPUs very popular field but.... • Cameras and sensors lack fast data transfer • Image formats not always compatible • IDE and interfaces not mature (OpenCL, OpenGL ES)

  29. Sensor processor assisted context recognition • Dedicated chips for sensor/camera processing • IVA2+, ISP • Based on DSP processors + HW codecs • Good interconnections with sensors/cameras • Reasonably good performance/efficiency but... • Complicated and obscure interfaces • Access not always allowed to regular developer • Limited flexibility

  30. Dedicated computing for vision-based User Interfaces • Dedicated (programmable) architectures offer: • Incredibly high performance (Hybrid SIMD/MIMD) or.. • Extremely good energy efficiency (TTA) but... • Not incorporated into current devices • Not likely to be anytime soon 

  31. Performance of different processors Platform: OMAP3530 (Nokia N900)

  32. Performance of different processors Platform: OMAP3530 (Nokia N900)

  33. Performance of different processors Platform: OMAP3530 (Nokia N900)

  34. Performance of different processors Platform: OMAP3530 (Nokia N900)

  35. Battery discharge time (constant load) Power consumed (mW) 1320mAh Battery time (h)

  36. Battery discharge time (constant load) Power consumed (mW) Battery time (h)

  37. Battery discharge time (constant load) Power consumed (mW) 2100 mAh Battery time (h)

  38. Battery discharge time (constant load) Power consumed (mW) Battery time (h)

  39. Battery discharge time (constant load) 7000 mAh !! Power consumed (mW) Battery time (h)

  40. Battery discharge time (constant load) Power consumed (mW) Battery time (h)

  41. Battery discharge time (constant load) Power consumed (mW) Battery time (h)

  42. Battery discharge time (constant load) Active-use zone Power consumed (mW) ”Knee” region Standby zone Battery time (h)

  43. Battery discharge time (constant load) Power consumed (mW) Battery time (h)

  44. Battery discharge time (constant load) VB UI (active state) Power consumed (mW) Battery time (h)

  45. Battery discharge time (constant load) VB UI (active state) Power consumed (mW) Battery time (h)

  46. Battery discharge time (constant load) VB UI (active state) Power consumed (mW) Battery time (h)

  47. Designing for interactivity • Mobile devices need architectural changes to incorporate Vision-Based Uis • Small footprint processors close to the sensors • Sensors ”always” ON at a small framerate • Only processed data arrives to the application processor

  48. Current platforms

  49. Current platforms

  50. IRcam IRcam QVGA VGA

More Related