Display Management: Sensing User Intention and Context

1. Display Management:Sensing User Intention and Context (HOTOS 2003)

2. Outline • Motivation and Research Objective • FaceOff Architecture and Prototype • Evaluation • Best Case Feasibility Study • Responsiveness Study • Future Work • Related Work • Dark Windows

3. Motivation • Current energy management techniques tied to process execution • Can we use low power sensors to match I/O behavior more directly to user behavior and reduce system energy consumption? Sensing User Intention and Context for Energy Management

4. Case Study: FaceOff • Displays: • Typically responsible for large power drain • Power State can be controlled by software • State transition strategies naïve A display is only necessary if someone is looking at it.

5. Image Capture Face Detector Main Control Loop No Face=off Face=on

6. Prototype • IBM ThinkPad T21 running RedHat Linux • Base Power Consumption = 9.6 Watts • Max CPU = 8.5 Watts over Base • Display = 7.6 Watts • Logitech QuickCam Web Cam • Power Consumption = 1.5 Watts • Software components: • Image capture, face detection, display power state control

7. Face Detection • Skin detection used for prototype • Real time proprietary methods exist

8. Outline • Motivation and Research Objective • FaceOff Architecture and Prototype • Evaluation • Best Case Feasibility Study • Responsiveness Study • Future Work • Related Work • Dark Windows

9. Best Case Feasibility Study • What is the potential for energy savings? • Assume perfect accuracy • Best case user behavior – start it and leave. • Tradeoff of energy costs: • CPU/Camera vs. Display • Effect on System Performance • Network file transfer (113 MB) • CPU intensive process (Linux kernel compile) • MP3 Song (no display necessary)

10. File Transfer Traces

11. Kernel Compile Traces

12. Energy and Time Comparisons

13. MP3 Application • Playing an MP3 • Display not necessary • Song completes before default timeout turns off display • Energy comparison • 3,403 J with FaceOff vs. 4,714 J with Default • 28% energy savings • No noticeable effect on playback

14. Responsiveness Study • Use full prototype including skin detection • Establish baseline timing • Examine Responsiveness • varying system load • varying polling rate

15. Responsiveness Timing polling latency detection latency Face arrives (or departs) Image acquired detection complete display signaled Total responsiveness latency

16. Baseline Detection Latency • Measured over a period of one hour with no programs other than background processes running • Latency increased over time • Started at ~110ms • Increased to ~160ms • Why? • Appears to be an effect of Linux scheduler reducing priority of long running jobs

17. Detection Latency over Time

18. Detection Latency Under Load

19. Outline • Motivation and Research Objective • FaceOff Architecture and Prototype • Evaluation • Best Case Feasibility Study • Responsiveness Study • Future Work • Related Work • Dark Windows

20. Varying Polling Rate • Reduce overhead by reducing polling rate • Increases responsiveness latency • Adaptive polling rate • Eliminate polling in presence of UI events • Begin polling as duration without UI events increases and face is detected • Reduce polling when no face present • Similar problem with latency increase upon return

21. Optimization with Motion Sensor • Combine adaptive polling & motion sensing • Meet responsiveness requirements with minimal FaceOff system overhead • Eliminate image polling when no motion • Switch display state on immediately when motion detected and restart image polling

22. Implementation • Prototype using X10 ActiveHome Wireless Motion Sensor and Receiver • Receiver connects to serial port • Reading port blocks until sensor triggers • Takes up to 10 seconds to recharge • Promising addition to FaceOff system

23. More Roles for Sensors • Touch Sensor • Detect picking up of a PDA • Light, Sound sensors • Adjust display brightness (Compaq iPAQ) • Adjust speaker volume • 802.11 Signal Strength sensor • Determine possibility of offloading computation

24. Enhanced Sensors • “Active Camera” • Perform some or all of the face detection • Color filtering • Preprocessing skin color segmentation • Low Power processor for external sensor control, computation

25. Discussion: Other ideas for using sensors to save energy?

26. Future Work • Continue work on optimizing responsiveness • Comprehensive user study • Survey of usability • Characterization of usage patterns • End-to-end experiment • Implementation with available very low power camera/motion sensor and prototype for small device (handheld)

27. Conclusions • Context information offers promising method of energy management • FaceOff illustrates feasibility of approach • Available very low power sensors as well as optimization techniques would improve upon the FaceOff energy savings

28. Outline • Motivation and Research Objective • FaceOff Architecture and Prototype • Evaluation • Best Case Feasibility Study • Responsiveness Study • Future Work • Related Work • Dark Windows

29. Related Work • Display Power Management • Industry Specifications • APM, ACPI, DPMS • Zoned Backlighting • Energy-Adaptive Display System Design • Attentive/Perceptual UIs • Smart Kiosk System: Gesture analysis • CAMSHIFT: Game control • IBM PupilCam: Head gesture recognition

30. ACPIAdvanced Configuration and Power Initiative Brought to you by Intel, Microsoft, and Toshiba and designed to enable OS Directed Power Management (OSPM). • Goal is to be able to move power management into software for more sophisticated policies • Abstract OS-HW interface • Replaces APM interface

31. What ACPI Offers • Standardization industry-wide (Vendors to support ACPI in products instead of building their own power mgt) • System and device power states • Thermal model • Thermal zones, indicators, cooling methods • BIOS interfaces • Motherboard configuration tables • Interpreted control methods • Plug-and-play • Complexity moved into OS

32. What ACPI Offers • System • Mechanisms for putting computer as a whole in sleep/wake states • Devices • ACPI tables describe motherboard devices • Power states • Controls for managing states • Processor • Detecting idle state and swapping to low power • Batteries • Querying and controlling battery behavior

33. S4 S3 S2 S1 G1Sleep Power States G: global states apply to entire system and are visible to user D: states of individual devices S: sleeping states within the G1 state C: CPU states G3mech off wakeup G0-S0working Legacy G2-S5Soft off Dxmodem x DxHDD xDxcdrom x Cxcpu

34. ACPI Internal Structure

35. Transmeta Crusoe ACPI Power States

36. Applications OnNow WIN32 ext OS ACPI Spec HW OSDM: OnNow SetSystemPowerState • initiate sleep state, query apps(?) SetThreadExecutionState • specifies level of support needed (e.g. display required) WM_POWERBROADCAST • a message notifying of power state changes to which applications can respond SetWaitableTimer • ensure PC is awake at scheduled time RequestDeviceWakeup RequestWakeupLatency - to specify latency requirements GetSystemPowerStatus and GetDevicePowerState

37. Outline • Motivation and Research Objective • FaceOff Architecture and Prototype • Evaluation • Best Case Feasibility Study • Responsiveness Study • Future Work • Related Work • Dark Windows

38. Energy-Adaptive Display System Designs for Future Mobile Environments S. Iyer, L. Luo, R. Mayo, P. Ranganathan, MOBISYS 2003

39. Opportunity • New technology: OLEDs – Organic Light Emitting Diodes • Energy consumption on a per-pixel basis by determining each pixel’s brightness and color • Energy consumption of different regions of screen to be changed independently • No separate backlight • In development by Kodak, Sanyo, Sony,…

40. Dark Windows • Software support – modifications to windowing system to ensure energy is spent mostly on window-of-focus (capturing user’s area of visual interest) • Non-active screen area is changed (dimmed or re-colored for energy optimization)

41. Justification • Usage study – what are the user’s needs and how well do they match display characteristics? • Characterization of display usage in Microsoft Windows by 17 “typical” users • Application logger – recorded, for up to 14 days, when user was active. • Window of focus (the one accepting keyboard input) – its size, location, title • Size of total screen area used (all non-minimized windows

42. Screen Usage Results • On averageonly 59% of area used by window-of-focus,additional 17% bybackground windows • High variability among users • Large fraction of smaller windows have very low content (notifications, alerts) – don’t need full color characteristics of display to convey it.

43. Dark Windows Design • Prototyped on X Windows under Linux • Used VNC – Virtual Network Computing Server – provides a virtual representation of display – virtual framebuffer where pixels can be manipulated between application and display • Track window-of-focus, apply modifications to pixels outside of it

44. Modifications • Half Dimmed – areas outside window-of-focus are dimmed to 50% brightness • Fully Dimmed – areas outside are turned off • Gray Scale – areas outside are changed to gray by setting rgb to average value. • Green Scale – areas outside are changed to green which is lowest power color for OLEDs. Dims to 67%

46. Evaluation • Energy benefits measured by generating synthetic trace from usage study and playing trace on prototype. • 15 inch OLED displays were not available so they used a software power model to calculate power consumption • Controller power set to 0.5W • Driver power – 1W • 1024x768 pixels individually consume -- red power 4.3mW, green power 2.2mW, blue power 4.3mW. • User experience – users want to see background but willing to use dark windows

47. Results based on default Teal Background* * Benefits depend on original Choiceof backgroundand windowcolors.

48. Conclusions • While benefits depend on usage scenario, significant energy savings can be achieved with these optimizations • Further opportunities for application specific adaptivity • More meaningful notions of area of focus can be defined (e.g. most recent email message, most recently changed region of screen) • Better match to (low) content – e.g., notifications could be audible signal instead of popup window