Gaze-Contingent Displays: Review and Current Trends

1. Adaptive Displays Conference 2004 Gaze-Contingent Displays:Review and Current Trends Andrew T. Duchowski

2. Acknowledgements • National Science Foundation • NASA Ames • DoD / Navy / SPAWAR Charleston • Students • Nathan Cournia • Hunter Murphy • Scott Gibson (Summer Research Internship)

3. Overview • Brief review of Gaze-Contingent Displays (GCDs) • “Tuning” of GCDs via real-time metrics: • update rates • display / interface design • Current trends • GPU-programmable image processing • Eye tracking technology: state-of-the-art

4. Gaze-Contingent Displays • Motivation: • minimize display or user’s attentional bandwidth • balance displayed information against visual information processing capacity (via eye tracking) • Generally, partition display in two distinct regions: • high-resolution foveal region • low-resolution peripheral surround • Critical concerns: • how, what, and when to display in the periphery

5. GCD Strategies • Three approaches (low- to high-level): • screen-based displays (manage pixels) • model-based displays (manage graphics objects) • Attentive User Interfaces (manage interface) • Basic assumption: • observer’s real-time focus of attention coincides with fixation point (eye movements) • measured unobtrusively by eye tracker

6. Attentive User Interfaces Fig.1: AUIs; (a) eyeCONTACT sensor, (b) Light fixture with eyeCONTACT sensor, (c) eyePROXY, (d) attentive TV (Courtesy Roel Vertegaal, see Shell et al. (2003))

7. Model-Based Approach • Model-Based Gaze-Contingent Applications: • manipulate geometry just prior to display • degrade resolution of peripherally located objects • Not as much progress in this area (vs. screen-based) despite recent advancements in Level Of Detail (LOD) modeling techniques • Most relevant established technique is that of isotropic LOD management, as originally proposed by Clarke (1976)

8. Model-Based Approach • Isotropic LOD management: • pre-computed fine-to-coarse object hierarchies • resolution is uniformly degraded as object recedes from view (based on projected pixel area coverage) • Isotropic LOD management not always desirable, especially when viewing large objects close up • Due to advancements in multiresolution modeling, it is now becoming feasible to extend LOD approach to nonisotropic object rendering

9. Model-Based Examples • Luebke et al.’s (2002) gaze-contingent LOD graphics • O’Sullivan et al.’s (2002) temporal graphics degradation Fig.2: gaze-contingent graphics

10. Model-Based Examples • Our work: • Gaze-contingent terrain generation (AAAI Symposium 2000) • Gaze-contingent graphics modeling (EuroGraphics 2001) Fig.3: gaze-contingent terrain, model

11. Model-Based Metrics • Key questions: is it worth it? • how much to degrade, where? • does degradation reduce rendering time? • how does degradation impact user? • Parhurst and Niebur (2004) discuss classic speed / accuracy tradeoff: • degradation improves display speed (interaction) • reaction times GCDs impede target indentification

12. Screen-Based Approach • Strategy: • manipulate framebuffer just prior to display • periphery is often masked or smoothed to compress image information (bits-per-pixel) • Good deal of progress in this area since design of early eye-slaved flight simulators • Research is rooted with classic Psychology research on reading perception and “moving window” paradigm (McConkie and Rayner 1975)

13. Screen-Based Image Coding (a) Recorded scanpath (b) Reconstructed image Fig.4: HVS-matching wavelet coefficient scaling (Haar wavelets emphasizing degradation effects)

14. Screen-Based Video Coding • Bergström (2003) incorporates MAR-based visual acuity model into transform coders Fig.5: The Barbara image DCT coded (left) and MDCT coded (right). The focus point is in the centre of the MDCT coded image. Fig.6: Barbara coded by the DWT coder (left) and the MDWT coder (right).

15. Screen-Based Metrics • Loschky and McConkie (2000) found that to be imperceptible, image changes must start within 5 ms following end of saccade • Parkurst et al. (2000) found visual search performance with 5º window comparable to uniform resolution display Fig.7: Gaze-contingent multi-resolution displays

16. Current Trends • Good deal of work recurring in screen-based approaches: • introduction of arbitrary visual maps • hardware-accelerated processing • new applications (fisheye displays) • New research facilitated by rapid progress in eye tracking technology

17. Arbitrary Visual Maps • Perry and Geisler (2002) introduced arbitrary visual fields • An important advancement since degradation function of any shape can be created • Simulations of visual dysfunctions possible (e.g., glaucoma) Fig.8: Arbitrary Visual Fields

18. Hardware-Accelerating Imaging • GPU programs allow per-fragment resolution selection from mipmap pyramid • For image-based GCDs, image processing bottleneck effectively eliminated • Peripheral color degradation now possible Fig.9: GPU-programmable mipmap lookup

19. New Applications • PDT lens is a type of focus plus context screen with focal area magnified • Gaze-Contingent applications as yet unexplored Fig.10: Pliable Display Technology (PDT) lens

20. Eye Tracking Technology • 1st generation: plaster-of-paris, scleral coils • eye-in-head measurements; invasive • 2nd generation: photo- and video-oculography • 3rd generation: video-based corneal reflection • 4th generation: digital video, DSP computer vision algorithms (for face, eye-in-head detection) • much easier to use, still as accurate and fast • still not auto-calibrating, but getting closer

21. State-of-the-Art • State-of-the-art eye trackers facilitate rapid application dev. • Key GCD research question: • perception or performance? • Real-time applications: • simulation, telecommunication, etc. • Other directions: • diagnostic (off-line) uses, testing purposes, etc. Fig.11: Tobii eye tracker

22. References Bergström, P. (2003). Eye-Movement Controlled Image Coding. PhD Dissertation (No. 831), Institute of Technology, Linköping University, Linköping, Sweden. Clarke, J. H. (1976). Hierarchical Geometric Models for Visible Surface Algorithms. Communications of the ACM 19, 10, 547-554. Danforth, R., Duchowski, A., Geist, R., McAliley (2000). A Platform for Gaze-Contingent Virtual Environments. In Smart Graphics (Papers from the 2001 AAAI Spring Symposium, Technical Report SS-00-04), Menlo Park, CA, AAAI, pp. 66-70. Luebke, D., Reddy, M., Cohen, J., Varshney, A., Watson, B., and Huebner, R. (2002). Level of Detail for 3D Graphics. Morgan-Kaufmann Publishers, San Francisco, CA. McConkie, G. W. and Rayner, K. (1975). The Span of the Effective Stimulus During a Fixation in Reading. Perception & Psychophysics 17, 578-586.

23. References Murphy, H. and Duchowski, A. (2001). Gaze-Contingent Level Of Detail. In EuroGraphics (Short Presentations), Manchester, UK, EuroGraphics. O’Sullivan, C., Dingliana, J., and Howlett, S. (2002). Gaze-Contingent Algorithms for Interactive Graphics. In The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research, J. Hyöna, R. Radach, and H. Duebel, Eds., Elsevier Science, Oxford, England. Parkhurst, D., Culurciello, E., and Niebur, E. (2000). Evaluating Variable Resolution Displays with Visual Search: Task Performance and Eye Movements. In Eye Tracking Research & Applications Symposium p.105-109. Palm Beach Gardens, FL. Parkhurst, D. J., & Niebur, E. (2004). A Feasibility Test for Perceptually Adaptive Level of Detail Rendering on Desktop Systems. In Applied Perception and Graphics Visualization (APGV). ACM, Los Angeles, CA, to appear.

24. References Perry, J. S. and Geisler, W. S. (2002). Gaze-Contingent Real-Time Simulation of Arbitrary Visual Fields. In Human Vision and Electronic Imaging, San Jose, CA, SPIE. Shell, J. S., Selker, T., and Vertegaal, R. (2003). Interacting with Groups of Computers, Communications of the ACM 46, 3 (March), 40-46.