Interaction with Visualizations

1. Interaction with Visualizations Dr. Yan Liu Department of Biomedical, Industrial and Human Factors Engineering Wright State University

2. Introduction • Why is Interaction Important in Visualization • A good computer-based visualization should allow users to directly interact with and dynamically change the visualizations according to their viewing purposes • Capable of displaying more information as needed, disappearing when not needed, and accepting user commands to help with the thinking processes • What is Interactive Visualization • A process made up of a number of interlocking feedback loops that fall into three broad classes • Data selection and manipulation loop • Exploration and navigation loop • Problem-solving loop

3. Data Selection and Manipulation Loop • Process • Objects are selected and moved using the basic skills of eye-hand coordination • Timely reaction is crucial • Choice Reaction Time • The reaction time when there are multiple signals • Can be modeled by the Hick-Hyman law Reaction time = a + b∙log2(C) (Eq. 1) a, b: empirically determined constants C: the number of choices log2(C): the amount of information processed by the human operator, expressed in bits of information • Under normal conditions, the response time per bit of information is about 160 msec plus the time to set up the response • e.g. If there are 8 choices (3 bits of information), the response time will typically be on the order of the simple reaction time plus approximately 480msec

4. Data Selection and Manipulation Loop (Cont.) • Factors that Affect Choice Reaction Time • Distinctness of signals • The amount of visual noise • Stimulus-response compatibility • The degree to which what people perceive is consistent with the actions they need to take • It will be generally easier to execute tasks in computer interfaces if the interfaces are designed in such a way that they take advantage of previously learned ways of doing things • The degree of accuracy required • People respond faster if they are allowed to make mistakes occasionally (speed-accuracy tradeoff)

5. Data Selection and Manipulation Loop (Cont.) • 2D Positioning and Selection • The time taken to select a target that has a particular position and size can be estimated using the Fitts’ law • Positioning and Selection in Complex Visualization Systems • Significant lag between a hand movement and the visual feedback provided on the display • Modified Fitt’s law that includes lag (Eq. 2) Selection time = a + b∙log2(D/W+1) a, b: empirically determined constants D: the distance to the center of the target W: the width of the target log2(D/W+1): the index of difficulty (ID) (Eq. 3) Mean time = a + (Human Time + Machine Lag)∙log2(D/W+1) • The effects of lag increase as ID increases • Fitt's law is part of ISO standard 9214-9, which sets out protocols for evaluating user performance and comfort when using pointing devices with visual display terminals

6. Data Selection and Manipulation Loop (Cont.) • Hover Query • Passing the mouse cursor over an object to view more information about the object • It is usually implemented with a little delay • Path Tracing • Involve continuous ongoing control • Continually make a series of corrections based on visual feedback about the results of our recent actions • The speed of path tracing (Eq. 4) v = W/τ • v: the velocity • W: the path width • τ: a constant that depends on the motor control system of the person doing the tracing (between 0.05 and 0.11 sec)

7. Data Selection and Manipulation Loop (Cont.) • Two-Handed Interaction • Guiard’s kinematic chain theory • The left hand and right hand form a kinematic chain • For right-handed individuals the left hand provides a frame of reference for movements with the right • Applications in interface design • e.g. In an innovative computer-based drawing package, an artist could move templates (such as the French curve) rapidly over a drawing with his/her left hand while using his/her right hand to paint around the shape (Kurtenbach et al., 1997)

8. Exploration and Navigation Loop • Important when data are mapped to an extended visual space • Helps to understand the overall structure of how information is organized • Helps to identify particular objects of interest and paths to reach there • Issues • Theories of pathfinding and map use • Cognitive spatial metaphors • Issues related to direct manipulation and visual feedback • Information Presented as 3D Landscape • We should find it easy to navigate through data presented in this way because of our real-world experiences

9. Websites Arranged as a Data Landscape • Each site is represented a ziggurat crowned with a globe • The diameter of the ziggurat expresses the number of pages in the site • The height of the ziggurat indicates the visibility of the site • Measured by the number of other sites that have pointers to it • The size of the globe shows the luminosity of the site • Measured by the number of pointers with which it casts navigational light off-site • Color represents the site’s domain • green – education, red – government, and blue – commercial • Sites are distributed in space based on the strength of the linkages between them

10. Exploration and Navigation Loop (Cont.) • Gibson’s Affordance Theory (1986) • Locomotion is largely about perceiving and using the affordances offered for navigation by the environment • e.g. A path affords pedestrian locomotion from one place to another and features that prevent locomotion (such as obstacles, barriers, water margins, and brinks) • Can be applied in designing virtual environment • e.g. Designers create barriers and paths in order to encourage visits to certain location and discourage others • Depth Cues • All the perspective cues are important in providing a sense of scale and distance • The stereoscopic cue is important only for close-up navigation (e.g. walking through a crowd) • Stereoscopic depth perception is based on binocular disparity

11. Exploration and Navigation Loop (Cont.) • Design of Virtual Navigation Aids • The environments should be sparsely populated with discrete but separately identifiable objects • There must be enough landmarks so that several are always visible at any instant • Frame rates ideally should be at least 2 frames per second • Low frame rates cause lag in visual feedback and thus can introduce serious performance problems

12. Exploration and Navigation Loop (Cont.) • Constraints in Spatial Navigation Metaphors • Cognitive constraint • The metaphor should provide the user with a model that enables the prediction of system behavior given different kinds of input actions • A good metaphor is one that is apt, matches the system well, and is also easy to understand • Physical constraint • A metaphor can make some actions easy or difficult to carry out physically • e.g. A walking metaphor limits the viewpoint to a few feet above ground level and the speed to a few meters per second • Extend the notion of Gibson’s affordances to apply to both the physical characteristics of the user interface and the presentation of data • An interface with the right affordances is one that makes the possibility for actions plain to the user and gives feedback that is easy to interpret

13. Exploration and Navigation Loop (Cont.) • Types of Spatial Navigation Metaphors in Virtual 3D Spaces • World-in-hand • Eyeball-in-hand • Walking • Flying • World-in-Hand • The user metaphorically grabs some part of the 3D environment and moves it • e.g. Moving the viewpoint closer to some point in the environment actually involves pulling the environment closer to the user • May be optimal for viewing discrete, relatively compact data objects • e.g. virtual vases, telephone • It does not provide affordances for navigating long distances over extended terrains

14. Virtual Scene Illustration of world-in-hand metaphor

15. Exploration and Navigation Loop (Cont.) • Eyeball-in-Hand • The user imagines that he/she is directly manipulating his/her viewpoint • The virtual eyeball (a spatial positioning device) is placed at the desired viewpoint and the scene from this viewpoint is displayed on the monitor • One of the least effective methods for controlling the viewpoint • Consciously calculated activity is involved in setting a viewpoint (large individual differences) • Some viewpoints can lead to considerable confusion • e.g. If the eyeball is pointed away from the screen, the correspondence between hand motion and the image motion is confusing • Physical affordances are limited by the positions in which the user can physically place his/her hand • Certain views far above or below cannot be achieved or blocked by the physical objects in the room

16. Virtual Scene Illustration of eyeball-in-hand metaphor

17. Exploration and Navigation Loop (Cont.) • Walking • The user navigates the virtual environment by walking • Limited by the space of the virtual environment • Methods to address the issue of limited space of virtual environment • Devices like exercise treadmills so that people can walk without actually moving • Something like a pair of handlebar is used to steer directions • A system is created to capture the characteristic up-and-down head motion that occurs when people walk in place and then move the virtual viewpoint forward in the direction of the detected head orientation

18. Illustration of walking metaphor with a treadmill controller

19. Exploration and Navigation Loop (Cont.) • Flying • Enable users to smoothly create an animated sequence of views of the environment (such as using aircraft-like controls) • The goal is to make it easy for the user to get around in 3D space in a relatively unconstrained way (not modeling actual flight dynamics) • The interface based on this metaphor is considered more flexible and useful than those based on world-in-hand and eyeball-in-hand metaphors

20. Virtual Scene Illustration of flying metaphor

21. Exploration and Navigation Loop (Cont.) • Summary of Spatial Navigation Metaphors • The optimal navigation method depends on the exact nature of the task • A virtual walking interface may be the best way to give a visitor a sense of presence in an architectural space • Something loosely based on the flying metaphor may be a more useful way of navigating through spatially extended data landscape • The affordances of the virtual data space, the real physical space, the input device all interact with the mental model of the task that user has constructed

22. Interaction Techniques • Two Categories of Interaction Techniques (de Oliveira & Levkowitz, 2003) • Techniques that distort the image to allow visualization of a larger amount of data (distortion techniques) • The basic idea is to show portions of the data with a high level of detail while others are shown with a lower level of detail, so that the overview of the data is preserved • e.g. perspective wall, fisheye views, hyperbolic trees, etc. • Techniques that support more effective data exploration by allowing dynamic or interactive mapping of data attributes to visualization parameters or direct interaction with visualization models • e.g. interactive filtering, linking-and-brushing, overview+detail, etc.

23. Illustration of perspective wall Perspective Wall (Mackinlay et al., 1991) • A physical metaphor of folding is used to distort an arbitrary 2D layout into a 3D visualization • The wall has a panel in the center for viewing details and two perspective panels on either side for viewing context • The perspective panels are shaded to enhance the perception of 3D • Provides efficient space utilization for 2D layouts with wide aspect ratios (the ratios of the width to the height of the displays) • Files in a computer system are arranged on the perspective wall • Horizontal axis corresponds to the modification date • Vertical axis corresponds to the file type

24. Original graph Fisheye view of the graph Fisheye Views (Furnas, 1986) • The picture is scaled non-uniformly • The objects far away from the focus point are shrunk while objects near the focus point are magnified • The degree of visual distortion depends on the distance from the focus point

25. Interactive Filtering • Interactive determination of subsets of the dataset to be displayed • Directly selecting the desired subset • e.g. magic lens • Specifying criteria of the desired subset through queries • e.g. dynamic queries • Magic Lens (Example) • A movable, arbitrarily shaped region plus a filter that selectively filter the data in the considered areas • Multiple lenses can be used for multi-level filtering (allowing complex conditions) • Dynamic Query (Example) • As users adjust sliders, buttons, checkboxes, and/or other control widgets, the visual display is updated rapidly accordingly

26. Magic lenses that show details of two different locations A magic lens that shows details of a location A magnification lens and a lens that shows a wired view Back

27. Dynamic HomeFinder query system (Ahlberg &Shneiderman, 1994) • Users can adjust the upper and lower bounds on home prices and see points of light on a map indicating available properties • Users can adjust sliders to indicate the distances to locations A and B and the number of bedrooms • Users can select toggles to indicate desired features (garage, fireplace, central air-conditioning, and new house) and housing options (house, town house and condo)

28. Interactive Linking and Brushing • Plots are linked when changes to features of objects in one plot are reflected in the corresponding objects in the other plots • Brushing is a process in which users can highlight, select, or delete a subset of elements being graphically displayed by pointing at the elements with a mouse or other suitable input devices • In situations where multiple views of data are being shown simultaneously, brushing is often associated with linking, which make it possible to combine different visualization methods to take advantage of the strength of each of them

29. The user brushes some points in one plot, which causes the highlighting effect to be applied on points in the other plots that represent the same data records • Selecting some districts in the map of city Dublin highlights the highest levels of average incomes in the histogram

30. Overview+Detail (Plaisant et al., 1995) • Two or more levels of linked visualization windows are used • Overview window : Displays either all of the objects or at least some visual framework that spans all of the objects • Detailed window: Shows a detailed view of the object; the nodes in the detailed view are marked as a region that can be moved in the overview window • Multiple windows can be used to show detailed views at different levels • Global view shows the entire map of France • The selected region (surrounded by a red rectangle) in the global view is shown in more detail the intermediate view • The selected region (surrounded by a green rectangle with Paris at the center) in the intermediate view is shown in more detail in the detailed view Three-level Map browser with global, intermediate and detailed views

31. References • Mackinlay, J.D., Robertson, G.G., & Card, S.K. (1991). The perspective wall: detail and context smoothly integrated. Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, New Orleans, LA, 173-176. • Furnas G. (1986). Generalized fisheye views.Proc. Human Factors in Computing Systems CHI‘86 Conf., Boston, MA, 18-23. • Stone M. C., Fishkin K., & Bier E. A. (1994). The movable filter as a user interface tool.Proc. Human Factors in Computing Systems CHI ‘94 Conf., Boston, MA, 306-312. • Ahlberg, C., & Shneiderman, B. (1994).Visual information seeking: tight coupling of dynamic query filters with starfield displays. Conference on Human Factors in Computing Systems, Boston, MA, 222-228. • de Oliveira, M., & Levkowitz, H. (2003). From visual data exploration to visual data mining: a survey. IEEE Transactions on Visualization and Computer Graphics, 9(3), 378-394. • Plaisant, C., Carr D., & Shneiderman, B. (1995). Image-browser taxonomy and guidelines for designers.IEEE Software, 12(2), 21-32.