Saliency-guided Enhancement for Volume Visualization

1. Saliency-guided Enhancement for Volume Visualization Youngmin Kim and Amitabh Varshney Department of Computer Science University of Maryland at College Park

2. Motivation • The volume datasets have grown in complexity • Visible Human Project • 13GB ~ 60GB • National Library of Medicine (NIH) • Richtmyer-Meshkov Instability Simulation • 2 TB (= 7.5GB * 273 time steps) • Lawrence Livermore National Laboratory • Human visual capabilities remain fixed • The need to draw visual attention to appropriate regions in their visualization

3. Motivation • We can draw viewer attention in several ways • Obtrusive methods like arrows or flashing pixels • Distracts the viewer from exploring other regions • Principles of visual perception used by artists and illustrators • Gently guide to regions that they wished to emphasize

4. Contributions • A new saliency-based enhancement operator • Guides visual attention in volume visualization without sacrificing local context • Considers the influence of each voxel at multiple scales • Augments the existing visualization pipeline • Enhances regional visual saliency • Validation by eye-tracking-based user study • Our method elicits greater visual attention

5. Related Work - Saliency • Computation and Evaluation • Computational models for image [Itti et al. PAMI 98] and mesh [Lee et al. SIGGRAPH 05] • Evaluation by predicting eye movements [Parkhurst et al. 02], [Privitera and Stark PAMI 00] Mesh Saliency • Use of eye movements • Volume composition [Lu et al. EuroVis 06] • Abstractions of photographs [DeCarlo and Santella SIGGRAPH 02, NPAR 04] • Use of Saliency • Progressive visualization [Machiraju et al., 01] • Importance-based enhancement [Rheingans and Ebert TVCG 01] • Interior and exterior visualization [Viola et al. TVCG 05] • Generalizing focus+context [Hauser Dagstuhl 03] • Saliency has not been used for guiding visual attention

6. Related Work – Transfer Functions • Transfer Functions map the physical appearance to the local geometric attributes such as: • Gradient magnitude [Levoy CG&A 88] • First and second derivatives [Kindlmann and Durkin Volume Rendering 98] • Multi-dimensional transfer functions [Kindlmann et al. Vis 03], [Kniss et al. TVCG 02], [Kniss et al. Vis 03], [Machiraju et al. 01] • Have played a crucial role in informative Visualization • Difficult to emphasize (or deemphasize) regions specified exclusively by locations in a volume

7. Transfer Functions Saliency Enhancement Enhancement Operators Emphasis Field Computed Saliency Field by User Input Saliency-enhanced Volume Rendering Validation by eye-tracking device Overview • Saliency Field • Enhancement Operators • Emphasis Field • Saliency Enhancement • Saliency-enhanced Volume Rendering • Validation by eye-tracking based user study

8. Basic idea from Saliency Computation C: Mean curvature • Saliency map is: • Mesh saliency based on curvature values • Image saliency based on intensity and color • In general, saliency may be defined on a given scalar field S (v)=|G(C, v, σ) – G(C, v, 2σ)|

9. Unknown Known Known Unknown Emphasis Field Computation • Mesh Saliency: S(v) = G(C, v, σ) – G(C, v, 2σ) • We introduce the concept of an Emphasis Field Eto define a Saliency Field S in a volume S (v) = G(E, v, σ) – G(E, v, 2σ) Given a saliency field, can we design some scalar field that will generate it?

10. = Emphasis Field Computation • Expressible as simultaneous linear equations • Saliency Enhancement Operator (C-1) • CE =S , which implies E = C-1S • Given a saliency field S, the enhancement operator C-1 will generate the emphasis field E where cij is the difference between two Gaussian weights at scale σ and at scale 2σ for a voxel vj from the center voxel vi

11. Emphasis Field Computation • We like to use enhancement operators at multiple scales σi • Let E i be the emphasis field at scale σi • Compute this by applying the enhancement operator Ci-1 on the saliency field S • Final emphasis field is computed as the summation of E i

12. Emphasis Field in Practice • A system of simultaneous linear equations in n variables • Generally, can handle arbitrary saliency regions and values • Computationally expensive: O(kn2) or O(n3) • Alleviate this by solving a 1D system of equations • Given a saliency field • Solve 1D system of equations at multiple scales and sum them up • Approximate results using piecewise polynomial radial functions [Wendland 1995] • Interpret results to be along the radial dimension • Assume spherical regions of interest (ROI)

13. Visualization Enhancement • Emphasis Fields can alter visualization parameters in several ways • Various rendering stylizations and effects possible • We outline a couple of possibilities • Brightness • Widely used to elicit visual attention by artists • Modulate the Value parameter in the HSV modelas follows: • Vnew(v) = V(v)•(1+E (v)), where –λ- ≤E (v) ≤ λ+ • Used 0.4 ≤ λ+ ≤ 0.6 and 0.15 ≤ λ- ≤ 0.35 • Saturation • Can modulate Saturation instead of Value if the latter is not effective (for instance, in regions already very bright)

14. Gaussian-based vs. Saliency-guided Enhancement • Previous Gaussian-based Enhancement of a Volume • Volume Illustration [Rheingans and Ebert TVCG 01] • Importance-based regional enhancement • We use a Gaussian fall-off from the boundary of ROI

15. Visualization Enhancement - Brightness Traditional Volume Rendering Gaussian-based Enhancement Saliency-guided Enhancement Traditional Volume Rendering Gaussian-based Enhancement Saliency-guided Enhancement

16. Visualization Enhancement - Saturation • Increasing brightness diminishes the appearance of blood vessels at the center of the Sheep Heart model Traditional Volume Rendering Saliency-guided Enhancement

17. User Study • Validated results by an eye-tracking-based user study • Hypotheses: The eye fixations increase over the region of interest (ROI) in a volume by the saliency-guided enhancement compared to • the traditional volume visualization (Hypothesis H1) • the Gaussian-based enhancement (Hypothesis H2)

18. User Study – Experimental Design • Eye-tracker and General Settings • ISCAN ETL-500 • Records eye movements at 60Hz • 17-inch LCD monitor • With a resolution of 1280x1024 • Placed at a distance of 50cm (19.7’’) from the subjects • Eye-tracker Calibration • Desired accuracy of 30 pixels • Two-step calibration process • Standard calibration with 5 points • Look and click on 13 points • Triangulation and interpolation with 4 corner points • Accuracy test on 16 random points

19. User Study – Experimental Design Extracting fixations from raw points • Raw points: all points from the eye-tracker • Saccade Removal • Velocity > 15°/sec • Fixation combining • Filter out the points which stay less than 100ms within 15 pixels • Average eye locations within 15 pixels and 100ms

20. User Study – Experimental Design • Image Ordering • 10 users (who passed the accuracy tests) • Total of 20 images: 4 models * (1 original + 2 regions * 2 different enhancement methods (Gaussian, Saliency)) • Each user saw 12 images out of these 20 images • 4 models * (1 original + 2 altered)) • Enhanced different regions with different methods • Placed similar images far apart to alleviate differential carryover effects • Randomized the order of regions and the order of enhancement types (Gaussian and saliency-based) to counterbalance overall effects • Duration • 12 trials (images), each of which takes 5 seconds

21. User Study – Result I Gaussian-based Enhancement Gaussian-based Enhancement With Fixation Points Saliency Field Saliency-guided Enhancement With Fixation Points Saliency-guided Enhancement Traditional Volume Rendering Traditional Volume Rendering With Fixation Points

22. User Study – Result II Gaussian-based Enhancement Gaussian-based Enhancement With Fixation Points Saliency Field Saliency-guided Enhancement With Fixation Points Saliency-guided Enhancement Traditional Volume Rendering Traditional Volume Rendering With Fixation Points

23. Data Analysis I The percentage of fixations on the ROI for the original, Gaussian-enhanced, and Saliency-enhanced visualizations

24. Data Analysis II • A two-way ANOVA on the percentage of fixations for two conditions, regions and enhancement methods for each volume • For regions, no statistically significant results as expected • F(1,34) = 0.2827 ~ 3.3336, p > 0.05 • For enhancement methods, statistically significant results • F(2,34) = 7.2668 ~ 31.479, p ≤ 0.01

25. H2 H2 H2 H2 H1 H1 H1 H1 Data Analysis III • Carried out a pairwise t-test on the percentage of fixations before and after we applied enhancement techniques for each model • Found a statistically significant difference in the percentage of fixations with saliency-guided enhancement for all the models • Hypothesis H1:More fixations than the traditional • Hypothesis H2:More fixations than the Gaussian

26. Conclusions • Introduced the concept of the Emphasis Field for selective visual emphasis (or de-emphasis) • Developed the computational framework to generate the Emphasis Field from a given Saliency Field • Illustrated the use of the Emphasis Field in Visualization • Validated its ability to successfully guide visual attention to desired regions • Saliency-guided Enhancement provides a powerful tool to help scientists, engineers, and medical researchers explore large visual datasets

27. Future Work • Measure comprehensibility of the volume rendered images • Explore other appearance attributes such as opacity and texture detail • Generalize to handle time-varying datasets with multiple superposed scalar and vector fields • Identify the relative importance of various scales

28. Acknowledgments • Datasets: Stefan Roettger (University of Erlangen) and Dirk Bartz (University of Tuebingen) • Discussions: David Jacobs, François Guimbretière, Derek Juba, and Robert Patro (University of Maryland) • Eye-tracker: François Guimbretière • The Anonymous Referees • Supported by NSF grants: CCF 05-41120, CCF 04-29753, CNS 04-03313, and IIS 04-14699

29. Questions ?? Lab:Project:Images: www.cs.umd.edu/gvil www.cs.umd.edu/gvil/projects/sevv.shtml Supplemental material in the DVD-ROM