Interaction Techniques in Medical Volume Visualization

1. Interaction Techniques in Medical Volume Visualization Bernhard Preim

2. Interaction Tasks and Techniques • Interaction Tasks • Exploration of original data • Data reduction • Manipulation of transfer functions • Multiplanar reformatting (MPR) Bernhard Preim

3. Interaction Tasks and Techniques: Exploration of Original Data • “Browsing” through the slice data • Simple contrast and brightness control via mouse movement (windowing) • Flexible definition of slices in a corresponding visualization • Cine mode for animation impression Bernhard Preim

4. Interaction Tasks and Techniques: Exploration of Original Data • Opening and closing of a legend in the viewer • Patient information (name, date of birth, gender, Id) • Image information (modality, voxel size, recording date) • Coordinate and value of the selected voxel • Option: more or less detailed legend • Synchronized display of two data sets • Example: Liver CT; first data set without contrast agent, second data set with CA • Synchronization related to windowing parameters and the displayed layer • Selection of the viewing direction (coronary, sagittal, axial) Bernhard Preim

5. Interaction Tasks and Techniques: Exploration of Original Data • Example for legends, data: Univ. Hospital Leipzig Bernhard Preim

6. Interaction Tasks and Techniques: Exploration of Original Data • Change of contrast and brightness, data: Univ. Hospital Leipzig Bernhard Preim

7. Interaction Tasks and Techniques: Exploration of Original Data • Browsing through the slices (interactive or as movie) Bernhard Preim

8. Interaction Tasks and Techniques: Exploration of Original Data • Synchronized illustration. Left: original data, right: filtered data Bernhard Preim

9. Interaction Tasks and Techniques: Exploration of Original Data • Moving of a cross line in communicated views (Peter Hastreiter, Uni Erlangen) Historical model: Drawings by Dürer Bernhard Preim

10. Interaction Tasks and Techniques: Data Reduction • Why? • Focus on certain problems • Reduction of the data volume (memory requirements, rendering speed) • How? • Data selection in a certain interval (e.g. iso-surface) • Definition of a volume of interest (cuboid partial volume) • Subsampling of data (e.g. reduction by factor 2 in x and y direction) Bernhard Preim

11. Interaction Tasks and Techniques: Data Reduction • Definition of a VOI in orthogonal views Bernhard Preim

12. Interaction Tasks and Techniques: Transfer Functions • Transfer functions: Mapping of data onto presentation parameters (colors, gray values, transparency) • Determine the visibility and perceptibility of structures • Parametrization of TFs is an essential interaction for the exploration of volume data. • Challenges: • Exploration of data sets with unknown structures • Exploration of data sets with different structures of similar intensity Bernhard Preim

13. Interaction Tasks and Techniques: Transfer Functions • Three volume visualizations of one CT data set with different opacity transfer functions. Skin Bones Teeth Bernhard Preim

14. Interaction Tasks and Techniques: Transfer Functions • Requirements • Selection of predefined TFs (e.g. liver CT, lung CT) • Targeted search for suitable TFs • Correlation between adjustable parameters and characteristics of the resulting images • Definition flexibility • Fast preview Bernhard Preim

15. Interaction Tasks and Techniques: Transfer Functions • Typical transfer functions: • Windowing • Bi-/trilevel windowing • Inverse windowing • Piecewise linear functions • Polynoms of higher degree/splines • Problem: No recognizable relation between TF characteristics and visualization  Bernhard Preim

16. Interaction Tasks and Techniques: Transfer Functions • Thorax CT data set, emphasis of skeletal structures Bernhard Preim

17. Interaction Tasks and Techniques: Transfer Functions Thorax CT data set, emphasis of blood vessels Bernhard Preim

18. Interaction Tasks and Techniques: Transfer Functions • Representation and application of TFs • Discrete representation in lookup tables • Size: e.g. 4096 entries with 32 bit (8 bit each for RGB and alpha) • Hardware support for Lookup tables • Problem: hardware dependency w.r.t. size of color tables Bernhard Preim

19. Interaction Tasks and Techniques: Manipulation of Transfer Functions • Sophisticated concepts: • Stochastic generation of TFs that may be selected by the user (multilevel iterative search), presentation as thumbnails (He et al. [1996], König et al. [2001]) • Image-based TF design (Fang et al. [1998]) • Enhanced TF • Integration of image processing filters (e.g. edge recognition) • Local TF • Multidimensional TF (illustration of derived data, e.g. gradient fields, Levoy [1988]) Bernhard Preim

20. Interaction Tasks and Techniques: Manipulation of Transfer Functions • Stochastic generation of TFs: • Iterative search process (He et al. [1996]): • 1. Use of an initial TF library • 2. "Mutation" of this function through a genetic algorithm (25 generations) • 3. Direct volume rendering (back then with VolVis 100x100 pixel, 10s) • 4. Subjective result analysis by the user Bernhard Preim

21. Interaction Tasks and Techniques: Manipulation of Transfer Functions Source: König et al. [2001] Bernhard Preim

22. Interaction Tasks and Techniques: Transfer Functions • Image-based TF design • Idea: Definition of the transfer function, image information serve as context (Castro et al. [1998]) • Global histogram • Histogram along a layer • Histogram along a ray Bernhard Preim

23. Interaction Tasks and Techniques:Transfer Functions • Histogram along the orange ray as context for TF specification eye ball (light) muscles (dark) Image: Dirk Bartz, Univ. Leipzig Bernhard Preim

24. Interaction Tasks and Techniques: Transfer Functions • RGBAlpha and gray value Alpha TF (Peter Hastreiter, Uni Erlangen) Bernhard Preim

25. Interaction Tasks and Techniques: Transfer Functions • Composition of a TF as weighted sum of component functions • Parameters of component functions: • Sb, Sc - inner sampling points, Sa, Sd- outer sampling points Bernhard Preim

26. Interaction Tasks and Techniques: Transfer Functions • Adaptation of a trapezoid template to the local histogram of a rectangular region. Source: Castro et al. [1998] Bernhard Preim

27. Interaction Tasks and Techniques: Transfer Functions • “Implicit” segmentation of the white brain substance through suitable transfer functions • Emphasis of the histogram area between the maxima of gray and white brain substance Histogram TF (purple: opacity values, green: gray values) Intersection of gray and white substance Bernhard Preim

28. Interaction Tasks and Techniques: Transfer Functions • “Implicit” segmentation of the white brain substance through suitable transfer functions • Emphasis of the histogram area between the maxima of gray and white brain substance Bernhard Preim

29. Interaction Tasks and Techniques: Transfer Functions • The Transfer Function Bake-Off, Data: Sheep heart (IEEE CG&Application 5/6 2001) • Comparison of different TF specification techniques • ISO rendering of the segmented raw data (sheep heart) • Trial&Error - (20 min) with VolumePro • Without data model - ISO automatically selected according to the maximum gradient magnitude • 2D TF with data model – automatic distance map, semiautom. opacity, manual color map Bernhard Preim

30. Interaction Tasks and Techniques: Transfer Functions • Data-based Techniques • Selection of a transfer function that emphasizes the edges. • Edge model: • Perfect intersection between 2 structures is "blurred" through an error function. Assumption: Blurring through an isotropic Gaussian function. -> fits to CT data well • Source: Kindlman, Durkin [1998] Bernhard Preim

31. Interaction Tasks and Techniques: Transfer Functions • Data-based Techniques: edge enhancement • Edge criteria: strong gradient g, very small second derivative h (zero crossing): • -h(v) • p(v) = • g(v) • Data values along an edge, 1st and 2nd derivative Bernhard Preim

32. Interaction Tasks and Techniques: Transfer Functions • Determination of g (v) and h (v) via average determination from all first and second derivatives of all voxels with value v. • Internal representation: • Histogram volume H: • x-axis → f (v) • y-axis → f“(v) • z-axis → f´(v) • Algorithm: • 1. Determine min. and max. values • for f‘‘(v) and the maximum for f´(v). • Minimum for f´(v) is assumed to be 0. • 2. Fill H, whereas the values are scaled • such that min and max are depicted from • f´ and f´´ to 0 and 256. Bernhard Preim

33. Interaction Tasks and Techniques: Transfer Functions • Data-based Techniques: edge enhancement • What can be determined from the histogram volume? • Edge positions w.r.t. the data • What can be entered by the user? • A selection of the "peaks" that shall be depicted • Form of the depicted peaks via boundary emphasis function (bef) • Typical forms of bef() Bernhard Preim

34. Interaction Tasks and Techniques: Transfer Functions • Data-based Techniques: edge enhancement • Applied 2D opacity function • and volume rendering of the • Visible Woman data set • (TF automatically determined). • The small image indicates the 2D • Histogram (intensity values vs. • Gradient magnitude) • Brightness indicates frequency of. • Source: Kindlmann, Durkin (1998) Bernhard Preim

35. Interaction Tasks and Techniques: Transfer Functions • Data-based Techniques: edge enhancement • Comparison of edge-enhancing direct volume rendering and iso-surface rendering Illustration of a spiny dendrite based on microscopy data • Source: Kindlmann and Durkin 1998 Bernhard Preim

36. Interaction Tasks and Techniques: Transfer Functions • Data-based Techniques: edge enhancement • Preconditions for successful application: • Existence of clear object boundaries • Homogenous data • Only little noise, no "outliers" • Medicine: CT data (if CA is applied, it must be equally distributed) Bernhard Preim

37. Interaction Tasks and Techniques: Transfer Functions • Data-basedTechniques • Useof a oncespecified TF asreference • Goal: "Re-use" of an empiricallyspecified TF • Application: targetedillustrationof a structure in a modality (e.g. aneurysms in MR) • Procedure: • Selectionof a referencedatasetDrefand a TFTref(v) • UseofthenormalizedhistogramsofthedatasetsH(Dref) andH(Dstudy) • Non-linear transformationtoftheintensityvaluesof Dstudy, such thatH (Dstudy) ~ H (t(Dref)) • Hence, Tstudy (v) = Tref (v) Bernhard Preim

38. Interaction Tasks and Techniques: Transfer Functions • Data-based Techniques: reference TF • Determination of the similarity of the histograms • 1. Idea: minimization of the histogram distances • 2. Better idea: use of the p-function by Kindlmann (considers also f‘(v) and f‘‘(v)) • In case of comparable data sets the p-values are similar to the histograms • Literature: Rezk-Salama et al., VMV [2000] Bernhard Preim

39. Interaction Tasks and Techniques: Transfer Functions • Data-based Techniques: Reference TF • Visualization of blood vessels in the brain with CT angiography, left: no adaptation, middle: illustration of the first idea (histogram transformation), right: adaptation of the p-function Source: Rezk-Salama et al. [2000] Bernhard Preim

40. Interaction Tasks and Techniques: Transfer Functions • Multidimensional TFs • 1D TF: Map data onto opacity/colors • Multidimensional TFs: Use additionally derived information, e.g. strength of the gradient or the second derivative • Typical example: Adaptation of the opacity to the strength of the gradient, emphasis of data intersections • Advantage: Additional degrees of freedom to generate high-quality images • Disadvantage: High interaction costs Bernhard Preim

41. Interaction Tasks and Techniques: Transfer Functions • Multidimensional TFs • Consideration of the 2nd derivative (1st derivative of a scalar field → vector, 2nd derivative → matrix) • Hessian Matrix: • Criterion (scalar value) for the 2nd derivative: largest eigenvalue of the Hessian Matrix and strength of the 2nd derivative, respectively in direction to the gradient (instead of the Hessian Matrix) Bernhard Preim

42. Interaction Tasks and Techniques: Transfer Functions • Multidimensional TFs • Gradient calculation usually via central differences • Mapping of the gradient size to the opacity (gradient magnitude weighted transparency) Bernhard Preim

43. Interaction Tasks and Techniques: Transfer Functions • Multidimensional TFs • Volume visualization with a gradient-dependent TF for opacity, accord. to Levoy [1988]) (Visible Human CT data set) Bernhard Preim

44. Interaction Tasks and Techniques:2D Transfer Functions • Starting point for a simple specification: gradient intensity histograms. Filtering is important. Goal: accentuation of intersections. Bernhard Preim

45. Interaction Tasks and Techniques:2D Transfer Functions Dense tissue and bone parts with additional gradient emphasis (green marking) Image courtesy: Hoen-Oh Shin and Benjamin King, MH Hannover [2004] Bernhard Preim

46. Interaction Tasks and Techniques: 2D Transfer Functions More dense soft tissue (yellow marking) Image courtesy: Hoen-Oh Shin and Benjamin King, MH Hannover [2004] Bernhard Preim

47. Interaction Tasks and Techniques: 2D Transfer Functions Regions with high gradients are visualized (red marking) Image courtesy: Hoen-Oh Shin and Benjamin King, MH Hannover [2004] Bernhard Preim

48. Interaction Tasks and Techniques: 2D Transfer Functions Source: Stölzl [2004] Bernhard Preim

49. Interaction Tasks and Techniques: 2D Transfer Functions • Edge detector as input to define arcs Source: Stölzl [2004] Bernhard Preim

50. Interaction Tasks and Techniques: Transfer Functions • Local TFs • Motivation: Often, global TFs enable no sufficient differentiation • Example: Division of a lookup table into 4 segments for 4 different illustrations Caution: Interpolation beyond segment borders is not allowed! Source: Rezk-Salama [2002] Bernhard Preim