Real-time Visualization of Massive Imagery and Volumetric Datasets

1. Real-time Visualization of Massive Imagery and Volumetric Datasets Thesis Defense by Ian Roth

2. Overview • Introduction • Out-of-core software architecture • Asynchronous data I/O engine • Hierarchical caching • Interactive visualization techniques • Viewing modes for image generation • Per-tile rendering methods • User interface and navigation modes • Performance and benchmarking • Features and extensions • Conclusions and future work

3. Kolam Software Package • Developed at University of Missouri, Columbia • Version 1.0 by Joshua Fraser (2000 – ‘04) • Written in C, OpenGL and GLUT • Version 2.0 by Ian Roth (2001 – ‘04) • Written in C++, OpenGL and Qt • Additional code by Jared Hoberock (2001 – ‘02) • Dataset I/O utilities • Other contributors • Dr. K. Palaniappan, Ian Scott, Dave Metts, Vidyasagar Chada, Mike Sullivan, Ryan Calhoun

4. Motivation • Massive datasets • Too large to fit entirely in memory • Examples: • IKONOS/MODIS panchromatic/multispectral satellite imagery • CT/MRI/PET scans, Visible Human Project • Out-of-core approach • Require segmentation (tiling) for efficient roam operations • Require multiple resolutions (LODs) for efficient zoom operations

5. Definitions • Image or Dataset • Encapsulation of a pyramid file (disk cache) • Ir(x) : image at resolution r, pixel coordinate x • Layer • A dataset with associated viewing parameters • Associated colormaps, heightmaps, etc. • Stores x/y offset and scale for embedded datasets • Colormap • Maps dataset values to RGB color values • Heightmap • Maps image pixels to height values

6. Kolam Software Architecture • Engine • Based on the workpile design pattern • Multiple worker threads process requests from application and perform data I/O operations • Application • Displays data and GUI components • Determines which tiles are visible and at what resolution • Posts request which are handled by engine’s worker threads • Application Extensions • Plugins: dynamically linked libraries • Provide additional GUI components, dataset I/O routines, image processing methods, image viewing and projection modes, etc.

7. Kolam Engine • Asynchronously processes requests from application • Proceedure: • Worker threads are initially idle • For each frame, the application determines visible tiles and posts requests • Each posted request awakens a worker thread • Repeat forever for each awakened thread • Worker thread attempts to pop request from queue; if no requests are on the queue, then exit loop. • Worker thread reserves a free slot in cache • Worker thread performs tile I/O • Application is notified of new data in cache • Worker thread enters idle state

8. Kolam Engine Versions • Version 1.0 • Each layer has its own resources • Per layer: cache space, threads, tile lookup table, requests • Fine for non-embedded datasets • Version 2.0 • Engine has global resources, shared by layers • Engine: cache space, threads, requests • Per layer: tile lookup table • Allows active layers to seize resources from inactive layers

9. Hierarchical Caching • Three levels: • Disk cache • Stores tiles on disk • Original source may be row-major order image file on disk or tiles streamed over network • Main memory cache • Stores fixed number of tiles in memory • Texture cache • Stores fixed number of tiles on graphics hardware • Passed directly to real-time display component

10. Disk Cache • Pyramid file format • Additional resolutions appended to end of image file • Requires additional space (no more than 2 ´ original) • 1D image takes 2 ´ space of original • 2D image takes 4/3 ´ space of original • 3D image takes 8/7 ´ space of original • Wavelet encoded file format • Additional resolutions embedded in image • Requires no additional space • Complex to encode/decode, especially in n-D • Examples: • JPEG 2000 • MrSID (LizardTech)

11. Main Memory Cache • Asynchronous I/O • Multiple read/write threads • Allows display thread to continue working while data is loading • Spreads out I/O lag time over shorter intervals • Caching: allocates/deallocates memory for tile data • Cache methods available to application developers • checkout tile, checkin tile, add request, query tile/request, invalidate tile/request • Variations of the checkout method • basic checkout, checkout first available, wait for tile • Paging: swaps old data for new • Temporal strategies – view projection independent • LRU priority queue – O(# of layers) • Spatial strategies – view projection dependent • Brute force distance calculation – O(# of cached tiles) • Modular ROI array – O(1) • Graph traversal algorithms – O(# of cached tiles)

12. Texture Cache • Similar to main memory LRU cache • Replace allocator object • Synchronous I/O • Adequate for single data display window • Can be implemented asynchronously • Useful for multiple displays • One-to-one correspondence between thread and context • Each display is essentially synchronous • Frame rate not bounded by longest display time • Implemented by OpenRM Scene Graph

13. Interactive Visualization Components • Viewing modes • Orthogonal projection • Oblique projection (separate source tree) • Spherical projection • Arbitrary geometry projection (future work) • Volumetric visualization (separate source tree) • Tile renderers • Raster renderer • Texture renderer • Terrain renderer • Ray casting renderer (separate source tree) • Navigation modes • Roam, zoom, and tilt • Fly mode (separate source tree) • Embedded dataset positioning • Image crop (future work) • Geometry and terrain manipulation (future work)

14. Visualization Running Time • Measured by number of tiles processed per frame • Definitions • n(I) : total number of tiles in image I • m(I, t) : number of visible tiles in image I at time t • Processing O(n) tiles per frame is too much • Processing W(m) tiles per frame is the goal • Analytic solution: • O(m) : single resolution solution • O(m + lgn) : multiple resolution solution • Not suitable for explicitly defined projections, i.e. triangular meshes • View projection dependent, often difficult to implement • Iterative solution: • O(mlgm + lgn) : single/multiple resolution solution • Relies on quadtree/octree/scenegraph traversal to iteratively refine solution from coarse to fine resolutions • View projection independent, relatively east to implement

15. Orthogonal Projection Analytic Solution • t : 2D translation/position vector in original resolution pixel coords • z : pixel zoom factor • {wT, hT} : tile width/height • {wV, hV} : viewport width/height • Global image resolution: • Scaling matrix (to tile coordinates): • Visible rectangular region width/height: • Visible rectangular region x/y offset: • Resulting region R is in tile coordinates at scale r

16. Oblique ProjectionAnalytic Solution • Single resolution solution • Intersect view frustum planes with image plane • Result is a polygon in the image plane • Clip polygon to image boundaries • Run polygon fill algorithm to find tiles in polygon at a fixed resolution • Flood fill algorithm (recommended) • First use Bresenham to discretize polygon to tile coords • Scanline fill algorithm • Multiple resolution solution • First subdivide frustum into regions • Calculate distance from eye p when z = 1 • q : FovY/2, r : resolution, d : distance from eye • Assume pixel aspect ratio = 1 • d and p both measured in pixels • Perform single resolution solution for each section

17. Oblique ProjectionIterative Solution • Quadtree traversal • Determine if tile is visible • View-frustum culling test • Performed in either R3 or canonical view volume (CVV) space • Test 4 tile vertices w/ 6 frustum planes (24 tests) • Test 4 tile edges w/ 6 frustum planes (24 tests) • Test 12 frustum edges w/ tile plane (12 tests) • Maximum of 60 tests per tile • Determine tile resolution • Project to viewing plane and calculate quadrilateral area • Caclulate resolution at sample points

18. Spherical ProjectionAnalytic Solution • Convert Euler coords to spherical • u : [0, 2p], v : [-p/2, p/2] • Intersect view frustum planes with sphere • Result will be up to 6 circles in 3D space • n : unit normal of plane P,p : any point on P • cc/s : circle/sphere center, rc/s : circle/sphere radius • Find distance d from cs to plane along n • Project circles of intersection onto image plane • Parametric equations for circle and sphere • t : [0, 2p], x : vertex in R3 • n : unit normal of circle plane, p : up vector • a = n´p, b = n´a • Solve for u and v • For each circle • Pick sample t values • u(t) and v(t) are points in image plane • Compute polygon intersections • Tesselate non-convex polygons • Perform pairwise convex polygon intersection tests

19. Spherical ProjectionIterative Solution • Determine visible tiles • Tiles no longer planar, can be composed of many polygons • Use bounding spheres to approximate curved tile shapes • Similar to vertex/frustum intersection test • Hidden surface removal • Back-face culling methods don’t work on curved surfaces • Additional clipping plane to divide sphere into front- and back-facing regions • Orthogonal to vector v • Need to determine distance from eye • Determine tile resolution • Projecting and calculating area is difficult for curved tiles • Overlapping regions not accounted for • Caclulate resolution at sample points

20. Arbitrary Geometry ProjectionIterative Solution • Image projected onto arbitrary triangular mesh • Similar to spherical projection • Requires mapping from mesh vertices to u/v coords • Determine visible tiles • Use hierarchy of bounding spheres • Hidden surface removal • ID image : a lookup table of visible triangles • Determine tile resolution • Caclulate resolution at sample points

21. Volumetric VisualizationIterative Solution • Octree traversal • Determine if tile (cube) is visible • View-frustum culling test • Performed in either R3 or canonical view volume (CVV) space • Test 8 tile vertices w/ 6 frustum planes (48 tests) • Test 12 tile edges w/ 6 frustum planes (72 tests) • Test 8 frustum vertices w/ 6 tile planes (48 tests) • Test 12 frustum edges w/ 6 tile planes (72 tests) • Maximum of 240 tests per tile • Using bounding spheres only 6 tests required, but less accurate • Optimization: use spheres during octree traversal, use cubes on leaf nodes • Determine tile resolution • Project to viewing plane and calculate convex hull (polygon) area • Caclulate resolution at sample points

22. Raster Renderer • Renders images directly to framebuffer • Advantages • Simple to implement • No additional texture/VRAM cache required • No additional overhead from texture load time • Tile data changes are applied instantly (on next display update) • No need to flush texture cache • Disadvantages • Can only be used in orthogonal viewing mode • Not all affine transformations supported • Rotations and shearing • Only 2D transformations • Can be slow on some graphics hardware

23. Texture Renderer • Stores tile data in texture/VRAM memory on graphics hardware • Advantages • Often faster than rendering directly to framebuffer • Allows any affine transformation to be applied to a tile • Allows hardware-accellerated interpolation to be applied when rendering • Can be used by all viewing modes • Volumetric viewing mode requires 3D texture support, otherwise must use ray casting renderer • Disadvantages • Wasted space when compositing color channels • Luminance component always stored in red channel in OpenGL • Must use RGB color textures for color masking to be effective • Need to flush texture cache when tile data changes or global filters are applied • Texture cache • Similar to main memory LRU cache • Synchronous rather than asynchronous • Can reuse some code by replacing the allocator object • Texture borders • Use OpenGL GL_CLAMP_TO_EDGE texture parameter • Widely supported, produces artifacts when z  1 • Use OpenGL texture borders • Not widely supported, often slow • Tradeoff between efficiency and storage • Simulate texture borders (not implemented) • Tile size: 2n – 2, Texture coords: [1 / 2n,(2n – 1) / 2n] • Requires major change to file format • Tradeoff between efficiency and storage

24. Terrain Renderer • Similar to texture renderer • Also converts a heightmap to visible terrain • Rendering a single tile requires neighboring heightmap tiles to be available • Two possible implementations • Triangular mesh terrain • Problems with seams across tiles of disparate resolutions • Bump-mapped terrain (Ian Scott) • Horizon still looks flat • Requires fragment shader support (GPU) • Requires additional texture memory

25. Ray Casting Renderer • Used by volumetric view mode • Can be implemented on GPU • Graphics hardware must support 3D textures and fragment shaders • Proceedure • Project tile onto viewing plane • To find regions of plane that intersect tile • Intersect 6 tile planes with viewing plane • Map polygon vertices to voxel coordinates To find regions of plane that do not intersect tile • Project 8 tile vertices onto viewing plane • Compute convex hull of result (a polygon) • Traverse each pixel of polygon in voxel space • Flood fill algorithm • Scanline fill algorithm (recommended) • For each pixel of polygon • If pixel coordinate in voxel space intersects tile • Tile voxel coordinate = pixel coordinate in voxel space modulo tile width/height/depth • Can Otherwise • Tile voxel coordinate = intersection of ray from eye through pixel with frontmost tile plane • Traverse ray from eye through pixel starting with voxel coordinate • Accumulate voxel data along ray

26. Navigation Modes • Roam, zoom and tilt • Drag image using combination of mouse buttons • Fly mode • Constant velocity based on ray from screen center to mouse cursor • Embedded dataset positioning • Position, resize and rotate a single dataset • Image crop • Drag edges of dataset to change size without stretching • Geometry manipulation • Move individual vertices of geometry • Select group of vertices to move at once • Use mass-spring system to pull connected vertices • Use finite element method to simulate objects with physical properties

27. Optimizations • Hiding disk latency • Search upward through quadtree for equivalent coarser resolutions • Never see blank tiles, see blurred tiles instead • Implemented in “checkout first available” cache checkout method • Maintaining constant FPS • Maintain maximum number of visible tiles • Replace group(s) of 2–4 tiles with a single coarser tile while roaming and/or zooming • Prioritizing requests • Maintain list of visible tiles for current and previous frames • Compare lists • Remove requests for tiles that are no longer visible • Move requests for tiles that are no longer visible to lower priority queue • Prefetching • Load constant size radius of tiles from around ROI when idle • Load adjacent tiles from coarser/finer resolutions when idle • Use motion vectors from recent user navigation to predict next ROI

28. Performance • Interactivity • FPS & # of dropped requests w/ varying number of layers • FPS & # of dropped requests w/ varying cache sizes • FPS w/ varying # of (embedded/non-embedded) layers • Hardware-specific tests • FPS w/ and w/o colormaps • FPS of raster vs. texture mode • Cache efficiency • I/O latency w/ and w/o compression • Paging time analysis • File format efficiency • Compression ratios

29. Performance – Multithreading

30. Performance – Cache Size

31. Application Extensions • Plugins: dynamically linked libraries • Plugins have access to all application data • Plugins can attach processing routines to predefined breakpoints in the code • A list of only the attached routines is processed at each breakpoint • Typical extension categories • Image/dataset file formats • GUI components • Viewing modes • Tile renderers • Navigation modes

32. Example Plugins • Apply Colormap • Provides GUI control similar to layer dialog • Uses “prerender” breakpoint to apply colormap to image before display • Histogram Display & Enhancement • Provides GUI control w/sliders • Uses “layer initialized” breakpoint to spawn new thread to compute histogram of overview • Uses “wait for tile” tile checkout method to avoid busy waiting • Uses “prerender” breakpoint to apply histogram enhancement to image before display

33. Conclusions and Future Work • Conclusions • A stable and basic visualization tool • Many features yet to be implemented • Need funding for full time programmers! • SBIR Proposal • Extension to n-dimensional data • View slices of both sparse and dense datasets • Adoption of HDF5 file format • Chunking, grouping, n-D support • Possibly provide our own extensions to the format • Image projection onto arbitrarily large triangular meshes • Unified framework for sphere mapping, terrain, etc. • Fast image registration • Incorporate modeling and deformation tools • Network streaming of tile data • Similar to Keyhole software

34. Similar Large Data Visualization Software • ESRI – ArcGIS • Full-featured GIS package • May not handle large images (bigger than available RAM/virtual memory) • Keyhole – EarthViewer • Network streaming of remote sensing & vector data • LizardTech – GeoExpress View • MrSID file format & image viewing software • Achieves high compression ratios at expense of quality • Sensor Systems – Remote View • Image processing and analysis software used in DoD • Doesn’t handle large images (bigger than available RAM/virtual memory) • EPFL – Real-time Visible Human Navigator • Only works on visible human dataset • SGI – OpenGL Performer/Volumizer • C/C++ API for 2D/3D large data visualization • IRIX/Linux platforms only • Kitware – VTK (Visualization Toolkit) • C++/Tcl/Java/Python API for 2D/3D large data visualization • Large mesh dataset support based on multiple LODs, no inherent datset segmentation

35. Screenshot – Display Windows

36. Screenshot – GUI & Colormaps

37. Animation • Embedded Datasets • Layers & Colormaps • Spherical View Mode

38. IBM T221 Display