1 / 61

Interactive Exploration in Virtual Environments

Interactive Exploration in Virtual Environments. Robert G. Belleman Universiteit van Amsterdam Email: robbel@science.uva.nl. Overview. Large data spaces Interactive exploration environments Objectives Design An example: Simulated vascular reconstruction in a virtual environment

elvina
Download Presentation

Interactive Exploration in Virtual Environments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Interactive Exploration in Virtual Environments Robert G. Belleman Universiteit van Amsterdam Email: robbel@science.uva.nl

  2. Overview • Large data spaces • Interactive exploration environments • Objectives • Design • An example: • Simulated vascular reconstruction in a virtual environment • MSc projects

  3. Data explosion • High Performance Computing: • computing power increases (Moore’s “Law”) • storage capacity increases Result: “data spaces” get increasingly largeand complex; multi-dimensional, time dependent.

  4. What are “data spaces”? Roughly speaking: • Data sets • the “files” or “states” that are generated • Parameter spaces • the number of “variables” and their allowed “freedom” (range, resolution) in a program

  5. Large data sets: examples • Medical images (i.e. CT, (f)MRI, PET) • 512 x 512 at 16 bits slices in common use • tens to hundreds slices: 5Mb to5Gb per scan • 1024 x 1024 at 16 bit in verynear future: 20Mb to 20Gbper scan • time variant scans

  6. … more examples ... • Simulation experiment results • FEM, MD, lattice Boltzmann • often dimensionality > 3(e.g. time variant) • multiparameter data fields • gigabytes of data per “run”

  7. … more examples ... • Measurements • high-speed data acquisition devices: particle accelerators (Large Hadron Collider, LHC), microbeam scanners, DNA scanners, CLSM • Financial data, etc. Terabytes of data per experiment is no longer an exception!

  8. Parameter spaces • Simulation of complex systems • intractable: a certain timestep in a simulation can only be reached by starting at t=0 • NP complete: time and space requirements grow exponentially with problem size • Explicit simulation by a guided search through parameter space required (non-deterministic algorithms: SA, CA, NN, LBM, etc.)

  9. Parameter spaces: examples • Molecular dynamics • picosecond timeresolution • Molecular docking involves searchthrough large problemspaces

  10. … more examples … • Finite Element Methods (FEM) • large (hierarchically) structured meshes • Lattice Boltzmann methods (LBM) • large 3D (hierarchically) structured grids • large parameter spaces • “Optimization” problems in general

  11. From data to knowledge • Analysis of data spaces is often difficult • no analysis methods known, or ill-posed • size of data sets too large or too complex Often data analysis or simulation runs can take days, sometimes weeks!

  12. Bring in the expert Presentation is often the only way to obtain insight (note: not limited to visualization) Is it possible to make short cuts? E.g. by putting “an expert” in the loop?

  13. Human In The Loop • HITL • a.k.a. “interactive exploration” • a.k.a. “exploratory analysis” • a.k.a. “computational steering” • a.k.a. “problem solving environments” • a.k.a. “virtual laboratory”

  14. Interactive Exploration Environments Objective: to provide an interactive environment that allows for the exploration of large data spaces. Responsive, fast, accurate, reliable, easy-to-use, scalable, efficient, versatile, etc. The design of Interactive Exploration Environments is very multidisciplinary.

  15. Prerequisites • Intuitive interaction • increased functionality requires a well considered user interface • Real-time feedback: < 0.1 sec delay • Quality presentation • Informative, avoid clutter • Why Virtual Reality? • Rapid update rate: for continuous perception: • > 10 fps for vision • > 20 cps for sound • > 1000 cps for haptics These often conflict one another.

  16. Design • MVC (Model View Controller)design pattern Model Data values Computation View Translators Visualization State request Representation State Update state Update view Controller Session control Interaction User input User

  17. Interactive Dynamic Exploration Environments (IDEE) • Exploring dynamically changing data from “living” simulations • Changing parameters: What if...? • Requires “coordination”:

  18. Coordination params = Initialize(); running = true; While (running) { simdata = compute(params); visdata = visualize(simdata, params); render(visdata, params); params = interaction(); } Synchronous (lockstep)

  19. Coordination params = Initialize(); running = true; While (running) { simdata = compute(params); } params = Initialize(); running = true; While (running) { visdata = visualize(simdata, params); } params = Initialize(); running = true; While (running) { params = interaction(); } params = Initialize(); running = true; While (running) { render(visdata, params); } Asynchronous

  20. Coordination params = Initialize(); running = true; While (running) { simdata = compute(params); } params = Initialize(); running = true; While (running) { visdata = visualize(simdata, params); } params = Initialize(); running = true; While (running) { params = interaction(); } params = Initialize(); running = true; While (running) { render(visdata, params); }

  21. Shared memory Coordination on a single host params = Initialize(); running = true; While (running) { simdata = compute(params); } params = Initialize(); running = true; While (running) { visdata = visualize(simdata, params); } params simdata Question: What is wrongwith this method? visdata params = Initialize(); running = true; While (running) { params = interaction(); } params = Initialize(); running = true; While (running) { render(visdata, params); }

  22. Coordination on a single host params = Initialize(); running = true; While (running) { P(simdata_sema, params_sema); simdata = compute(params); V(simdata_sema, params_sema); } params = Initialize(); running = true; While (running) { P(simdata_sema, visdata_sema, params_sema); visdata = visualize(simdata, params); V(simdata_sema, visdata_sema, params_sema); } Shared memory params semaphores simdata visdata params = Initialize(); running = true; While (running) { P(params_sema); params = interaction(); V(params_sema); } params = Initialize(); running = true; While (running) { P(visdata_sema, params_sema); render(visdata, params); V(visdata_sema, params_sema); }

  23. Coordination on distributed hosts params = Initialize(); running = true; While (running) { simdata = compute(params); } params = Initialize(); running = true; While (running) { visdata = visualize(simdata, params); } Computing Cluster Vector processor Peer-to-peer networkconnections But communicationtakes time (precioustime)… PDA CAVE params = Initialize(); running = true; While (running) { params = interaction(); } params = Initialize(); running = true; While (running) { render(visdata, params); }

  24. Communication latency hiding • Multiple connections • Waiting for acknowledgements is hidden • Packets can travel through different routes

  25. Payload reduction • Data encoding • Decreases level of detail • Accuracy determined by type of representation • Lossy compression • Induces latency • “Standard” compression libraries (zlib) • Reduction to 10% not uncommon • Lossless compression • Induces latency

  26. Communication pipeline • Cascade of encoding, compression and multiple socket connections

  27. Pipeline performance Latency hiding: Payload reduction:

  28. Tuple space Coordination on distributed hosts params = Initialize(); running = true; While (running) { simdata = compute(params); out(simdata, timestep); } params = Initialize(); running = true; While (running) { in(simdata, timestep); visdata = visualize(simdata, params); out(visdata, timestep); } Computing Cluster Vector processor {simdata, 2} Operations: out insert rd blocking read rdp non-blocking read in blocking read and delete inp non-blocking read and delete {visdata, 2} {simdata, 3} {params} PDA CAVE params = Initialize(); running = true; While (running) { params = interaction(); out(params); } params = Initialize(); running = true; While (running) { inp(visdata, timestep); render(visdata, params); }

  29. Rollback Time management • Continuous time • Discrete time • Discrete event • Conservative • Optimistic -> rollback High Level Architecture (HLA, DMSO): • Federates in a federation exchange time-stamped messages (sounds familiar?) • Federates may be a combination of: • Time regulating or not • these advance time at will • Time constrained or not • these request time advance and wait for grant • TSO: Time Stamped Order • Messages may not be timestamped in the past ! ! !

  30. VR interaction techniques • XiVE: X in Virtual Environments • CSSR: Context Sensitive Speech Recognition • GEOPROVE: Geometric Probes for VEs • SCAVI: Speech, CAVE and VTK Interaction

  31. VR interaction techniques XiVE: X in Virtual Environments • There is no standard WIMP toolkit for VEs • XiVE “swallows” GUIs into a VE Allows existing applications to be used in VEs without change.

  32. VR interaction techniques CSSR: Context Sensitive Speech Recognition • Interaction with visual constructs can be hard in a VE • Speech is a different modality • Adding context decreases word-error-rate • Fast, intuitive interaction • “come here”, “make blue”, “increase size by 200%”

  33. VR interaction techniques SCAVI: Speech, CAVE and Vtk Interaction • Direct interaction with Vtk “actors” using pointer or voice • select, drag, scale, rotate, copy, paste, etc. • event handlers; whenin focus, when dragged,when selected, whenspoken to, etc.

  34. VR interaction techniques GEOPROVE: Geometric Probes for VEs Making Measurements in VR • Uses probes consisting of markers

  35. VR interaction techniques GEOPROVE: Geometric Probes for VEs

  36. Let’s look at an example...

  37. Simulated vascular reconstructionin a virtual environment

  38. Overview • Interactive virtual environments for the exploration of • Multi-dimensional datasets • Multi-parameter spaces (computational steering) • Visualization and interaction in Virtual Reality (VR) • Applied to a test case:simulated vascular reconstruction in VR

  39. VRE • The Virtual Radiology Explorer(VRE): • Static exploration of 3D medical datasets • Virtual Reality (VR) interface • CAVE at SARA, Amsterdam • Portable ImmersaDesk • Surface/volume rendering • Virtual endoscopy • PACS data and computinginterface • Data storage and processingon parallel system (IBM SP2)

  40. Vascular disease • Stenosis:Treatment: thrombolysis, balloon angioplasty, stent placement, endarterectomy, bypass • Aneurysm:Treatment: shunt, bypass

  41. The problem • Best treatment often not obvious • read: the parameter space • Human body is a complex structure • read: the data space • A treatment is not always best under all situations • read: combination of both

  42. Pre-operative planning

  43. Traditional treatment of vascular disease

  44. Interactive simulated vascular surgery

  45. Simulated Vascular Reconstruction • Simulated vascular reconstruction • Patient specific angiographydata • Fluid flow simulationsoftware • Simulation of reconstructivesurgical procedure in VR • Interactive visualization ofsimulation results in VR • Pre-operative planning • Explore multiple reconstructionprocedures

  46. Preprocessing • Segmentation of patient specific MRA/CTA scan • Isolates region of interest • Lattice Boltzmann grid generation • Defines solid and fluid nodes, inlet and outlet conditions

  47. Parallel fluid flow simulation • Lattice Boltzmann Method (LBM) • Lattice based particle method • Regular lattice, similar to CT or MRI datasets • Allows irregular 3D geometry • Allows changes at run-time • Velocity, pressure and shearstress calculated fromparticle densities • Non-compressiblehomogeneous fluid,laminar flow • Spatial and temporallocality • Ideal for parallelimplementation

  48. Parallel fluid flow simulation

  49. Interactive exploration in VR • Visualize simulation results • Flow field, pressure, shear stress • Real time

  50. Interactive exploration in VR • Visualize simulation results • Flow field, pressure, shear stress • Real time

More Related