Interactive Exploration in Virtual Environments

1. Interactive Exploration in Virtual Environments Robert G. Belleman Universiteit van Amsterdam Email: robbel@science.uva.nl

2. Overview • Large data spaces • Interactive exploration environments • Objectives • Design • An example: • Simulated vascular reconstruction in a virtual environment • MSc projects

3. Data explosion • High Performance Computing: • computing power increases (Moore’s “Law”) • storage capacity increases Result: “data spaces” get increasingly largeand complex; multi-dimensional, time dependent.

4. What are “data spaces”? Roughly speaking: • Data sets • the “files” or “states” that are generated • Parameter spaces • the number of “variables” and their allowed “freedom” (range, resolution) in a program

5. Large data sets: examples • Medical images (i.e. CT, (f)MRI, PET) • 512 x 512 at 16 bits slices in common use • tens to hundreds slices: 5Mb to5Gb per scan • 1024 x 1024 at 16 bit in verynear future: 20Mb to 20Gbper scan • time variant scans

6. … more examples ... • Simulation experiment results • FEM, MD, lattice Boltzmann • often dimensionality > 3(e.g. time variant) • multiparameter data fields • gigabytes of data per “run”

7. … more examples ... • Measurements • high-speed data acquisition devices: particle accelerators (Large Hadron Collider, LHC), microbeam scanners, DNA scanners, CLSM • Financial data, etc. Terabytes of data per experiment is no longer an exception!

8. Parameter spaces • Simulation of complex systems • intractable: a certain timestep in a simulation can only be reached by starting at t=0 • NP complete: time and space requirements grow exponentially with problem size • Explicit simulation by a guided search through parameter space required (non-deterministic algorithms: SA, CA, NN, LBM, etc.)

9. Parameter spaces: examples • Molecular dynamics • picosecond timeresolution • Molecular docking involves searchthrough large problemspaces

10. … more examples … • Finite Element Methods (FEM) • large (hierarchically) structured meshes • Lattice Boltzmann methods (LBM) • large 3D (hierarchically) structured grids • large parameter spaces • “Optimization” problems in general

11. From data to knowledge • Analysis of data spaces is often difficult • no analysis methods known, or ill-posed • size of data sets too large or too complex Often data analysis or simulation runs can take days, sometimes weeks!

12. Bring in the expert Presentation is often the only way to obtain insight (note: not limited to visualization) Is it possible to make short cuts? E.g. by putting “an expert” in the loop?

13. Human In The Loop • HITL • a.k.a. “interactive exploration” • a.k.a. “exploratory analysis” • a.k.a. “computational steering” • a.k.a. “problem solving environments” • a.k.a. “virtual laboratory”

14. Interactive Exploration Environments Objective: to provide an interactive environment that allows for the exploration of large data spaces. Responsive, fast, accurate, reliable, easy-to-use, scalable, efficient, versatile, etc. The design of Interactive Exploration Environments is very multidisciplinary.

15. Prerequisites • Intuitive interaction • increased functionality requires a well considered user interface • Real-time feedback: < 0.1 sec delay • Quality presentation • Informative, avoid clutter • Why Virtual Reality? • Rapid update rate: for continuous perception: • > 10 fps for vision • > 20 cps for sound • > 1000 cps for haptics These often conflict one another.

16. Design • MVC (Model View Controller)design pattern Model Data values Computation View Translators Visualization State request Representation State Update state Update view Controller Session control Interaction User input User

17. Interactive Dynamic Exploration Environments (IDEE) • Exploring dynamically changing data from “living” simulations • Changing parameters: What if...? • Requires “coordination”:

18. Coordination params = Initialize(); running = true; While (running) { simdata = compute(params); visdata = visualize(simdata, params); render(visdata, params); params = interaction(); } Synchronous (lockstep)

19. Coordination params = Initialize(); running = true; While (running) { simdata = compute(params); } params = Initialize(); running = true; While (running) { visdata = visualize(simdata, params); } params = Initialize(); running = true; While (running) { params = interaction(); } params = Initialize(); running = true; While (running) { render(visdata, params); } Asynchronous

20. Coordination params = Initialize(); running = true; While (running) { simdata = compute(params); } params = Initialize(); running = true; While (running) { visdata = visualize(simdata, params); } params = Initialize(); running = true; While (running) { params = interaction(); } params = Initialize(); running = true; While (running) { render(visdata, params); }

21. Shared memory Coordination on a single host params = Initialize(); running = true; While (running) { simdata = compute(params); } params = Initialize(); running = true; While (running) { visdata = visualize(simdata, params); } params simdata Question: What is wrongwith this method? visdata params = Initialize(); running = true; While (running) { params = interaction(); } params = Initialize(); running = true; While (running) { render(visdata, params); }

22. Coordination on a single host params = Initialize(); running = true; While (running) { P(simdata_sema, params_sema); simdata = compute(params); V(simdata_sema, params_sema); } params = Initialize(); running = true; While (running) { P(simdata_sema, visdata_sema, params_sema); visdata = visualize(simdata, params); V(simdata_sema, visdata_sema, params_sema); } Shared memory params semaphores simdata visdata params = Initialize(); running = true; While (running) { P(params_sema); params = interaction(); V(params_sema); } params = Initialize(); running = true; While (running) { P(visdata_sema, params_sema); render(visdata, params); V(visdata_sema, params_sema); }

23. Coordination on distributed hosts params = Initialize(); running = true; While (running) { simdata = compute(params); } params = Initialize(); running = true; While (running) { visdata = visualize(simdata, params); } Computing Cluster Vector processor Peer-to-peer networkconnections But communicationtakes time (precioustime)… PDA CAVE params = Initialize(); running = true; While (running) { params = interaction(); } params = Initialize(); running = true; While (running) { render(visdata, params); }

24. Communication latency hiding • Multiple connections • Waiting for acknowledgements is hidden • Packets can travel through different routes

25. Payload reduction • Data encoding • Decreases level of detail • Accuracy determined by type of representation • Lossy compression • Induces latency • “Standard” compression libraries (zlib) • Reduction to 10% not uncommon • Lossless compression • Induces latency

26. Communication pipeline • Cascade of encoding, compression and multiple socket connections

27. Pipeline performance Latency hiding: Payload reduction:

28. Tuple space Coordination on distributed hosts params = Initialize(); running = true; While (running) { simdata = compute(params); out(simdata, timestep); } params = Initialize(); running = true; While (running) { in(simdata, timestep); visdata = visualize(simdata, params); out(visdata, timestep); } Computing Cluster Vector processor {simdata, 2} Operations: out insert rd blocking read rdp non-blocking read in blocking read and delete inp non-blocking read and delete {visdata, 2} {simdata, 3} {params} PDA CAVE params = Initialize(); running = true; While (running) { params = interaction(); out(params); } params = Initialize(); running = true; While (running) { inp(visdata, timestep); render(visdata, params); }

29. Rollback Time management • Continuous time • Discrete time • Discrete event • Conservative • Optimistic -> rollback High Level Architecture (HLA, DMSO): • Federates in a federation exchange time-stamped messages (sounds familiar?) • Federates may be a combination of: • Time regulating or not • these advance time at will • Time constrained or not • these request time advance and wait for grant • TSO: Time Stamped Order • Messages may not be timestamped in the past ! ! !

30. VR interaction techniques • XiVE: X in Virtual Environments • CSSR: Context Sensitive Speech Recognition • GEOPROVE: Geometric Probes for VEs • SCAVI: Speech, CAVE and VTK Interaction

31. VR interaction techniques XiVE: X in Virtual Environments • There is no standard WIMP toolkit for VEs • XiVE “swallows” GUIs into a VE Allows existing applications to be used in VEs without change.

32. VR interaction techniques CSSR: Context Sensitive Speech Recognition • Interaction with visual constructs can be hard in a VE • Speech is a different modality • Adding context decreases word-error-rate • Fast, intuitive interaction • “come here”, “make blue”, “increase size by 200%”

33. VR interaction techniques SCAVI: Speech, CAVE and Vtk Interaction • Direct interaction with Vtk “actors” using pointer or voice • select, drag, scale, rotate, copy, paste, etc. • event handlers; whenin focus, when dragged,when selected, whenspoken to, etc.

34. VR interaction techniques GEOPROVE: Geometric Probes for VEs Making Measurements in VR • Uses probes consisting of markers

35. VR interaction techniques GEOPROVE: Geometric Probes for VEs

36. Let’s look at an example...

37. Simulated vascular reconstructionin a virtual environment

38. Overview • Interactive virtual environments for the exploration of • Multi-dimensional datasets • Multi-parameter spaces (computational steering) • Visualization and interaction in Virtual Reality (VR) • Applied to a test case:simulated vascular reconstruction in VR

39. VRE • The Virtual Radiology Explorer(VRE): • Static exploration of 3D medical datasets • Virtual Reality (VR) interface • CAVE at SARA, Amsterdam • Portable ImmersaDesk • Surface/volume rendering • Virtual endoscopy • PACS data and computinginterface • Data storage and processingon parallel system (IBM SP2)

40. Vascular disease • Stenosis:Treatment: thrombolysis, balloon angioplasty, stent placement, endarterectomy, bypass • Aneurysm:Treatment: shunt, bypass

41. The problem • Best treatment often not obvious • read: the parameter space • Human body is a complex structure • read: the data space • A treatment is not always best under all situations • read: combination of both

42. Pre-operative planning

43. Traditional treatment of vascular disease

44. Interactive simulated vascular surgery

45. Simulated Vascular Reconstruction • Simulated vascular reconstruction • Patient specific angiographydata • Fluid flow simulationsoftware • Simulation of reconstructivesurgical procedure in VR • Interactive visualization ofsimulation results in VR • Pre-operative planning • Explore multiple reconstructionprocedures

46. Preprocessing • Segmentation of patient specific MRA/CTA scan • Isolates region of interest • Lattice Boltzmann grid generation • Defines solid and fluid nodes, inlet and outlet conditions

47. Parallel fluid flow simulation • Lattice Boltzmann Method (LBM) • Lattice based particle method • Regular lattice, similar to CT or MRI datasets • Allows irregular 3D geometry • Allows changes at run-time • Velocity, pressure and shearstress calculated fromparticle densities • Non-compressiblehomogeneous fluid,laminar flow • Spatial and temporallocality • Ideal for parallelimplementation

48. Parallel fluid flow simulation

49. Interactive exploration in VR • Visualize simulation results • Flow field, pressure, shear stress • Real time

50. Interactive exploration in VR • Visualize simulation results • Flow field, pressure, shear stress • Real time