Brookhaven Science Associates U.S. Department of Energy

1. Information Technology Division Interactive Parallel Visualization for Commodity-Based Clusters S.Tomov, R.Bennett, M.McGuigan, A.Peskin, G.Smith, J.Spiletic Brookhaven Science Associates U.S. Department of Energy

2. Objectives • Develop visualization : • with real time interaction (high performance); • for large sets of data (1GB and larger); • high resolution (but cost efficient). • Today’s technological advances • increasingly complicated mathematical models; • masses of data produced by devices such as MRI and laser-microscope scanners, etc.; • the increasing need for interactive extraction of insight from extremely large data sets. Motivated by

3. Some Applications MRI Mouse Brain volume visualization (512 x 256 x 256) Material Micro-Geometry Studies crushed rock volume visualization (1007 x 1007 x 256) Fluid Flow isosurface visualization

4. Approach • Remote parallel rendering to a large tiled display; • Clusters connected with high speed network; • Extend and combine already existing APIs: • Chromium; • Open Inventor; • VTK, etc. • Alternatives • High resolution flat panels driven by a single workstation; • Faster machines; • Build entire visualization system from scratch.

5. ivview window interface display with 4 tiles An Example Mouse brain volume visualization (512 x 256 x 256)

6. The Visualization Pipeline OpenGL Applications Rendering Servers # 1 # 1 # 2 # 2 • Tiled Display OpenGL calls • • • • • # M # N Mouse (and keyboard input) High Speed Network

7. Chromium Overview • Based on WireGL, developed at Stanford University; • Open-source API providing : • OpenGL interface for rendering on commodity-based clusters; • scalable display technology. • How does it work? Accelerate the data flow (1) simplified network protocol • OpenGL calls (2) “bucketing” Rendering Servers • (3) state tracking • + display lists

8. Chromium Parallelization Model (CPM) • glXMakeCurrent(getDisplay(), getNormalWindow(), getNormalContext()); • if (clearFlag) • glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT); • glBarrierExecCR(MASTER_BARRIER); • do sequential OpenGL rendering of the local scene • glBarrierExecCR(MASTER_BARRIER); • if (swapFlag) • glXSwapBuffers(getDisplay(), getNormalWindow()); • else • glXSwapBuffers(getDisplay(), CR_SUPPRESS_SWAP_BIT); • Display is render: • - first :clearFlag = swapFlag = 0, and both 1 for proc. 0 • - last : clearFlag = 1, swapFlag = 0, except swapFlag = 1 for proc. 0

9. Open Inventor extension • Open Inventor : an open source API of C++ objects and methods for interactive 3D graphics applications. Extend to support interactive Cluster Visualization (1) Apply the CPM to the Open Inventor rendering method: SoXtRenderArea::redraw() (2) Implement the “ParaMouse” parallel user interface (the idea sketched on slide #6) or extend the Open Inventor GUI next slide

10. Open Inventor GUI extension Extend callback functions triggered by mouse, keyboard, etc. MPI_Isend the invoked user input in the Master node to the Slave nodes • We want the GUI that ivview provides (modify main()): • declare processor with rank 0 as Master; • if (Master) • run GUI as implemented in ivview; • else • listen for instruction from the Master; SoDB::init(); // Initialize Inventor SoNodeKit::init(); SoInteraction::init(); SoSeparator *root = new SoSeparator; // Build the Inventor’s objects and SGs root->ref(); readScene(root,files); SoExaminerViewer *viewer = new SoExaminerViewer(crrank); viewer->SetSceneGraph(root); crctx = crCreateContextCR(0x0, visual); // Chromium Initialization crMakeCurrentCR(crwindow, crctx); glBarrierCreateCR(MASTER_BARRIER, crsize); glEnable(GL_DEPTH_TEST); viewer->mainLoop();

11. The Result GUI Tiled display ivview window interface • Performance (discussion): • Beowulf cluster, 4 nodes, each with 2 Pentium III processors, 1GHz each Quadro2 Pro graphics cards, communications through 100Mb/s fast Ethernet or 1Gb/s fiber optic network (machine performance [8]); • consider static scene: • spheres with spikes : 11,540 triangles

12. Performance, experiences and conclusions • for the parallel model considered performance is problem specific; • GUI communications time is negligible compared to the visualization time, leading to scalability results in [2, 3, 6]; • can send approximately 5,000 “small” messages/s (see [8]); • Chromium automatically minimizes data flow, except for “geometry” flow : consider animation with small static scenesRendering 2 spheres : Sequential 2 CPU, 1 tile 2 CPU, 2 tiles 117 f/s 3 f/s 6 f/s 2,709,130 tr./s Minimize geometry flow using display lists 80 f/s 90 f/s

13. Other experiences and conclusions • Fast sequential visualization often relies on • graphics acceleration hardware; • data sampling methods; • static scenes; • data size within the hardware limitations; • The proposed visualization system (as a remote visualization) • does not require expensive graphics acceleration; • high resolution and interactive speed with ray casting (no need of data sampling); • facilitates well the visualization of time-varying scenes; • can handle extremely large data sets. VTK : a good candidate for parallel GUI extension

14. VTK and its extension • VTK : an open source API for visualization and image processing (see [11]); Extend to support interactive Cluster Visualization (as on slide 9) (1) Apply the CPM to VTK’s rendering methods : done by David Thompson, SNL; (2) Implement the “ParaMouse” parallel user interface (as shown on slide 6) or extend the VTK’s GUI : Extend vtkXRenderWindowInteractor by: (1) implement “pure” X Windows GUI (no GL calls); (2) add MPI send/receive calls for mouse & keyboard events.

15. Conclusions • We developed Interactive Parallel Visualization by • remote parallel rendering to a large tiled display; • commodity-based visualization clusters connected with high speed network; • extending APIs. • The proposed idea works • Open Inventor; • VTK. • Achieved objectives • real time interaction (high performance); • for extremely large sets of data; • high resolution • inexpensive : easy to develop.

16. References [ 1] J.Ahrens, C.Law, K.Martin, M.Papka, W.Schroeder, A Parallel Approach for Efficiently Visualizing Extremely Large, Time-Varying Data Sets, LANL, TR #LAUR-00-1620. [ 2] I.Buck, M.Eldridge, P.Hanharah, G.Humphreys, Distributed Rendering for Scalable Displays, Proceedings of Supercomputing, 2000. [ 3] I.Buck, P.Hanharah, and G.Humphreys, Tracking Graphics State for Networked Rendering, Proceedings of SIGRAPH/Eurographics Workshop on Graphics Hardware, August 2000. [ 4] Eric S. Brugger (LLNL), Visit: A Component-Based, Parallel Visualization Package, DOE, Computer Graphics Forum 2001, Internet address (accessed on 08/2002): http://www.emsl.pnl.gov:2080/docs/doecgf2001/abstractlist.html [ 5] W. Gropp, Tutorial on MPI: The Message-Passing Interface, Mathematics and Computer Sci. Division, ANL, Internet address (accessed on 06/2002): http://www-unix.mcs.anl.gov/mpi/tutorial/index.html. [ 6] P.Hanharah, H.Igely, G.Stroll, The Design of a Parallel Graphics Interface, Proceedings of SIGRAPH, 1998. [ 7] G. Humphreys, Chromium Documentation, Version BETA, Internet address (accessed on 06/2002): http://graphics.stanford.edu/~humper/chromium_documentation/ [ 8] V. Mirinavicius, Investigation of MPI performance in Parallel Linear Algebra Software on Linux Beowulf Supercomputers, BNL, Technical Report (to appear). [ 9] Kitware Inc., ParaView, Internet address (accessed on 08/2002): http://public.kitware.com/ [10] NASA, Jet Propulsion Laboratory, ParVox, A Parallel Splatting Volume Rendering System, Internet address (accessed on 08/2002): http://alphabits.jpl.nasa.gov/ParVox/ [11] W.Schroeder, K.Martin, L.Avila, C.Law, The VTK User’s Guide, Kitware, Inc. [12] J. Wernecke, The Inventor Mentor : Programming Object-Oriented 3D Graphics with Open Inventor, Addison-Wesley Publishing Company, NY, November 1995. [13] Open Inventor C++ Reference Manual: The Official Reference Document for Open Inventor, Release 2, Addison-Wesley Publishing Company, January 1994.