Commodity Clusters for Immersive Projection Environments

1. Commodity Clusters for Immersive Projection Environments • High Performance VR • Jérémie Allard, • ID/IMAG • Jeremie.Allard@imag.fr

2. Goal • From a bunch of PC toward a high performance plateform for intensive VR applications

3. VR Cluster How-To (1) • Our new cluster • First step: install the OS

4. OS Requirements on a Cluster • Distributed environment • Fast installation/update on several nodes • Shared directory (NFS) • User accounts (NIS) • Remote shell service (SSH) • Network access (DHCP, DNS)

5. Linux Cluster Distribution • http://clic.mandrakesoft.com • Based on MandrakeLinux • Preconfigured solution to easily deployand manage a cluster

6. Installation • Simple and Fast Installation • Install one node as a server • Install one slave • Automatic cloning to install all other slaves (200+ PCs in 20 min.) Cluster ready!

7. Tools • Lots of integrated tools: • Communication: MPICH, LAM-MPI • Computing: ATLAS, LAPACK, SCALAPACK, PETSC • Job Managment: OpenPBS • Monitoring: Ganglia • Administration: Ka-tools, pconsole, urpmi –parallel • VR/Graphics: Povray, Net Juggler

8. VR Cluster How-To (2) • Cluster installed • Next step: Connect the projectors and stereo system

9. Time SoftGenLock: Multi-Display Active Stereo • Alternate projection of the right and left eye images synchronized with shutter glasses • Multiple projectors: glasses shutters must be flipped when all the displays are in the vertical blanking period genlock Vertical blanking Projector 1 Left Right Left Projector 2 Left Right Left Left eye Right eye

10. SoftGenLock: Software Implementation • http://softgenlock.sf.net/ • Software approach to enable activestereo and genlock on Linux clusters • Synchronize the video signals • Sequentially display left and right images • Send signal to the glasses (LCD shutters) • Works with potentially any graphics card • New pixel-clock based algorithm (SoftGenLock 2.0) • Master/Slave synchronization using the parallel port

11. SoftGenLock: Master/Slave Algorithm • For each frame: • On the master • send a flip signal to the glasses and the slaves • flip the displayed image • On the slaves • read the master signal • accelerate or slowdown the video signal • flip the displayed image

12. SoftGenLock: Custom Parallel Port Network Slaves Glasses + ground connections Master Can be a simple tweakedcable for a few nodes:

13. SoftGenLock: Requirements • Environments supported • Linux with a standard 2.4 kernel • Useful for testing but genlock may be unstable (jittering) • Real-time kernel: RT-Linux, RTAI (preferred) • Best performance (no jitter) • Require to patch the kernel • Future 2.5/2.6 kernels • expected to have less jittering and much better scheduling precision

14. VR Cluster How-To (3) • Displays ready • Next step: VR Toolkit

15. VR Toolkits:VR Juggler • http://www.vrjuggler.org • Open Source Platform for VR • Abstraction of the I/O devices • Direct access to graphics API: • OpenGL, Performer, Open Inventor, OpenSG • Take advantage of the latest shader technologies • Linux, Windows, IRIX, Solaris, HP Unix • VR Juggler 1.0 released in april 2001, 2.0 currently in active development (first alpha in march 2003)

16. VR Juggler Simulator Mode Performance Monitoring PC + mouse + keyboard On-line Reconfiguration Configuration Files VJ Application Tracker + Displays + … vjUser, vjButton,…

17. VR JugglerConfiguration system • Configuration data are organized in ConfigChunks • JugglerUser Name "User1" headPos { "VJHead" } interocular_distance { "0.229" } end Configuration editor GUI:

18. VR Cluster How-To (4) • VR toolkit installed • Next step: cluster support

19. Clustering Solutions for VR Juggler • Data Lock at • Input Event Level: • Cluster Juggler • Net Juggler • Scene Graph Level: • OpenSG • Graphics Primitive Level: • WireGL

20. Net Juggler • http://netjuggler.sf.net • Extends VR Juggler 1.0 for clusters • Abstract the cluster into one system • Only few modifications to VR Jugglerapplications source code • One configuration for the whole cluster • One operation to launch the application(using command line or GUI)

21. Net Juggler Simulator Mode Performance Monitoring Net Juggler On-line Reconfiguration Configuration Files(add each PC role) VJ Application PC1 -> Tracker + left display PC2 -> Front display PC3 -> Wand + Right display

22. Net JugglerImplementation Configuration events Tracker event Wand events Net Juggler communication layer (data lock) VJ Application VJ Application VJ Application Net Juggler swaplock

23. Net JugglerCommunications • Data in transit (device + config events): • Small amount • Independent of the scene’s graphics complexity • Communication Layer implementation: • Data aggregation • Optimized collective communications Scalable Low bandwidth required Fast Ethernet Network sufficient

24. Net Juggler application launching GUI: NjRun Set/Load/Save a configuration Actual nodes of the cluster Template configuration files : hosts are defined as @pc1@ Application executable • Generate the configuration files and the mpi script • Launch the application • Kill the application

25. Cluster Juggler / VR Juggler 2 • http://www.vrjuggler.org • Currently (july 2003) in development • Data Lock: Input event distribution scheme • Directly integrated in VR Juggler 2.0 • Based on TCP • Heterogeneous cluster support

26. Net Juggler vs Cluster Juggler • Both use similar distribution schemes • Net Juggler is based on VR Juggler 1.0 and MPI • Collective communication for high scalability • Applications can use MPI based distributed computations • Cluster Juggler is based on VR Juggler 2.0 and TCP • Heterogeneous clusters • ApplicationData: API for broadcasting application-specific chunk of data

27. VR Cluster How-To (5) • VR toolkit with cluster support • Next step: performance intensive applications

28. Developing applications for Net Juggler/Cluster Juggler • Performance issues • Replication on all nodes: • Low communication overhead • data and computations duplicated • fps on one PC = fps on a X nodes PC cluster • Performance improvements:distribute data and computations • Application dependant bottleneck • adhoc parallelization • Tradeoff between distribution and extra communication costs

29. Node 1 Node 2 Node 3 Node 4 Distributed Interactive Fluid Simulation • Fluids simulation: Stam’s Navier-Stokes equation solver • 2D grid distributed on the nodes + parallel Navier-Stokes equation solver • Implementation using the PETSc math library (based on MPI)

30. 1 display • 4 displays • Solver duplicated • 8 fps • 8fps • Solver distributed on 4 nodes • 21 fps • 20 fps • Solver distributed on 6 nodes • 24 fps • 22 fps NjFluid • Results: • Dual PIII-800 MHz • GeForce 2 GTS 64 MB • 100 Mbits/s Fast Ethernet Network • 128x128 grid

31. Summary • We combined several tools • Cluster OS: CLIChttp://clic.mandrakesoft.com • Stereo system: SoftGenLockhttp://softgenlock.sf.net • VR Toolkit: VR Jugglerhttp://www.vrjuggler.org • Cluster support: Net Jugglerhttp://netjuggler.sf.net • Distributed computations: PETSChttp://www.mcs.anl.gov/petsc • Everything is Open-Source!

32. VR Cluster How-To (6) • High performance VR plateform • Next step: invent!

33. Advanced Applications Real-time video processing and scene reconstruction Simulation-based design Rich virtual environments Complex virtualenvironments MOVI, INRIA Jin Tech, VRAC

34. Toward Rich Virtual Environments • Integrating multiple simulations in a graphical intensive world • Fluid, Grass/Wind, Fur • Better immersion • Interactive simulations for industrial design • Fast workflow • Require advanced tools • on going work at ID/IMAG