HLA Grid Based Support for Simulation of Vascular Reconstruction

1. HLA Grid Based Support for Simulation of Vascular Reconstruction Katarzyna Rycerz, Marian Bubak,Maciej Malawski, Peter M.A. Sloot ICS AGH, Kraków & UvA, Amsterdam

2. Outline • Motivation • Grid HLA Management System (G-HLAM) • HLA-Speaking Service for multiple processes • G-HLAM with surgical vascular reconstruction application • Results • Conclusions and future work

3. Interactive simulation HLA Grid HLA Management System Grid Infrastructure (Grid Services,GRAM, GSI, MDS) Grid HLA Management System (G-HLAM) Motivation • A system designed for management and steering of HLA-based distributed interactive simulations in a non-dedicated, shared and unpredictable Grid environment • Allows for application performance to achieve the most efficient execution on the available resources in the Grid. • Uses the High Level Architecture (HLA) - the standard for distributed interactive simulations

4. G-HLAM Architecture User Site A Application Benchmark Application Broker HLA Speaking Federate Services Monitoring Client Service Code Services RTIExec Migration Service Service Site B Broker Infrastructure Service Monitoring Migration HLA Services HLA Performance Support Interfacing Bus Decision Services Services Broker Support Service Services Application N - th Grid site supporting HLA Monitoring Main Service Manager Broker Support HLA Speaking Services Site Site C C Service RTIExec HLA Registry Registry Migration Interfacing Service Service Support Services RTIExec Services Service Grid site supporting HLA • HLA interfacing services • HLA-Speaking Service for managing federates • (partly presented on GAMW’03 and DSRT’03) • RTIExec Service for managing RTIExec (coordination process in RTI) • Broker for setting up a federation and making migration decisions • Broker decision services • Registry for storinglocation of HLA-speaking services • Infrastructure Monitoring/Benchmarks for checking environment of HLA service • (presented on ICCS’04) • Application Monitoring for monitoring performance • (presented on DSRT’04) • Migration support services • Migration Servicefor performing migration • Migration partly presented at GAMW’03 and DSRT’03

5. Site B Site A Site C HLA-Speaking Service RTIExec Service HLA-Speaking Service Management level Grid Service Communication RTI bus Communication Application level RTI Exec Fed Fed Fed Fed Fed Functionality of HLA-Speaking Service • Manages execution of legacy HLA federates on a single site • Submits federates on its site and forwards saving/restoring requests • Two kinds of HLA-Speaking Services were created: • For single federate process (presented on GAMW’03, DS-RT’03) • For multiple federate processes (presented here)

6. HLA-Speaking Service multiple processes HLA-Speaking Service Grid Service Porttype • Command for submission based on globus RSL • GRAM used for actual submission • Control federation for sending save and restore commands • GridHLA Controller library as an interface between user code and G-HLAM HLA-Speaking Porttype Process 1 Process N Start GHCL GHCL GHCL GHCL Save state GHCL GHCL Restore Control Federate Control Federate Control Federate User Federate 1 User Federate N save, restore control federation at site A user federation HLA-Speaking Service Grid Service Porttype HLA-Speaking Porttype Process1 Process M Start GHCL GHCL GHCL GHCL Save state GHCL GHCL Restore Control Federate Control Federate Control Federate User Federate 1 User Federate M control federation at site B save, restore

7. migrate migrate from A to B startControl requestMigrationSave save save resign user fed resign control fed save done save done take and submit get codes and checkpoints restore join application fed join control fed restore done done done done Federate Migration multiple processes HLA Speaking Service site B HLA Speaking Service site A GridFTP Server Site A Federates site B Federates site A Monitoring Service Broker Service RTI Migration Service

8. Possible HLA Migration Support Approaches How to migrate one federate without disturbing others ? • Develop new HLA standard implementation • Transparent, but sticks to particular implementation • Use HLA Management Object Model to save federates state • No support for user state, MOM information can get old (no state coherency) • Use HLA save/restore API • Assures internal RTI state coherency (and HLA RTI specific services), but no support for user data • Sophisticated migration algorithms • [Z. Yuan at al. - ICCS 2004]– efficient, but require from a user to port his HLA application to some high level library

9. Grid HLA Controller Library (GHCL) Design • We have chosen to: • Use HLA save API • Build runtime support to facilitate that API • Build runtime support to facilitate saving user data • GHCL contains functions to: • Start up and connect to RTI API classes • Check if external (Migration Service) save/restore request came • Check if internal (RTI) save/restore request came • Save/restore user values

10. X-ray angiography or CT or MRI scan Surgerical Vascular Reconstruction Application • Analysis and segmentation module -automaticallyfinds the lumen border between the blood and non-blood in input images • 3D editing tool - allows for editing stereoscopic images and executing experimental visualisation studies on realistic arteliar geometries • Simulation module - a parallel computationalsolver that computespressure, velocities, and shear stresses of blood flow. The simulator is based on the Lattice-Boltzmann method (LBM). • Visualisation and exploration module - uses a Virtual Reality Explorerwhere the patient's data is visualized as a 3D stereoscopic imagetogether with the graphical interpretation of thesimulation results. The user can then manipulate the 3Dimages of arteries, patient's body and blood flowstructures in virtual reality. modify bypass raw data analisys and segmentation control Visualisation and exploration In VE 3D editing tool Simulation data data User data data reanalise

11. Collaborative Environment for Vascular Reconstruction Application • Many users (surgeons) can run many simulations • Every user can: • stop/pause the chosen simulation during runtime (the simulation that is owned by him), • switch between outputs from different simulations (as output can be received from only one - so-called foreground - simulation at the time), • switch ownership of simulations (ability to stop/pause a particular simulation), • restart with a different bypass, • reanalyze raw data and reset segmentation

12. Collaborative Environment for Vascular Reconstruction Application reanalise modify bypass User A raw data analisys and segmentation Simulation Owner A, foreground data data data Visualisation And exploration In VE control 3D editing tool control data data Simulation Owner A background data control switch simulation control change simulation owner Communication bus (HLA-RTI) raw data analisys and segmentation control Visualisation And exploration In VE Simulation Owner B foreground data data data data 3D editing tool control data data Simulation Owner B background switch simulation control control change simulation owner modify bypass reanalise User B

13. Useful Features of HLA • supports building geographically distributedsystem, • allows for dynamic joining of simulation/visualisation to the application- useful e.g. for late arriving simulations, • allows for easy switchingbetween background and foreground simulations using HLA subscribe/unsubscribe mechanism, • allows for dynamic resigning of simulation/visualisation from application - the user can quit the application or stop some of simulations without disturbing others, • supports notification- useful for notifying the simulationabout the user commands i.e. pause, cancellation, • allows for easy switching between ownership- useful when a user that created a simulation wants to give its control away to a different user, • allows for concurrency control -- related to ownership -- assures thatonly one user can control the simulation at the time.

14. Grid for Collaborative Environment of Vascular Reconstruction • different modules require different resources: • quick access to images' database (segmentation tool), • computational power (flow simulation), • specyfic Virtual Reality hardware (editing and visualisation tools), • if more simulations need to be run concurrently, one site with computational power is not sufficient, • there is a need to support many visualisations (users) locatedin different places wanting to observe the same simulation

15. G-HLAM in Vascular Reconstruction Application Site M Site N simulation • Simulation consists of a number of MPI processes • In each simulation, a root MPI process is connected with other simulations and visualisations via HLA application federation as in HLA legacy systems • Each process is connected with G-HLAM by HLA Speaking Service via control federation • HLA-Speaking Service interfaces application to G-HLAM simulation User B HLA-Speaking Service visualisation and exploration HLA-Speaking Service HLA RTI (application federation) visualisation and exploration simulation Process 0 Process N User A MPI G-HLAM HLA (control federation) HLA-Speaking Service Site P

16. Testbed Infrastructure

17. Results • Type of module (simulation root process or visualisation process) does not impact migration overhead • 12 MPI processes in each simulations • 52000 velocity vectors of blood flow in 3D space • GTv 3.2 and HLA RTI 1.3v5 Migration overhead Migration impact on simulation performance Type of modules: 1 migrated simulation and 25 visualisations Type of modules: 1 migrated simulation, 4 other simulations, varied number of visualisations

18. Conclusions HLA-Speaking Serviceenables efficient management ofthe execution of HLA federates on the Grid: • provides universal interface for user HLA federates to G-HLAM, • used for running, saving and restoring one or more federate processes on the Grid site on which it resides, • the whole Grid site is efficiently used by taking advantage of GRAM interface to local job manager, • GridHLAController library is an easy to use API for interaction of user codes with the HLA-Speaking Service, • HLA standard advanced features useful for distributed simulation are ported to the Grid, • legacy HLA applications are adapted to the Grid environment in a robust and efficient way.

19. Future Work • Possible extension of the G-HLAM functionality • Broker Service • many already exist • Performance Decision Service • Fault tolerance • migration mechanisms is a first step we already have • Technological migration • WSRF • Component technologies

20. Application Architecture • N-body simulation of dense stellar system parallelized with MPI • Requires variable time step in simulation (i.e. for binary stars) • Many stars to calculate (order 10^6) • requires computational power • A user can change stars parameters at any time during simulation

21. Impact of Migration 20 MPI Processes, 24000 stars

22. Migration Stages 5000 stars

23. reanalise modify bypass User A raw data analisys and segmentation Simulation Owner A data data data Visualisation And exploration In VE control 3D editing tool control data data Simulation Owner A data control switch simulation control change simulation owner Communication bus (HLA-RTI) raw data analisys and segmentation control Visualisation And exploration In VE data data Simulation Owner B data data 3D editing tool control data data Simulation Owner B switch simulation control control change simulation owner modify bypass reanalise User B