1 / 26

Grid computing for medical Monte Carlo application

Grid computing for medical Monte Carlo application. Giorgio Russo 1 , R. Barbera 2 , G.A.P. Cirrone 3 , G. Cuttone 3 , F. Di Rosa 3 , E. Giorgio 2 , S. Lo Nigro 2 , F. Romano 3 , M.G. Sabini 4 Consorzio COMETA – INFN LNS Università di Catania, INFN sezione di Catania LNS-INFN

smcculloch
Download Presentation

Grid computing for medical Monte Carlo application

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grid computing for medical Monte Carlo application Giorgio Russo1, R. Barbera2, G.A.P. Cirrone3, G. Cuttone3, F. Di Rosa3, E. Giorgio2, S. Lo Nigro2,F. Romano3, M.G. Sabini4 Consorzio COMETA – INFN LNS Università di Catania, INFN sezione di Catania LNS-INFN A. O. Cannizzaro di Catania Grid Open Day al LNS Catania, 23.01.2008 giorgiorusso@lns.infn.it

  2. Outline • Monte Carlo Method – GEANT4 toolkit • Our Applications: • HadronTherapy, CATANA beam line • GammaKnifeRS, Gamma Knife • Monte Carlo Parallelization • How Grid works • Grid technology implementation • Grid problems • Our experiences • Results Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  3. Monte Carlo Method – GEANT4 toolkit In our applications we have used the GEANT4 Toolkit • Advantages: • Detailed geometry description • Precise magnetic/electric field description • Easier material composition • Variety of physics processes • Precise visualization • Extendibility • Open Source • Disadvantage: • Learning curve • It is not a black box to give correct results • Time run Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  4. Simulation of CATANA beam line HadronTherapy is a GEANT4 Advanced Example, It is a free application downloadable from GEANT4 home page http://geant4.web.cern.ch/geant4/ All components of CATANA beam line are simulated HadronTherapy can be used to simulate and to optimize a generic eye proton therapy beam line Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  5. HadronTherapy At the Isocenter a detector is simulated to reconstruct the Bragg Peak and the Spread Out Bragg Peak The simulation was used to validate the Treatment Planning Software, employed at CATANA facility EyePlan, in a clinical configuration Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  6. Simulation of Gamma Knife Gamma Knife is a stereotactic radiosurgery for brain tumor Stereotactic radiosurgery is a technique for treating intracranial lesions with a single high dose of ionizing radiation It uses the convergence of 201 beams from 60C0 radioactive sources Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  7. Simulation of Gamma Knife • All physical and geometrical characteristics are simulated • The application has been validated comparing the results with the experimental ones • The application can be used: • to verify the dose distribution inside the patient brain • to verify and understand the limitations of the Treatment Planning System (TPS) Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  8. Clinical Treatment Simulation Air Cube Water Air Tumor Bone Shell • The TPS ignores tissue inhomogeneities by assuming the human head as water equivalent • Simulation of a cavernous sinus meningioma treatment • It is simulated on an inhomogeneous phantom Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  9. Clinical Treatment Simulation • The main drawback for a Monte Carlo application is the long calculation time • needed to acquire an adequate level of accuracy. • For example: • HadronTherapy needs 10 hours to obtain a good information about dose distributions. • ( 1 Mega protons using a Pentium 4 1.6 GHz ) • GammaKnifeRS needs 21 days to obtain a good information about dose distributions for a single shot. • ( 1 Billion photons using a Pentium 4 1.6 GHz) BUT • Photon histories are completely independent from each other SO • Parallel computing is therefore particularly adequate Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  10. Parallelization • The large number of histories required is divided into shorter simultaneous subtasks • The subtasks are generated with different seeds to ensure that the histories are statistically independent • The reduction time is related to number of simultaneous subtasks A cluster could be used for this issue but GRID is the best solution !! Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  11. How Grid works Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  12. Matchmaking & Job Run • The Resource Broker • Looks for user requirements to run the job • Obtains the list of computing element (CE) matching user requirements • Chooses the best one and submits the job • The Computing Element • Runs the job • Generates the output Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  13. Job Description Language • A JDL file has attributes to specify: • the InputSandbox and the OutputSandbox, for the input and output files • how the job has to run setting “Executable” and ”Arguments” • Requirements • If there are some files stored in the SE Type = "Job"; JobType = "Normal"; Executable = "/bin/sh"; StdOutput = "hadrontherapy.out"; StdError = "hadrontherapy.err"; InputSandbox = {"hadrontherapy.sh"}; OutputSandbox = {"hadrontherapy.err","hadrontherapy.out","piccoGrid.dat","disAngGrid.dat","disEnGrid.dat"}; RetryCount = 7; Arguments = "hadrontherapy.sh"; Requirements = Member("GEANT4-6",other.GlueHostApplicationSoftwareRunTimeEnvironment); Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  14. Job Status • Submitted: the job is submitted from the user but it is not processed by RB yet • Waiting: the job is in the RB and it is waiting to be processed • Ready: the job is processed by RB and it is waiting to be sent to a CE • Scheduled: the job is in the CE and it is waiting to be run • Running: the job is running • Done: the job is finished successfully • Aborted: the job is aborted by the Workload Management System • Cancelled: the job is cancelled by the user • Cleared: the job is done and the output is recovered by the user Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  15. Status of computing element (CE) and storage element (SE) **************************************************************** These are the related data for gridit: (in terms of queues and CPUs) **************************************************************** #CPU Free Total Jobs Running Waiting ComputingElement ---------------------------------------------------------- 10 10 0 0 0 dgt01.ui.savba.sk:2119/jobmanager-lcgpbs-long 16 16 0 0 0 grid010.ct.infn.it:2119/jobmanager-lcgpbs-long 10 10 0 0 0 grid011f.cnaf.infn.it:2119/jobmanager-lcgpbs-long 6 6 0 0 0 grid004.iucc.ac.il:2119/jobmanager-lcgpbs-infinite 16 16 0 0 0 grid010.ct.infn.it:2119/jobmanager-lcgpbs-short …… ************************************************************** These are the related data for gridit: (in terms of SE) ************************************************************** Avail Space(Kb) Used Space(Kb) Type SEs ---------------------------------------------------------- 912356296 51120 grid005.iucc.ac.il 143511344 2509092 cn02.be.itu.edu.tr 68930000 6200000 egee016.cnaf.infn.it 26700000 8010000 lxcde07.pd.infn.it …. Example of a: List of available CEs for a VO with the number of CPU and the Job in running or waiting List of available SE for a VO with the available and used disk space Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  16. Grid Technology Implementation (1) • The Grid Technology Implementation in a GEANT4 application does not need to modify any line code • First step is to install in some CE the GEANT4 libraries • There are two possible solutions: • To make the RPM of the ToolKit, like Geant4 • To import, using a tar file, only the libraries that the application calls during the run • Second step is to publish the TAG, that you will use as requirements in your jdl file Requirements = Member("GEANT4CT“ ,other.GlueHostApplicationSoftwareRunTimeEnvironment); Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  17. Grid Technology Implementation (2) • Third step is to make a bash script to distribute the simulation work in many jobs • Generally some hundred of jobs • Fourth step is to make a bash script to get output automatically, it: • verifies the job status • counts the number of job “Done” • gets the output when the number of job done crosses the threshold chosen Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  18. Grid Technology Implementation (3) ###Submitted Job Ids### https://grid014.ct.infn.it:9000/buovpo3IU824wnwMByMIgA https://grid014.ct.infn.it:9000/fi0MAH-qxfTWIxNnE5PHaA https://grid014.ct.infn.it:9000/gi63k_dC77H1lHPgG0OBeg https://grid014.ct.infn.it:9000/j0HVFj6L0Y5gTRXk3_CGVA https://grid014.ct.infn.it:9000/XiWJ1wBIxgEFId5FMXYxlw https://grid014.ct.infn.it:9000/PR-7FdD6-IW2OPV7N_rW8w https://grid014.ct.infn.it:9000/ztGb9cTpUHPhtFjZ1vrn0w https://grid014.ct.infn.it:9000/svso4aGy0flDi6z_jCMqYA ….. outputGRID/ russo_1ahuQWRpgMw4T4EZ8NHLRw russo_2kfjVOUYhz4l_xTradovew russo_8QJeNESyu18k_zZ3QUaoMA russo_96lwgOBuEhzj2mUcVad29g russo_aKw-LcyrVLA9Xw_N-0aJTw russo_B6dXBLBU0V04uMMYxnTZMQ russo_E14M-4BBQfPy2GYgCNG7Ww russo_EPxSVrT8VELFeYcHwDA2pw russo_GCoxwYVd1JWwkAJLeby_xw russo_HaxrPZOI6Xrg4v9LKAxzRQ russo_jGHzkOKq_dTI9muJ-FewNQ russo_JHRXWY7muzpEezwk6HPY9Q ……. List of Job Submitted Identify Number List of Output Directories • Fifth step is to make a C++ or Matlab application to sum the single job output, a matrix Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  19. Grid Problems • The implementation of the Grid technology needs to change the methodology to split the work • The nodes that provide CPU cycles in a Grid system have more likely to vary greatly than a cluster system • A node may go down or become inaccessible without notice while it is working on its task • A slow node might become the bottleneck of the whole computation • Each subtask is a “key” subtask Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  20. N-out-of-M strategy • In a Monte Carlo application we must achieve a certain accuracy • We do not much care which random sample set is estimated, provided that all the random samples are independent in a statistical sense • The statistical nature of Monte Carlo applications allows us to enlarge the actual size of the computation • Increasing the number of subtasks from N, classic method, to M • Therefore it is not necessary to wait for all M subtasks to finish • M value is chosen considering the Grid performance Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  21. Our Experience - GammaKnifeRS • We have used the COMETA resources • We have divided the work into 350 subtasks ( M value ) • The sufficient number of subtasks to obtain the required accuracy is 336 ( N value ) • We have reduced the time run from several days to 16 hours In this way Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  22. Results - GammaKnifeRS On the boundary between the water and air areas there are the greater differences The tumor is covered by the isodose curve 50%. The gamma index is visualized in blue levels if gamma < 1 and in red levels if gamma > 1 When gamma index > 1 the dose distributions are not in good agreement • We have compared the Monte Carlo dose distribution with the TPS one • The comparison is been done using the gamma function Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  23. Results – GammaKnifeRS • Percentage difference between the isodose 50% areas • Percentage of points that do not pass the g test • 18,2 % • Percentage difference of volumes covered by 50% isodose curves • 5,7 % Slices near the air volume This is the percentage of tumor mass that is incorrectly covered Some articles claim that when any portion of the lesion do not receive the therapeutic radio-surgical dose, tumor recurrence within 3 years is very probable* * Masahiro Shin et al., “Analysis of treatment outcome after stereotactic radiosurgery for cavernous sinus meningiomas”, J. Neurosurg. 95/2001 Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  24. Our Experience - HadronTherapy Easy to customize • We have used a Testbed (GILDA) • The run time using 20 CPU decrease by a factor 15 • A web interface has been created using “genius” interface Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  25. Conclusions • Sending of any Geant4 application on GRID is easy and it do not needs any code modification • The Grid Technology is a good solution to reduce the run time • The Grid could be a good solution to routine dose distribution validation needed in some radiotherapy treatments like Intensity-Modulated Radiation Therapy (IMRT) • showed in some articles, like “Routine IMRT verification by means of an automated Monte Carlo simulation system” Int. J. Radiation Oncology Biol. Phys. Vol. 56 Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

  26. Any Questions ? Thank you very much for your kind attention! Grid Open Day al LNS, 23.01.2008 Palermo, Grid Open Days all’Università di Palermo, 6-7.12.2007

More Related