Applications

1. Applications BioSim Mahantesh Halappanavar, Ashutosh Mishra, Ravindra Joshi, Mike Sachon SURAgrid “All Hands” Meeting, Washington DC March 14 – 16, 2007

2. BioSim: Bio-electric Simulator for Whole Body Tissues • Numerical simulations for electrostimulation of tissues and whole-body biomodels • Predicts spatial and time dependent currents and voltages in part or whole-body biomodels • Numerous diagnostic and therapeutic applications, e.g., neurogenesis, cancer treatment, etc. • Fast parallelized computational approach

3. Simulation Models • Whole-body discretized within a cubic space simulation volume • From electrical standpoint, tissues are characterized as conductivities and permittivities • Cartesian grid of points along the three axes. Thus, at most a total of six nearest neighbors * Dimensions in millimeters

4. Numerical Models • Kirchhoff’s node analysis • Recast to compute matrix only once • For large models, matrix inversion is intractable • LU decomposition of the matrix

5. Numerical Models [M] • Voltage: User-specified time-dependent waveform • Impose boundary conditions locally • Actual data for conductivity and permittivity • Results in extremely sparse (asymmetric) matrix Red: Total elements in the matrix Blue: Nonzero Values

6. Why Focus on Solvers? • Scaling: (Source: David Keys, NIA Nov 2006) • “Science” phase scales as: • “Solver” phase scales as • Computation will be almost all solver after several doublings • Optimal solver saves computation cycles for physics

7. Direct A = LU Iterative y’ = Ay More General Non- symmetric Symmetric positive definite More Robust More Robust Less Storage The Landscape of Sparse Ax=b Solvers Source: John Gilbert, Sparse Matrix Days in MIT 18.337

8. LU Decomposition Source: Florin Dobrian

9. LU Decomposition Source: Florin Dobrian

10. Computational Complexity • 100 X 100 X 10 nodes: ~75 GB of memory (8-B floating precision) • Sparse data structure: ~ 6 MB (in our case) • Sparse direct solver: SuperLU-DIST • Xiaoye S. Li and James W. Dimmel, “SuperLU-DIST: A Scalable Distributed-Memory Sparse Direct Solver for Unsymmetric Linear Systems”, ACM Trans. Mathematical Software, June 2003, Volume 29, Number 2, Pages 110-140. • Fill reducing orderings with Metis • G. Karypis and V. Kumar, “A fast and high quality multilevel scheme for partitioning irregular graphs”, SIAM Journal on Scientific Computing, 1999, Volume 20, Number 1.

11. Performance on compute clusters 144,000-node Rat Model Blue: Average iteration time Cyan: Factorization time

12. Output: Visualization with MATLAB Potential Profile at a depth of 12mm

13. Output: Visualization with MATLAB • Simulated Potential Evolution • Along the Entire 51-mm Width of the Rat Model

14. Deployment on • Mileva: 4-node cluster dedicated for SURAgrid purposes • Authentication • ODU Root CA • Cross certification with SURA Bridge • Compatibility of accounts for ODU users • Authorization • Initial Goals: • Develop larger whole-body models with greater resolution • Scalability tests

15. Grid Workflow • Establish user accounts for ODU users • SURAgrid Central User Authentication and Authorization System • Off-line/Customized (e.g., USC, LSU) • Manually launch jobs based on remote resource • SSH/GSISSH/SURAgrid Portal • PBS/LSF/SGE • Transfer files • SCP/GSISCP/SURAgrid Portal

16. Recent Efforts in grid-enabling: • Porting to 100% open source tools (GCC/GFORTRAN) • SURAgrid Sites: • Texas A&M University: Calclab • University of Virginia: Grid04 and Grid11 • Experiments with MUMPS 4 • Symmetric matrices and out-of-core • Acknowledgements: • Jim Jokl, Steve Losen, Steve Johnson, Brain Brooks, Kate Barzee and Mary Fran Yafchak

17. News: (February 14, 2007)

19. Conclusions • Science: • Electrostimulation has variety of diagnostic and therapeutic applications • While numerical simulations provide many advantages over real experiments, they can be very arduous • Grid enabling: • New possibilities with grid computing • Grid-enabling an application is complex and time consuming • Security is nontrivial

20. Future Steps • Grid-enable BioSim • Explore alternatives for grid enabling BioSim • Explore funding opportunities • Load Balancing • Establish new collaborations • Scalability experiments with large compute clusters accessible via SURAgrid • Future applications: • Molecular and Cellular Dynamics • Computational Nano-Electronics • Tools: Gromacs, DL-POLY, NAMD

21. References and Contacts • A Mishra, R Joshi, K Schoenbach and C Clark, “A Fast Parallelized Computational Approach Based on Sparse LU Factorization for Predictions of Spatial and Time-Dependent Currents and Voltages in Full-Body Biomodels”, IEEE Trans. Plasma Science, August 2006, Volume 34, Number 4. • http://www.lions.odu.edu/~rjoshi/ • Ravindra Joshi, Ashutosh Mishra, Mike Sachon, Mahantesh Halappanavar • (rjoshi, amishra, msachon, mhalappa)@odu.edu

22. Teaching Initiative CS775/875: Distributed Computing Ravi Mukkamala Professor, Department of Computer Science

23. Details: • Graduate course with ~15 students • Guest lecture • Followed by a homework • Familiarize with grid computing concepts • Hands-on approach • Experiment with Globus services & commands • Acknowledgements: • Jim Jokl, Steve Losen, Steve Johnson, Brain Brooks, Nicole Geiger, Kate Barzee and Mary Fran Yafchak

25. Conclusions: • Laboratory for testing the concepts • Potential to attract students • For SURAgrid • Large number of short-lived certificates • Cleanup … (CRLs?/home drives/…) • Centralized account creation (Still painful ) • Short term funding/internships for grad/under-grad students?

26. THANK YOU !!!