Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems

1. Support for Adaptive Computations Applied to Simulation of Fluids in Biological Systems Kathy Yelick U.C. Berkeley

2. Project Summary • Provide easy-to-use, high performance tool for simulation of fluid flow in biological systems. • Using the Immersed Boundary Method • Enable simulations on large-scale parallel machines. • Distributed memory machine including SMP clusters • Using Titanium, ADR, and KeLP with AMR • Specific demonstration problem: Simulation of the heart model on Blue Horizon.

3. Overview of FY03 Plan • Extend to new applications of the method • Support for non-fiber boundaries • Improved accuracy • Scalable solvers • Heart simulation • Improved visualization • Large application run • Data analysis • Use in model construction • Data compression for results

4. Support for New Applications • The heart model is based on • Fibers representing muscle fibers • Also used for blood clotting and other apps • Some problems require other structures • Plates, shells, sets of fibers, etc. • Used in cochlea model, insect flight,… • Initial prototype of plate in TIBM code • FY03 Plan • Made code more general (UCB) • Specific support for the cochlea model (UCB)

5. Cochlea Model • Model of the inner ear • Developed by Julian Bunn and Ed Givelberg • Contains new features, e.g. membranes • Implemented one of these last fall • Givelberg will be at UCB next year

6. Supporting Applications: Accuracy • The heart simulation uses • First order accurate method • Second order method known • Demonstrated in Fortran code • Necessary for fluids with high Reynold’s numbers • For example, air flow around insect wing • FY03 Plan • Implement the second order accurate method in the TIBM code. (NYU + UCB)

7. Support Applications: Scalability • Replacing 3D FFT with multigrid • Scallop: 3D Poisson solver using the “method of local corrections.” • Scallop algorithm is more scalable than traditional multigrid (fewer messages) • Complete by end of FY02 • FY03 Plan • Complete integration in TIBM (UCB, UCSD) • Evaluate solvers (UCB, UCSD)

8. Heart Simulation • Recent improvements • Support for heart input files • Generate data for NYU visualization • Basic visualization is now OpenGL • Checkpoint/restart (underway) • FY03 plans • Large application run (NYU) • Extend vis support in “open” code (NYU) • Validation (OS and NYU)

9. Data Analysis: Cardiac Simulations • Methods and tools to analyze 3D datasets from cardiac blood flow. • Outputs are velocity and pressure values on a 3D grid (1283) over many time steps. • Characterize behavior • Natural and artificial heart valves • Vary viscosity, density, fiber stiffness, … • FY03 Plan • Grid support for DataCutter • Performance tuning • Use of FASTR to reduce data output size

10. FY03 Summary • Demonstration of TIBM on large problem • Large heart simulation run • Improve visualization software • Extend community through more applications • Immersed boundaries other than fibers • Better accuracy • Scallop for improved scalability • Data analysis • Close the loop: Build input data & analyze results • Compress output

11. Backup Slides These came from the review, and have more details on the current status

12. Alpha Project Plans application data • Several categories • Application development • Heart and cochlea-component simulation • Application-level package • Generic immersed boundary method • Parallel for shared and distributed memory • Enables new larger-scale simulations; finer grid • Solver libraries • Method of Local Corrections • Improved scalability and load balance expected • Data analysis • Building input data and analysis of results software systems

13. Immersed Boundary Method • Recent Performance Improvements • Use of FFTW in Spectral solver • 10x performance improvement on t3e • Use on BH still pending • Use of scatter/gather communication • Copying bounding boxes is still faster • Depends on application and machine • Load balancing • Alignment of fluid grid (in slabs) and fiber • Multigrid solver might offer more possibilities

14. Titanium on Blue Horizon • Recent improvements: • Support for PAPI (performance analysis) • Cache optimizations • Portable runtime layer (maintainable) • Faster LAPI-based implementation • LAPI is IBM’s “active message” layer • FY03 Plans • Communication optimizations • Common runtime with UPC (possibly CAF)

15. MPI vs. LAPI on the IBM SP • LAPI bandwidth higher than MPI • Also better small-message overhead • 9usec vs. 11usec • Latest Titanium release leverages this

16. Immersed Boundary Method Structure 4 steps in each timestep Fiber activation & force calculation Fiber Points Interpolate Velocity Spread Force Interaction Navier-Stokes Solver FluidLattice

17. Scallop: Multigrid Poisson Solver • A latency tolerant elliptical solver library • Will be used to build Navier-Stokes Solver • Implemented in KeLP, with a simple interface • Work by Scott Baden and Greg Balls • Based on Balls/Colella algorithm • 2D implementation in both KeLP and Titanium • 3D Solver • Algorithm is complete • Implementation running, but performance tuning is ongoing • Interface between Titanium and KeLP developed

18. Elliptic solvers • A finite-difference based solvers • Good for regular, block-structured domains • Method of Local Corrections • Local solutions corrected by a coarse solution • Good accuracy, well-conditioned solutions • Limited communication • Once to generate coarse grid values • Once to correct local solutions • Trades off extra computation for fewer messages

19. KeLP implementation • Advantages • abstractions available in C++ • built in domain calculus • communication management • numerical kernels written in Fortran • Simple interface • callable from other languages • no KeLP required in user code

20. Load Balancing Fluid grid is divided in slabs for 3D FFT Pizza cutter Egg slicer

21. Application: Heart Simulation • Performance improvements over the last year Heart simulation on a Cray T3E • 64 node t3e ~= 2 node C90 ~= 1-node (8p) BH (probably)

22. Improved Heart Structure Model • Current model is • Based on dog heart, textbook anatomy • Approximation by composing cones • Building a more accurate model • Use modern imaging on human heart for model • Need to see individual fibers • Collaboration between • Joel Saltz’s group and • Dr. Robert DePhilip in Anatomy Division of Biomedical Informatics Dept. at Ohio State University • Long term goal • Specialize model to patient using MRI data