Martin Berzins (Steve Parker)

1. Martin Berzins(Steve Parker) What are the hard apps problems? How do the solutions get shared? What non-apps work is needed? Uintah solver for multiphase-fluid-structure interaction problems- explosive filled container in fire Thanks to DOE for funding since 1997, NSF since 2008

2. Martin Berzins(Steve Parker) What are the hard apps problems? ADAPTIVE DYNAMIC GLOBAL How do the solutions get shared? ENCAPSULATION ABSTRACTION What non-apps work is needed? APPLICATION DRIVEN TOOLS Uintah solver for multiphase-fluid-structure interaction problems- explosive filled container in fire Thanks to DOE for funding since 1997, NSF since 2008

3. Hard Apps Problems are Multi-Physics/Multiscale with Adaptive Methods and/or Global Comms. UPREDICTABLE!! • Lack of predictability forces use of dynamic load balancing methods. • AMR data structures – migration. • Radiation problems –global communications needed. • Particles move across grid with our methods • Any attempt to compute to a particular solution accuracy will need adaptive methods

4. Fundamental Uintah data Structure is a patch – multiple variable types SFC Load balancing uses patches Particle Variables Cell –Vertex Variables User writes code for a patch and its communications only - Uintah uses this information to construct communications pattern via a task graph Cell Centered Variables Uintah Domain Decomposition

5. Combining MPM and Mesh refinement

6. Problem Specification XML Load Balancer Simulation (One of Arches, ICE, MPM, MPMICE, MPMArches, …) Callbacks Assignments Tasks Data Archiver Scheduler Tasks Configuration Callbacks MPI How Does Uintah Work? Simulation Controller

7. Burgers Equation code void Burger::timeAdvance(const ProcessorGroup*, const PatchSubset* patches, const MaterialSubset* matls, DataWarehouse* old_dw, DataWarehouse* new_dw) { //Loop for all patches on this processor for(int p=0;p<patches->size();p++){ //Get data from data warehouse including 1 layer of "ghost" nodes from surrounding patches old_dw->get(u, lb_->u, matl, patch, Ghost::AroundNodes, 1); // dt, dx Time and space increments Vector dx = patch->getLevel()->dCell(); old_dw->get(dt, sharedState_->get_delt_label()); // allocate memory for results new_dw->allocateAndPut(new_u, lb_->u, matl, patch); // define iterator range l and h …… lots missing here and Iterate through all the nodes for(NodeIterator iter(l, h);!iter.done(); iter++){ IntVector n = *iter; double dudx = (u[n+IntVector(1,0,0)] - u[n-IntVector(1,0,0)]) /(2.0 * dx.x()); double du = - u[n] * dt * (dudx); new_u[n]= u[n] + du; }

8. Task graph • Each algorithm defines a description of the computation • Required inputs and outputs (names and spatial relationships) • Callbacks to perform each task on a single subregion of space • Communication is performed at the edges in the graph • Uintah uses this information to create a graph of computation and communication

9. Challenging dynamically changing workload AMR Scalability 8x8x8 patches: timings over about 30 timesteps 8K to 20K patches Original remeshes at every step Dilated only every 4 steps or so

10. Small Problem only 2-3 patches per proc at 4096 procs Atlas LLNL 1152 nodes of 4 Opteron dual-core Infiniband Thunder LLNL 1024 nodes of 4 Intel Itainum2 Quadrics switch Redstorm SNL 13284 nodes of Opteron dual-core XT3 Ranger UT Austin 3,936 nodes of 4 Barcelona quad-core Infiniband

11. Summary • Uintah is adaptive multi-physics AMR code • Clear separation between application user and system components • Very general CS approach to load balancing and scalability • Expensive multidisciplinary effort.

13. Existing AMR scalability work Strong scalability: fixed problem size doubling processors should halve execution time Weak scalability: problem size grows with processors Doubling processors give constant execution time • Brian Van Stralen: weak scaling easy strong scaling hard • Strong scaling problems include message overload, malloc inconsistencies and load imbalance. • Not clear that these problems cannot be solved • AMR does scale to 12K processors Flash Code BG/L and beyond (?)