Adaptive Mesh Refinement

1. Adaptive Mesh Refinement • Used in various engineering applications where there are regions of greater interest • e.g. applications are http://www.damtp.cam.ac.uk/user/sdh20/amr/amr.html • Global Atmospheric modeling • Numerical Cosmology • Hyperbolic partial differential equations (M.J. Berger and J. Oliger) • Problems with uniformly refined meshes for above applications • Grid is too fine grained thus wasting resources • Grid is too coarse thus the results are not accurate Puneet Narula

2. AMR Library • Implements the distributed grid which can be dynamically adapted at runtime • Uses the arbitrary bit indexing of arrays • Requires synchronization only before refinement or coarsening • Interoperability because of Charm++ • Uses the dynamic load balancing capability of the chare arrays Puneet Narula

3. Node or root Leaf Virtual Leaf 0,0,0 0,0,2 0,1,2 1,0,2 1,1,2 0,0,4 0,1,4 1,0,4 1,1,4 0,2,4 0,3,4 1,2,4 1,3,4 Indexing of array elements • Case of 2D mesh Puneet Narula

4. Indexing of array elements (contd.) • Mathematicaly: (for 2D) if parent is x,y using n bits then, child1 – 2x , 2y using n+2 bits child2 – 2x ,2y+1 using n+2 bits child3 – 2x+1, 2y using n+2 bits child4 – 2x+1,2y+1 using n+2 bits Puneet Narula

5. Alternate representation 0,0,4 Puneet Narula

6. Communication with Nbors • In dimension x the two nbors can be obtained by - nbor --- x-1 where x is not equal to 0 + nbor --- x+1 where x is not equal to 2n • In dimension y the two nbors can be obtained by - nbor --- y-1 where y is not equal to 0 + nbor--- y+1 where y is not equal to 2n Puneet Narula

7. Case 1 Nbors of 1,1,2 Y dimension : -nbor 1,0,2 Case 2Nbors of 1,1,2X dimension : -nbor 0,1,2 Case 3 Nbors of 1,3,4 X dimension : +nbor 2,3,4 Nbor of 1,2,4 X Dimension : +nbor 2,2,4 0,0,0 0,0,2 0,1,2 1,0,2 1,1,2 0,0,4 0,1,4 1,0,4 1,1,4 0,2,4 0,3,4 1,2,4 1,3,4 Puneet Narula

8. Communication (contd.) • Assumption : The level of refinement of adjacent cells differs at maximum by one (requirement of the indexing scheme used) • Indexing scheme is similar for 1D and 3D cases Puneet Narula

9. AMR Interface • Library Tasks - Creation of Tree - Creation of Data at cells - Communication between cells - Calling the appropriate func’s for computation in each iteration - Refining – Autorefine + Refine on Criteria (specified by user) • User Tasks - Coding the main chare to start the AmrCoordinator - Writing the user data structure to be kept by each cell - Fragmenting + Combining of data for the Neighbors - Fragmenting of the data of the cell for refine Puneet Narula

10. Basic Class Diagram AmrCoordinator AmrCoordinator(DMsg*) AmrCoordinator(StartUpMsg*) createTree() Synchronise() Main chare Main(CkArgMsg*) Jacobi Jacobi() void**fragNborData(void*,int*) void** getNborMsgArray(int* ) void store(void*,int,int) void combineAndStore(void*, void*,int,int ) bool refineCriterion(void) void**fragmentForRefine(int*) void doComputation(void){} Cell AmrUserData* userdata Int iterations Bitvec Parent; children[][]; CkChareId coordHandle Refine(); Synchronise() NborComm(); doIteration() AmrUserData AmrUserData() AmrUserData*createData() AmrUserData * createData(void* data, int size) Cell2D Cell2D(_ArrInitMsg) createTree(_ArrInitMsg) Puneet Narula

11. Main Chare Main(CkArgMsg*) { CProxy_AmrCoordinator::ckNew() /*1st Method*/ /*2nd Method*/ StartUpMsg *startMsg = new StartUpMsg(dep,synchInt, dim,totIterations) CProxy_AmrCoordinator::ckNew(startMsg,0) */ Puneet Narula

12. AmrUserData • Factory Methods that User has to Implement AmrUserData* createData(void) { Jacobi * instance = new Jacobi(size); return (AmrUserData*) instance; } AmrUserData* createData(void* data, int dataSize) { Jacobi* instance = new Jacobi(data, dataSize); return (AmrUserData *) instance; } Puneet Narula

13. AmrUserData • Template Methods NeighborMsg **fragment(NeighborMsg* msg); void combineAndStore(NeighborMsg *msg1,NeighborMsg *msg2); void store(NeighborMsg *msg); • Virtual Methods void ** fragmentNborData(void* data, int *sizePtr) void ** getNborMsgArray(int* sizePtr) void store(void* data , int dataSize, int neighborSide) void combineAndStore(void *data1, void *data2, int dataSize,int neighborSide) bool refineCriterion(void) void **fragmentForRefine(int *sizePtr) void reg_nbor_msg(int neighbor_side, NeighborMsg *msg); Puneet Narula

14. Library--Cell • Methods void init_cell(_ArrInitMsg *msg); void treeSetup(_ArrInitMsg *msg); void check_queue(void); int sendInDimension(int dim,int side,NeighborMsg* msg); void neighbor_data(NeighborMsg *msg); void doComputation(void) //Refinement and synchronisation int sendInDimension(int dim,int side); void refine(_RefineMsg* msg); void change_to_leaf(ChildInitMsg* msg); void refine_confirmed(_DMsg *msg); void resume(_DMsg *msg); void synchronise(_RedMsg *msg); void refineExec(_DMsg *msg); void checkRefine(_RefineChkMsg* msg); void refineReady(bitvec retidx); //Virtual Methods void create_children(_ArrInitMsg** cmsg); void doIterations(void) void forwardSplitMsg(NeighborMsg *msg ,int neighbor_side) Puneet Narula

15. Pros and Cons of the User Interface • Pros • The user has to implement code for one class which is very similar to the sequential code • The control is with the library • No assumption about the data structure • All the communication is taken care of for the user • The user does not have to implement a parallel tree • Cons • The control is with the library • The user has to do the fragmentation and combination of Data – This can possibly be a pain if the user’s understanding of the AMR technique is not complete Puneet Narula

16. AMR Libraries • PARAMESH Peter MacNeice et al. http://sdcd.gsfc.nasa.gov/RIB/repositories/inhouse_gsfc/Users_manual/amr.htm • This library is implemented in Fortran 90 • Does not provide an Object Oriented Interface • User is required to implement code in a number of files • Cartesian grids is the only data structure that is allowed in each cell • Only supported on CrayT3E and SGI’s Puneet Narula

17. AMR Libraries (contd.) • Parallel Algorithms for Adaptive Mesh Refinement, Mark T. Jones and Paul E. Plassmann, SIAM J. on Scientific Computing, 18,(1997) pp. 686-708. (Also MSC Preprint p 421-0394. ) http://www-unix.mcs.anl.gov/sumaa3d/Papers/papers.html • Uses triangles instead of rectangular cells • Checks for refinement after every timestep • Two adjacent triangles cannot refine simultaneously • It is required to communicate who the neighbors are after every refinement Puneet Narula

18. AMR Libraries (contd.) • DAGH-Dynamic Adaptive Grid Hierarchies By Manish Parashar & James C. Browne * In C++ using MPI * Implements the distributed grid (SDDG) * Maps the multidimensional grid via space filling curves to a linear structure * The linear structure is repartitioned at run time to dynamically load balance * Requires implementation of the driver by the user for - Initialization of MPI environment - I/p parameters for the problem - Setting up of the grid structure and grid functions Puneet Narula

19. contd. - User interaction or control - Update grid functions - Communicate result of updating with other grids - Shut down environment * Requires synchronization after every fixed number of steps * Only cartesian grids are allowed Puneet Narula

20. Future Work • The developer should have the option of taking the control from the library • Visualization tools • Determine which strategy works best for the load balancing and if needed implement others • Integration with the FEM framework • Checkpointing Puneet Narula