Parallel Algorithm Oriented Mesh Database

1. Parallel Algorithm Oriented Mesh Database • Jean-François Remacle, Mark S. Shephard & Joseph E. Flaherty • Scientific Computation Research Center • Rensselaer Polytechnic Institute • remacle@scorec.rpi.edu • Outline • Algorithm Oriented Mesh Data Structure (AOMD) • Parallel AOMD • Technical issues • Parallel adaptive example (DG) • http://www.scorec.rpi.edu/AOMD

2. Motivations of AOMD and PAOMD • Aim of AOMD is to provide services to mesh users • Geometry based analysis, relation mesh to model is maintained • Support of dynamic mesh adjacencies • Parallel services: message passing, adaptivity and load balancing capabilities (callback pattern) • No MPI calls visible to users, PAOMD hides parallel issues • AOMD and PAOMD is a toolbox • Standard C++ Iterators, generic design • 3000 lines of code for the serial part • 1000 lines of code for the parallel part • Compiles in 5 minutes with gcc 3.0 • Open source (BSD) : http://www.scorec.rpi/AOMD

3. Basics of the Algorithm Oriented Mesh Database • A mesh entity is described by a set of lower dimension entities • All vertices always required • Vertices are atomic mesh entities, must be differentiated (using iD’s, coordinates or anything else consistent) • Two entities are equal if their set of vertices are equal • Not absolutely general but key to practical implementation • Allows to compare mesh entities (<,>,=) independently of their representation • Some associative containers for mesh entities : add, remove search • Minimum information • Equally dimension classified entities must be present • All vertices, all regions, all edges classified on model edgesand all faces classified on model faces • This is a sufficient minimum: no geometrical checks

4. Basics of the Parallel AOMD • Basics of parallel AOMD • Partition boundaries treated like model boundaries • Equal order mesh entities must exist on partition boundaries (partition faces, edges and vertices) • Mesh vertices must be differentiated among partitions • Same iD’s • Same coordinates • ... • On processor: serial AOMD • Implementation aspects • Simplicity, no master, no owner • Round of communication standardized, no MPI calls visible, messages automatically packed

5. Parallel AOMD - Mesh Adaptation • Target is transient applications with thousands of mesh adaptation steps • Want fast and simple adaptation • Need efficient inter-processor communications • Mesh Refinement • Apply templates • Include support of non-conforming meshes and multigrid • Refined entities with remote copies must be split on all partitions • Mesh Coarsening • Collect all mesh entities involved onto one partition • Carry out operation using serial operators on processor

6. Dynamic Load Balancing and Mesh Migration • Need dynamic load balancing after mesh adaptation • Procedures build on balancing procedures in Zoltan (from Sandia) • Load balancing procedure indicates which mesh entities are to be migrated to which processor • PAOMD only migrates minimum set, unless user specifically asks to migrate other entities classification after load balancing and before migration configuration after migration

7. Steps in process Collect the mesh entities to be migrated to another partition Determine needed higher order mesh entities to be migrated (use AOMD to determine minimal set needed) Collect entities and any user attached data Perform communications to send entities and update links Message passing At PAOMD operator level it appears messages are sent one at a time This would lead to unacceptable communication costs Message packing used - AUTOPACK (from Argonne) Automatically controls message packing process Includes information and tools to optimize message size for network architecture used Mesh Migration

8. Implementation issues • Design • C++ and generic programming • STL, efficient hashing function • AOMD::iterators follow C++ standard, std::algorithms (>100) may be applied in combination with AOMD::iterators • AOMD::algorithms available: adjacency creation, building a graph, building a tree in a mesh, edge collapsing… • Some OO Patterns • Singleton, Visitor, Memento... • Tradeoff efficiency vs. flexibility • We believe there is no tradeoff • Templates, functors, inlining… C++ can be efficient • Classical example, quick sort • stl::sort is twice faster (with VC6) than C qsort • External libraries for Parallel • Autopack, automatic message packing • Zoltan, dynamic load balancing and partitioning

9. Mesh refinement • class myAOMD_RefCallback • {public : • int operator () (const meshEntity *); • void callback (std::list<meshEntity *> &before, • std::list<meshEntity *> &after); • }; • Conformal or not (hanging nodes or mixed meshes) • The Algorithm • AOMD:: RefUnref(theMesh, myAOMD_RefCallback);

10. Communications • class myAOMD_RoundOfComm • {public : • char * sendBuffer (const meshEntity *, • int dest_proc, • size_t &sizebuf) const; • void recvBuffer (const meshEntity *, • int src_proc, • char *buf) const; • }; • Messages are packed (autopack) • The Algorithm • AOMD::roundOfComm(theMesh, myAOMD_RoundOfComm);

11. Load balancing • class myAOMD_LBCallback • {public : • char * sendBuffer (const meshEntity *, • int dest_proc, • size_t &sizebuf) const; • void recvBuffer (const meshEntity *, • int src_proc, • char *buf) const; • }; • Messages are packed (autopack) • The Algorithm • AOMD::LB(theMesh, myAOMD_LBCallback);

12. Available on http://www.scorec.rpi.edu/AOMD Demonstration of Load Balancing

13. Demonstration of Load Balancing

14. Linear DG elements, 30,000 to 800,000 dof Atwood Number, A = 1/3 10 fourier modes in “random” distribution time for the bubble to reach top of the window (y = 0.5) : 5 sec This calculation:a = 0.06 Experiments:a = 0.058 - 0.065 Theory (Glimm, et al) a = 0.045 - 0.06 2-D Animation of Instability

15. Refined 3-D Meshes for Rayleigh Taylor Instability non-conforming hexahedron mesh light fluid heavy fluid 24 steps of refinement 104 steps of refinement 72 steps of refinement

16. Conclusions • PAOMD advantages • Quite small piece of software, documented • Focused, mesh management only • Asks for minimum user knowledge about parallel issues • Efficient implementation • Future work • Terascale computers • PAOMD concepts are theoretically scalable • Hardware heterogeneity, machine and network models have to be added in partitioners 64 Procs, 40 GDof