Charm++ FEM Framework Tutorial

1. Charm++ FEM FrameworkTutorial Orion Sky Lawlor olawlor@uiuc.edu 2003/10/20

2. Roadmap • Why use FEM? • FEM Concepts • FEM Basic Calls • FEM Advanced Calls • Extra Features

3. Why use FEM?

4. Why use the FEM Framework? • Makes parallelizing a serial code faster and easier • Handles mesh partitioning • Handles communication • Handles load balancing (via Charm) • Allows extra features • IFEM Matrix Library • NetFEM Visualizer • Collision Detection Library

5. Why not use the FEM Framework? • Does not help you write serial code • But it does help with parallel • Another thing to learn • But it has a manual and examples • Another thing to break • But it runs on native MPI as well as AMPI

6. Scalability of FEM Framework

7. “Overhead” of Multipartitioning

8. Load balancer in action Automatic Load Balancing in Crack Propagation 1. Elements Added 3. Chunks Migrated 2. Load Balancer Invoked

9. FEM Framework Users: Frac3D • Frac3D fracture mechanics • Basic structures code, but with “cohesive elements” for fracture • Authors: Scott Brietenfeld, Philippe Geubelle

10. FEM Framework Users: CSAR • Rocflu fluids solver, a part of GENx • Finite-volume fluid dynamics code • Uses FEM ghost elements • Author: Andreas Haselbacher Robert Fielder, Center for Simulation of Advanced Rockets

11. FEM Framework Users: DG • Dendritic Growth • Simulate metal solidification process • Solves mechanical, thermal, fluid, and interface equations • Implicit, uses BiCG • Adaptive 3D mesh • Authors: Jung-ho Jeong, John Danzig

12. NetFEM Client: pretty pictures • Wave dispersion off a crack (simplified frac3d)

13. FEM Concepts

14. FEM Basics • FEM programs manipulate elements and nodes • Element is a portion of problem domain, surrounded by nodes • Node is one point in the domain

15. Serial FEM Mesh

16. Partitioned Mesh

17. FEM Parallel Model: Shared Nodes • “Shared Node” model • Element computations based on values of surrounding nodes • Node values are sum of surrounding elements • Example: Mechanical Simulation • Element stresses are computed from locations of element’s nodes • Sum up forces at nodes

18. FEM Mesh: Node Communication Summing forces from other processors only takes one call: FEM_Update_field Adds values from shared nodes

19. FEM Parallel Model: Ghosts • “Ghost” model • Element computations based only on values of surrounding nodes and elements • Example: Fluid Dynamics • Element pressures and velocities come from neighboring element pressures and velocities, and node locations

20. FEM Mesh: Ghosts Serial Mesh 1 2 3 4 Partitioning Right Chunk Left Chunk 1 2 Ghost of 3 Ghost of 2 3 4 Communication

21. FEM Program Structure read mesh, connectivity, boundary conditions time loop           element loop- Element deformation applies forces to surrounding nodes           node loop- Forces and boundary conditions change node positions end time loop write out mesh data for postprocessing

22. Parallelization • Partition the FEM Mesh into multiple chunks • Distribute elements, replicate shared nodes and/or add ghosts • Keep track of communication • Partition so that communication is minimized

23. Parallel FEM Program read/get chunk mesh data, connectivity, shared nodes chunk time loop           element loop- Element deformation applies forces to surrounding nodes  <update forces on shared nodes>           node loop- Forces and boundary conditions change node positions end time loop write chunk mesh data for postprocessing

24. FEM Framework Program • Consists of at least two user-written subroutines • init • driver • initis called on processor 0 • driver is called on every chunk

25. init subroutine init          read the serial mesh and configuration data inform the framework about the mesh end subroutine

26. driver subroutine driver          get local mesh chunk           time loop                FEM computations communication               more FEM computations           end time loop end subroutine

27. Structure of an FEM Application init() Update driver Update driver Update driver

28. A Serial Program Processing Input Output

29. A Parallel Framework Program Parallel Processing Input Output Parallel Infrastructure

30. Real Names of Pieces Driver Init Finalize or Mesh_updated Charm++ FEM Framework

31. FEM Mesh Access Calls New and Improved!

32. Old FEM Mesh Access Routines FEM_Set_elem FEM_Set_elem_data FEM_Set_elem_conn FEM_Set_node FEM_Set_node_data FEM_Set_sparse FEM_Set_sparse_elem FEM_Get_elem FEM_Get_elem_data FEM_Get_elem_conn FEM_Get_elem_ghost FEM_Get_node FEM_Get_node_data FEM_Get_node_ghost FEM_Get_sparse_length FEM_Get_sparse FEM_Get_sym Also, F90 versions of each Missing: FEM_Get_sparse_ghost

33. New FEM Mesh Access Routine (!) void FEM_Mesh_data( int mesh, int entity, int attr, void *data, int first, int length, int datatype,int width ); Get/Set, and multi-mesh support NODE, ELEM, SPARSE (+GHOST) DATA, CONN, SYM, GLOBALNO,… User data: width x length array Apply to first…first+length-1 User data formatting

34. New Mesh Access: Example FEM_Mesh_data( mesh, FEM_NODE, FEM_DATA+23, coord, 0,nNodes, FEM_DOUBLE,3 ); e.g., FEM_Mesh_default_read() We’re changing node values User data (tag 23) An array of 3 x nNodes doubles Change all the nodes 3 doubles/node 3 X Y Z X Y Z X Y Z nNodes X Y Z X Y Z

35. New Mesh Access: Advantages • Novice users have only one routine to learn • User’s read and write routines can be similar or even identical (just like PUP) • Smart users can easily abstract over parameters • E.g., “higher-order function” that sets both ghosts and real elements for any entity type • Library developers can easily, naturally add: • New entities: FEM_SPARSE, FEM_GHOST • New data: FEM_COORD, FEM_NODE_PRIMARY

36. New Mesh Access: Disadvantages • 8 parameters is too many to keep straight • Invalid combinations detected at runtime, instead of compile time as with the old way • E.g., FEM_NODE doesn’t have FEM_CONN, so there’s no FEM_Set_node_conn • Solution: Keep the old routines • But implement them in terms of the new routine • Mix and match with new routine • Needed for backward compatability, too

37. FEM Communication Calls

38. Node Fields • Framework handles combining data for shared nodes and keeps them in sync • Framework does not understand meaning of node fields, only their location and types • Framework needs to be informed of locations and types of fields • Create_field once, Update_field every timestep

39. Create a Field integer function FEM_Create_simple_field( datatype, len) integer, intent(in) :: datatype, len

40. Update Field: Shared Nodes subroutine FEM_Update_Field(fid,nodes) integer, intent(in) :: fid varies, intent(inout) :: nodes

41. FEM Ghost Layers

42. Ghost Elements: Overview • Most FEM programs communicates via shared nodes, using FEM_Update_field • Some computations require read-only copies of remote elements—“ghosts” • Stencil-type finite volume computations • The push form of matrix-vector product • Many kinds of mesh modification • Ghosts are a recent addition to the FEM framework

43. Ghosts: 2D Example Serial Mesh 1 2 3 4 Right Chunk Left Chunk 1 2 Ghost of 3 Ghost of 2 3 4

44. Building Ghosts: • Add ghost elements layer-by-layer from init • A chunk will include ghosts of all the elements it is connected to by “tuples”—sets of nodes • For 2D, a tuple might be a 2-node edge • For 3D, a tuple might be a 4-node face • You specify a ghost layer with FEM_Add_ghost_layer(tupleSize,ghostNodes) • ghostNodes indicates whether to add ghost nodes as well as ghost elements.

45. Building Ghosts: • FEM_Add_ghost_elem(e,t,elem2tuple) • e is the element type • t is the number of tuples per element • elem2tuple maps an element to its tuples: • A tupleSize by t array of integers • Contains element-local node numbers • Repeat this call for each ghost element type

46. Ghosts: Node adjacency 1 2 0 /* Node-adjacency: triangles have 3 nodes */ FEM_Add_ghost_layer(1,0); /* 1 node per tuple */ const static int tri2node[]={0,1,2}; FEM_Add_ghost_elem(0,3,tri2node);

47. Ghosts: Edge adjacency 1 2 0 /* Edge-adjacency: triangles have 3 edges */ FEM_Add_ghost_layer(2,0); /* 2 nodes per tuple */ const static int tri2edge[]={0,1, 1,2, 2,0}; FEM_Add_ghost_elem(0,3,tri2edge);

48. Extracting and Using Ghosts • Ghosts are always given larger numbers than non-ghosts—that is, ghosts are at the end • FEM_Get_node_ghost() and FEM_Get_elem_ghost(e) • Return the index of the first ghost node or element • FEM_Update_ghost_field(fid,e,data) • Obtain other processor’s data (formatted like fid) for each ghost element of type e 0 g e

49. Update Field: Ghosts subroutine FEM_Update_ghost_field(fid,elType,elts) integer, intent(in) :: fid,elType varies, intent(inout) :: elts

50. Ghost Example: Mesh