Basic Concepts of FEM Framework & API

1. Basic Concepts of FEM Framework & API Gunavardhan Kakulapati (kakulapa@cs.uiuc.edu)

2. Introduction • FEM Framework overview • Using the framework API • Installation • Conversion Example

3. Motivation • Finite Element Method • Used extensively • Computationally intensive • Do it in parallel !!

4. FEM Programs • FEM programs manipulate elements and nodes • Element is a portion of problem domain, surrounded by nodes • Element computation: based on fields of surrounding nodes • Elements contribute to fields of surrounding nodes

5. FEM Mesh

6. FEM Program Structure read mesh, connectivity, boundary conditions time loop           element loop- Element deformation applies forces to surrounding nodes           node loop- Forces and boundary conditions change node positions end time loop write out mesh data for postprocessing

7. Parallelization • Partition the FEM Mesh into multiple chunks • Distribute elements, replicate shared nodes • Shared nodes = Communication !! • Partition so that communication is minimized

8. Partitioning

9. Parallel FEM Program read/get chunk mesh data, connectivity, shared nodes chunk time loop           element loop- Element deformation applies forces to surrounding nodes <update forces on shared nodes>          node loop- Forces and boundary conditions change node positions end time loop write chunk mesh data for postprocessing >>

10. FEM Framework: Goals • Separate parallel implementation from numerical algorithms • Parallel version to closely resemble the serial program • Allow features like load balancing, visualization.

11. FEM Framework: Responsibilities FEM Application (Initialize, Registration of Nodal Attributes, Loops Over Elements, Finalize) FEM Framework (Update of Nodal properties, Reductions over nodes or partitions) Partitioner Combiner METIS Charm++ (Dynamic Load Balancing, Communication) I/O

12. FEM Framework Program • May contain user-written, library-called subroutines: • init • driver • mesh_updated (more on this later…) • init andmesh_updated are called on processor 0 • driver is called on every chunk

13. Structure of an FEM Application init() Update driver Update driver Update driver Shared Nodes Shared Nodes

14. init subroutine init          read the serial mesh and configuration data inform the framework about the mesh end subroutine

15. driver subroutine driver          get local mesh chunk           time loop                FEM computations update shared node fields               more FEM computations           end time loop end subroutine

16. Data output • Parallel output at the end of driver • Possible to visualize the data rather than printing it out (netfem). • Framework calls like FEM_Mesh_updated. (more on this later …) >>

17. Framework Calls • FEM_Set_* • Called from initialization to set the serial mesh • Framework partitions mesh into chunks • FEM_Create_field • Registers a node data field with the framework, supports user data types • FEM_Update_field • Updates node data field across all processors • Handles all parallel communication • Other parallel calls (Reductions, etc.)

18. Framework Calls: Mesh • FEM_Set_* • From init, these routines describe the mesh to the FEM Framework • Framework calls the partitioner • FEM_Get_* • From each chunk, the local portion of the mesh is obtained by the driver

19. FEM_*_elem FEM_Set_elem(int elType,int nEl, int doublePerEl,int nodePerEl); subroutine FEM_Set_elem(elType,nEl,doublePerEl,nodePerEl) integer, intent(in) :: elType,nEl,doublePerEl,nodePerEl FEM_Get_elem(int elType,int* nEl, int* doublePerEl,int* nodePerEl); subroutine FEM_Get_elem(elType,nEl,doublePerEl,nodePerEl) integer, intent(in) :: elType integer, intent(out) :: nEl,doublePerEl,nodePerEl

20. FEM_*_node subroutine FEM_Set_node(nNode,doublePerNode) integer, intent(in) :: nNode,doublePerNode subroutine FEM_Get_node(nNode,doublePerNode) integer, intent(out) :: nNode,doublePerNode

21. Element Connectivity subroutine FEM_Set_Elem_Conn_r(elType,conn) integer, intent(in) :: elType integer, intent(in), dimension(nodePerEl,nEl) :: conn subroutine FEM_Get_Elem_Conn_r(elType,conn) integer, intent(in) :: elType integer, intent(out), dimension(nodePerEl,nEl) :: conn subroutine FEM_Set_Elem_Conn_c(elType,conn) integer, intent(in) :: elType integer, intent(in), dimension(nEl,nodePerEl) :: conn subroutine FEM_Get_Elem_Conn_c(elType,conn) integer, intent(in) :: elType integer, intent(out), dimension(nEl,nodePerEl) :: conn

22. Additional Data for Nodes and Elements subroutine FEM_Set_node_data_r(data) REAL*8, intent(in), dimension(doublePerNode,nNode) :: data subroutine FEM_Get_node_data_r(data) REAL*8, intent(out), dimension(doublePerNode,nNode) :: data subroutine FEM_Set_elem_data_r(data) REAL*8, intent(in), dimension(doublePerElem,nElem) :: data subroutine FEM_Get_elem_data_r(data) REAL*8, intent(out), dimension(doublePerElem,nElem) :: data

23. Node Fields • Framework handles combining data for shared nodes and keeps them in sync • Framework does not understand meaning of node fields, only their location and types • Framework needs to be informed of locations and types of fields • Create_field once, Update_field every timestep

24. FEM_Create_field • To handle the updating of shared node values. • Tell the framework where the shared data items of each node are. • Creates a “field” and pass the field ID for updating shared nodal values.

25. FEM_Create_simple_field function integer :: FEM_Create_simple_field( base_type, vec_len) integer, intent(in) :: base_type, vec_len • Base_type • FEM_BYTE- INTEGER*1, or CHARACTER*1 • FEM_INT- INTEGER*4 • FEM_REAL- REAL*4 • FEM_DOUBLE- DOUBLE PRECISION, or REAL*8

26. Create_simple_field Example ! 3D Force for each node ! stored as 3*n real*8 array REAL*8 ALLOCATABLE, DIMENSION(:) :: nodeForce INTEGER :: fid ... allocate nodeForce as 3*n_nodes... fid = FEM_Create_simple_field(FEM_DOUBLE,3)

27. FEM_Create_field function integer :: FEM_Create_Field(base_type, vec_len, offset, dist) integer, intent(in) :: base_type, vec_len, offset, dist

28. Node Fields

29. Create_field Example ! 3D force is contained as fXYZ variable ! in a user-defined type node_type TYPE(node_type), ALLOCATABLE, DIMENSION(:) :: nodes INTEGER :: fid ...allocate nodes array as n_nodes... fid = FEM_Create_Field(FEM_DOUBLE,3, offsetof(nodes(1), nodes(1)%fXYZ), offsetof(nodes(1), nodes(2)) )

30. Update and Reduce Field • subroutine FEM_Update_Field(fid,nodes) integer, intent(in) :: fid varies, intent(inout) :: nodes • subroutine FEM_Reduce_Field(fid,nodes,outVal,op) integer, intent(in) :: fid,op varies, intent(in) :: nodes varies, intent(out) :: outVal • op is • FEM_SUM • FEM_MIN • FEM_MAX

31. Utility function integer :: FEM_Num_Partitions() function integer :: FEM_My_Partition() function double precision :: FEM_Timer() subroutine FEM_Print_Partition() subroutine FEM_Print(str) character*, intent(in) :: str

32. Advanced FEM calls • FEM_Update_mesh • Reassembles chunks of the mesh • FEM_Add_node • Adds a new node into the mesh

33. FEM_Update_mesh • Reassembles all the chunks • Can be called only from driver • Must be called from all chunks • Useful scenarios • Giving out data as the simulation runs • Repartition the mesh • Serial output when simulation finishes

34. FEM_Update_mesh C call: void FEM_Update_mesh(int callMeshUpdated, int doWhat) Fortran call: subroutine FEM_Update_mesh(callMeshUpdated,doWhat) integer, intent(in) :: callMeshUpdated, doWhat

35. FEM_Update_mesh parameters • callMeshUpdated is non-zero => call mesh_updated (callMeshUpdated) • doWhat:

36. FEM_Update_mesh examples • FEM_Update_mesh(k,0) • Call mesh_updated(k) on assembled mesh, while the driver continues • FEM_Update_mesh(k,1) • Repartition after mesh_updated • FEM_Update_mesh(k,2) • Block driver routines till mesh_updated(k)

37. Adding Nodes • One can add new nodes, and update connectivity in driver • Use FEM_Set_* subroutines • New nodes are considered private • Framework can repartition the mesh • Optionally calls user’s mesh_updated subroutine

38. FEM_Add_node C call: void FEM_Add_node(int localIdx, int nBetween, int * betweenNodes) Fortran call: subroutine FEM_Add_node(localIdx,nBetween,betweenNodes) integer, intent(in) :: localIdx,nBetween integer, intent(in) :: betweenNodes(nBetween) >>

39. Installing FEM Framework

40. Where to Get It ? FEM Framework is included in Charm++ distribution, available under CVS CSH: setenv CVSROOT ":pserver:checkout@charm.cs.uiuc.edu:/cvsroot" Or BASH: export CVSROOT=":pserver:checkout@charm.cs.uiuc.edu:/cvsroot" You should now be able to do a > cvs login(no password needed, just type [Enter] at prompt) and then > cvs co -P charm to get the entire Charm++ source.

41. How to Build It ? > cd charm and do > ./build FEM net-linux -O This will make a net-linux directory, with bin, include, lib etc subdirectories. Platforms: net-sol, mpi-origin, mpi-linux etc.

42. How to Write Programs ? • Write from scratch • Concepts and API discussed earlier • Read the FEM-Framework Manual http://charm.cs.uiuc.edu/manuals/fem • Convert existing program • To be covered in next section

43. How to Compile & Link ? • Use “charmc”: available under bin • a multi-lingual compiler driver, understands f90 • Knows where modules and libraries are • Portable across machines and compilers • Linking • use “-language femf” : for F90 • Use “–language fem” : for C/C++ • See example Makefiles • pgms/charm++/fem/…

44. How to Run ? • Just run it! (net- versions only) • Serial, but nice for debugging/testing • Use Charmrun • A portable parallel job execution script • Specify number of processors: +pN • Special “nodelist” file for net-* versions • Multiple chunks per processor: use +vpM

45. Charmrun Example Nodelist File: $(HOME)/.nodelist group main host tur0001.cs.uiuc.edu host tur0002.cs.uiuc.edu host tur0003.cs.uiuc.edu etc… >> ./charmrun pgm +p4 +vp8

46. FEM Framework Conversion example

47. A Serial Program Processing Input Output

48. A Parallel Framework Program Parallel Processing Input Output Parallel Infrastructure

49. Real Names of Pieces Driver Init Finalize or Mesh_updated Charm++ FEM Framework

50. Serial Example Program • F90 example • Reads input mesh in “Triangle” format • Does simple explicit mechanics computation (CST triangles) • Writes Tecplot output • Not a toy example (by Philippe Geubelle) • Reads a parameter file • Applies boundary conditions • Uses real mechanics