Adaptive Mesh Applications

1. Adaptive Mesh Applications Sathish Vadhiyar Sources: - Schloegel, Karypis, Kumar. Multilevel Diffusion Schemes for Repartitioning of Adaptive Meshes. JPDC 1997 (Taken verbatim)

2. Adaptive Applications • Highly adaptive and irregular applications • Amount of work per task can vary drastically throughout the execution – similar to earlier applications, but.. • Has notions of “interesting” regions • Computations in the “interesting” regions of the domain larger than for other regions • It is difficult to predict which regions will become interesting

3. AMR Applications • An example of such applications is Parallel Adaptive Mesh Refinement (AMR) for multi-scale applications • Adaptive Mesh – Mesh or grid size is not fixed as in Laplace/Jacobi, but interesting regions are refined to form finer level grids/mesh • E.g.: to study crack growth through a macroscopic structure under stress

4. AMR Applications – Crack propagation • Such a system is subject to the laws of plasticity and elasticity and can be solved using finite element method • Crack growth forces the geometry of the domain to change • This in turn necessitates localized remeshing

5. AMR Applications- Adaptivity • Adaptivity arises when advances crosses from one subdomain to another • It is unknown in advance when or where the crack growth will take place and which subdomains will be affected • The computational complexity of a subdomain can increase dramatically due to greater levels of mesh refinement • Difficult to predict future workloads

6. Repartitioning • In adaptive meshes computation, areas of the mesh are selectively refined or derefined in order to accurately model the dynamic computation • Hence, repartitioning and redistributing the adapted mesh across processors is necessary

7. Repartitioning • The challenge is to keep the repartitioning cost to minimum limits • Similar problems to MD, GoL and Nbody • The primary difference in AMR is that loads can drastically change; cannot predict; will have to wait for refinement, then repartition

8. Structure of Parallel AMR

9. Repartitioning • 2 methods for creating a new partitioning from an already distributed mesh that has become load imbalanced due to mesh refinement and coarsening • Scratch-remap schemes create an entirely new partition • Diffusive schemes attempt to tweak the existing partition to achieve better load balance, often minimizing migration costs

10. Work Stealing • A method of load balancing • Each processor partnered with a neighbor • If its load goes below an application-level threshold, it sends messages to partner asking for work; i.e. tries to steal work • If partner does not have enough units, another partner is contacted • Can also be implemented using a master-worker approach; i.e., worker asking for more work from master

11. Graph Representation of Mesh • For irregular mesh applications, the computations associated with a mesh can be represented as a graph • Vertices represent the grid cells; vertex weights represent the amount of computations associated with the grid cells • Edges represent the communication between the grid cells; edge weights represent the amount of interactions

12. Graph Representation of Mesh • The objective is to partition across P processors • Each partition has equal amount of vertex weight • Total weight of the edges cut by the partition is minimized

13. Scratch-map Method • Partitioning from scratch will result in high vertex migration since as the partitioning does not take the initial location of the vertices into account • Hence a partitioning method should incrementally construct a new partition as simply a modification of the input partition

14. Notations • Let B(q) be the set of vertices with partition q • Weight of any partition q can be defined as: • Average partition weight: • A graph is imbalanced if it is partitioned, and:

15. Terms • A partition is over-balanced if its weight is greater than the average partition weight times (1+ ) • If less, under-balanced • The graph is balanced when no partition is over-balanced • Repartitioning – existing partition used as an input to form a new partition

16. Terms • A vertex is clean if its current partition is its initial partition; else dirty • Border vertex – adjacent vertex in another partition; those partitions are neighbor partitions • TotalV – sum of the sizes of the vertices which change partitions; i.e., sum of the sizes of the dirty vertices

17. 3 Objectives • Maintain balance between partitions • Minimize edge cuts • Minimize TotalV

18. Different Schemes • Repartitioning from scratch • Cut-and-paste repartitioning: excess vertices in an overbalanced partition are simply swapped into one or more underbalanced partitions in order to bring these partitions up to balance • The method can optimize TOTALV, but can have a negative effect on the edge-cut

19. Different Schemes • Another method is analogous to diffusion • Concept is for vertices to move from overbalanced to neighboring underbalanced partitions

20. Example(Assuming edge and vertex weights as equal to 1)

21. Example (contd..)

22. Analysis of the 3 schemes • Thus, cut-and-paste repartitioning minimizes TotalV, while completely ignoring edge-cut • Partitioning the graph from the scratch minimizes edge-cut, while resulting in high TotalV • Diffusion attempts to keep both TotalV and edge-cut low

23. Diffusion Scheme: An Algorithm

24. Multi-level Graph Repartitioning • Based on diffusive scheme • Multi-level graph repartitioning based on the k-way multilevel partitioning • Applying graph partitioning directly on a graph representation of a mesh can be time consuming since the graph can be very large

25. K-way multilevel partitioning algorithm • Has 3 phases: coarsening, partitioning, refinement (uncoarsening) • Coarsening - a sequence of smaller graphs constructed out of an input graph by collapsing vertices together

26. K-way multilevel partitioning algorithm • When enough vertices are collapsed together so that the coarsest graph is sufficiently small, a k-way partition is found • Finally, the partition of the coarsest graph is projected back to the original graph by refining it at each uncoarsening level using a k-way partitioning refinement algorithm

27. K-way partitioning refinement • A simple randomized algorithm that moves vertices among the partitions to minimize edge-cut and improve balance • For a vertex v, let neighborhood N(v) be the union of the partitions to which the vertices adjacent to v belong • In a k-way refinement algorithm, vertices are visited randomly

28. K-way partitioning refinement • A vertex v is moved to one of the neighboring partitions N(v) if any of the following vertex migration criteria is satisfied • The edge-cut is reduced while maintaining the balance • The balance improves while maintaining the edge-cut • This process is repeated until no further reduction in edge-cut is obtained

29. Multi-level diffusion repartitioning • In the coarsening phase, only pairs of nodes that belong to the same partition are considered for merging • Hence, initial partition of the coarsest level graph is identical to the input partition that is being repartitioned • This makes the coarsening phase completely parallelizable, as coarsening is local to each processor

30. Multi-level diffusion repartitioning • Uncoarsening phase contains 2 subphases • Multi-level diffusion • Multi-level refinement • Multi-level diffusion – balance is attempted on the coarsest graph

31. Multi-level diffusion repartitioning • A vertex is visited in a random order • If it is in overbalanced partition and is neighbors with nonoverbalanced partitions, moved to that nonoverbalanced partition that produces greatest improvement in edge-cut • Vertex is migrated even if this results in increased edge-cut • The process repeats until no balancing progress is made

32. Multi-level diffusion repartitioning • But it may not be possible to balance the graph at the coarsest level since there may not be many vertices of small weight • Hence graph needs to be uncoarsened one level to increase the number of smaller weight vertices • The process is repeated on the next coarsest graph and so on until some threshold

33. Multi-level diffusion repartitioning • After graph is balanced, multi-level diffusion ends, and multilevel refinement begins on the current graph • Here, the emphasis is on improving the edge-cut

34. Multi-level diffusion repartitioning • Each border vertex is visited randomly to potentially migrate based on the following refinement phase vertex migration criteria • The selected partition is the vertex’s initial partition from the input graph; the move does not increase the edge-cut; the balance is maintained • The edge-cut is reduced while maintaining the balance • The balance is improved while maintaining the edge-cut

35. Summary • Multi-level diffusion repartitioning algorithms are made up of three phases: graph coarsening, multilevel diffusion, multilevel refinement • Coarsening phase results in a series of contracted graphs • Multilevel diffusion phase balances the graph using the very coarsest graphs • Mutlilevel refinement phase seeks to improve the edge-cut disturbed by the balancing process

36. Illustration

37. Space Filling Curves for Partitioning and Load Balancing

38. Space Filling Curves • The underlying idea is to map a multidimensional space to one dimension where the partitioning is trivial • There are many different ways • But a mapping for partitioning algorithms should preserve the proximity information present in the multidimensional space to minimize communication costs

39. Space Filling Curve • Space filling curves are quick to run, can be implemented in parallel, and produce good load balancing with locality • A space-filling curve is formed over grid/mesh cells by using the centroid of the cells to represent them • The SFC produces a linear ordering of the cells such that cells that are close together in a linear ordering are also close together in the higher dimensional space

40. Space Filling Curve • The curve is then broken into segments based on the weights of the cells (weights computed using size and number of particles) • The segments are distributed to processors; thus cells that are close together in space are assigned to the same processor • This reduces overall amount of communication that occur, i.e., increases locality

41. SFC representation • One method is to define recursively – curve for a 2^k x 2^k grid composed of four 2^(k-1) x 2^(k-1) curves • Another method is using bit interleaving – the position of a grid point along the curve can be specified by interleaving bits of the coordinates of the point • The interleaving function is a characteristic of the curve

42. Z-curve or Morton ordering • The curve for a 2^k x 2^k grid composed of four 2^(k-1) x 2^(k-1) curves, one in each quadrant of the 2^k x 2^k grid • Order in which the curves are connected is the same as the order for 2x2 grid

43. Graycode Curve • Uses same interleaving function as Z-curve • But visits points in the graycode order • Graycode – two successive values differ in only one bit • The one-bit gray code is (0,1)

44. Graycode Curve • The gray code list for n bits can be generated recursively using n-1 bits • By reflecting the list (reversing the list) [1,0] • Concatenating original with the reflected [0,1,1,0] • Prefixing entries in the original list with 0, and prefixing entries in the reflected list with 1 [00,01,11,10]

45. Graycode Curve • 3-bit gray code: 000,001,011,010,110,111,101,100

46. Hilbert Curve • Hilbert curve is a smooth curve that avoids the sudden jumps in Z-curve and graycode curve • Curve composed of four curves of previous resolution in four quadrants • Curve in the lower left quadrant rotated clockwise by 90 degree, and curve in lower right quadrant rotated anticlockwise by 90 degree

47. SFCs for AMR • All these curve based partitioning techniques can also be applied for adaptive mesh by forming hierarchical SFCs