Regridding Between IPE and WAM Grids Progress Report Peggy Li March 29, 2013

1. Regridding Between IPE and WAM Grids Progress Report Peggy Li March 29, 2013

2. IPE Grid • An IPE grid is defined as a set of flux tubes. There are 80 x 170 flux tubes in each hemisphere and a total of 80x170x2=27200 flus tubes in the IPE grid. • Each flux tube has different number of grid points. They all start from 90KM height and can reach as high as 363000KM. • To construct an ESMF mesh from this grid, we first construct 2D grids using the grid points of the same height from all the flux tubes. At the lowest height, i.e. 90KM, there are 27200 grid points. When the altitude gets higher, there are fewer grid points. • We then construct the 3D meshes using the grid points at two consecutive heights. The mesh cells are mostly hexahedrons, but since the layer at higher altitude may have fewer grid points, we have to construct prism cells (a quad at the bottom layer and a line at the top layer).

3. The flux tubes along one longitude line, the tubes are cut off at 800KM

4. The 2D IPE Grid at the bottom 90KM height, there is a gap along the magnetic equator. The grid cells are represented as quads in ESMF

5. The magnetic north pole is not aligned with the pole

6. The magnetic equator is like a sinusodial curve. The grid points closer to the equator are much denser than the grids at the higher latitude.

7. The 2D IPE Grid at the 728 KM height, note that there are a lot fewer grid points close to the equator

8. The 3D grid cells represented as prisms

9. 3D WAM Grid • The WAM Grid at each horizontal layer is a reduced Gaussian grid with 94 points in latitude and maximal 192 grid points in longitude. The number of grids points at each latitude has a Gaussian distribution with the most points in the equator area. • The vertical layers are in the pressure field and it changes dynamically during the simulation. • In order to regrid it to the IPE grid, the vertical dimension has to be converted into a height field. • To represent it in ESMF, we first construct a 2D mesh for the horizontal field (which is the same for every vertical layer) using triangles and quads. Then we use the matching quads or triangles at adjacent vertical layers to form hexahedrons or prisms. • We used an sample average height fields for the vertical layer for now. Each vertical layer has a fixed height. The height fields are from 0KM to 591.44KM.

10. The 2D WAM Grid, note the triangles in the mesh since the number of grid points at higher latitude may be less than that at lower latitude

11. IPE to WAM Regridding • The 3D IPE grid and 3D WAM grid overlap in the height field between 90KM to 600KM (the WAM grid may go as high as 700KM). So, we only construct the 3D grid using the overlapped height fields. • The 3D WAM grid has 55 vertical layers with heights from 89.74KM to 591.44KM. The total number of grid points is 718960 and the total number of cells is 713124. The 3D IPE grid has 77 levels with the heights from 90KM to 620KM. The total number of grid points is 1684480 and the total number of cells is 1655680. • We generated the 3D IPE and 3D WAM grid in UGRID file format.

12. Regridding Results • We performed bilinear regridding from IPE to WAM and from WAM to IPE using the ESMF_RegridWeightGen application. • The weights are examined by applying an analytical function to the source grid and calculating the relative errors of the interpolated destination grid. We also checked if there is any “bad” weight values, i.e., the values out of the [0 1] range.

13. Weight Quality • There is no out-of-range weights. • Analytical function: 1.0 + 0.01*height(km)+cos(lat)2 * cos(2*lon) • All the errors are relative errors. The mean error is relatively high comparing with the 2D test cases included in the RegridWeightChecker, which are usually around 1.0e-5.

14. Why the interpolation errors are high? The plot to the left shows all the destination nodes that has a relative interpolation error > 0.05. Note that they are all located in the areas close to the equator (from 30S to 20N) and at low altitude (below 100 Km). The plot to the right shows the 2D IPE grid at 90KM height and close to the magnetic equator. The cell are very wide (about 500 km, or 5 degree in longitude) but very narrow. The heights of the 3D cells at low altitudes are only 2km. ESMF regridder maps the cells into the 3D Cartesian coordinate system, but the edges of the cell are straight lines, instead of great circle curves. Because of that, some of the destination nodes are mapped into the wrong source cells, thus resulting in the high interpolation errors.

15. Performance • We ran the ESMF_RegridWeightGen on yellowstone using 1 to 64 PEs and measure the timing of ESMF_FieldRegridStore(). We ran each regridding twice and here is the result: (all the times are in seconds)

16. Timing Results

17. Next Steps • Improve the bilinear interpolation weight quality • Map the 3D cells into great circle. • Figure out how to construct the ESMF Mesh using the IPE grid distribution (by flux tubes) used by the application. • Create a 3D WAM grid using variable heights at every grid point, then investigate the performance impact of the two step interpolation first from IPE to a fixed height WAM grid, then to a variable height WAM grid.