Evaluating Coupling Strategies

1. Evaluating Coupling Strategies Mike Hobson 20th April 2015

2. Evaluation of coupling strategies IS-ENES2 WP10 task 3 • This task followed on from work started at CW2013. • Sophie Valcke, Graham Riley, Rupert Ford & Mike Hobson • To find a method for evaluating different coupling strategies. • Provide benchmark problems as a way of comparing coupling technologies. • Provide “Reference Implementations” using common technologies.

3. Met Office interest/experience • The Met Office is currently investigating replacing its current Unified Model (UM) with a new more scalable model. • The project is called LFRic. • This model is likely to: • Move away from using lon/lat meshes. • Use a finite-element formulation.

4. Met Office interest/experience • We have a future requirement to have semi-structured or unstructured meshes. • But will need to work with regular lon/lat models. • Met Office took on responsibility to provide a benchmark for coupling models with the following meshes: • a regular longitude/latitude mesh • a semi-structured mesh

5. Structured rectilinear mesh Longitude-latitude • The mesh in use in the current Met Office Unified Model (UM). • Can use direct addressing – the cell at (i+1,j) is always to the right of that at (i,j). • Mesh will be useful infuture for outputtingdiagnostics. • Singularity at poles.

6. Semi-structured mesh Cube-sphere • An approximation to an unstructured mesh – appears as a 1d list of cells. • Requires indirect addressing. • Likely to be similar to thesorts of mesh that willbe used in futuremodels to avoid thesingularity at the poles.

7. Coupling benchmark • Should be technology agnostic. • Benchmarks are defined fully in the document “Benchmark definition for evaluation of coupling strategies”. • For my simple reference implementations: • Take a field from a model with one of the above meshes and transfer it to another model with the other mesh. • ...and the reverse.

8. Reference implementation no.1 Using OASIS3-MCT Model 1 oasis_init_comp oasis_def_partition oasis_def_var oasis_enddef Time-step loop oasis_put oasis_get oasis_terminate Model 2 oasis_init_comp oasis_def_partition oasis_def_var oasis_enddef Time-step loop oasis_put oasis_get oasis_terminate

9. Reference implementation no.1 Using OASIS3-MCT • System consists of two separate model components in two executables ...with calls to the OASIS3-MCT library from each • Very little code intrusion.

10. Reference implementation no.2 Using ESMF Driver ESMF_Initialize ESMF_Grid/CplCompCreate ESMF_Grid/CplCompSetServices ESMF_Grid/CplCompInitialize Time-step loop ESMF_Grid/CplCompRun ESMF_Grid/CplCompFinalize ESMF_Finalize ESMF Grid/Cpl Component Register Init(state,… Run(state,… Finalize(state,…

11. Reference implementation no.2 Using ESMF (Init) Model 1 init ESMF_GridCreate ESMF_FieldCreate ESMF_StateAdd Coupler init ESMF_StateReconcile ESMF_FieldRegridStore Model 2 init ESMF_MeshCreate ESMF_FieldCreate ESMF_StateAdd

12. Reference implementation no.2 Using ESMF (Run) Driver Model 1 run ESMF_StateGet (perform a time step of model 1) Coupler (model1 → model2) run ESMF_StateGet ESMF_FieldRegrid Model 2 run ESMF_StateGet (perform a time step of model 2) Coupler (model2 → model1) run ESMF_StateGet ESMF_FieldRegrid

13. Reference implementation no.2 Using ESMF (Finalize) Model 1 final ESMF_FieldDestroy ESMF_GridDestroy Coupler final ESMF_FieldRegridRelease Model 2 final ESMF_FieldDestroy ESMF_MeshDestroy

14. Reference implementation no.2 Using ESMF • Using ESMF as a coupler is much more about using the whole framework. • Would generally require much more of a rewrite compared to OASIS3-MCT. • But will deliver all the benefits of a better structured code.

15. What was the point of all that? • But we already knew all of that ...... so what was the point! • Well, that was the point. • Implementing the benchmark has given us a good handle on the non-functional characteristics of the coupling technologies.

16. Assessment of functional characteristics • It is easy to use the benchmark to answer the “Does it ...” type questions. • It is also reasonably straightforward to answer the correctness questions. • It is more difficult to answer the performance (MOTR – “Measure of Time Required”) questions.

17. Performance benchmarking • Care must be taken to make sure benchmarks are: • Representative • Fair • Correct • There are many technical challenges in achieving the above.

18. What to measure? • The first question is what do you measure? • Overall runtime to complete a coupling task – will not provide detailed enough information. • Isolating specific characteristics can be difficult – the technologies can approach the problem in very different ways.

19. Technical challenges • Compile each tool in a comparable way. • Make sure all debugging information is turned off in each case. • Ensure the processor configurations are the same.

20. MOTR for regridding • Choose a representative problem: • 0.25°lon/lat mesh (1440 x 720) • cube-sphere mesh with 1.5million cells. • (Both these meshes lead to about 20km resolution at UK latitudes) • Isolate regridding from redistribution. • Add calls to Fortran intrinsics to time the relevant calls to the coupling library in each case. • Optimise.

21. Result • My regridding problem was quite big, with only a small number of processors(so both technologies were quite slow) • ~ 0.25s per regridding operation. • Both results were within a few percent of each other. • My optimisations were probably not comprehensive – so I’m sure both times could be improved.

22. Further work Finite-Element Meshes • The Met Office also agreed go through the same process for models with finite-elements on similar meshes. • This hasn’t been done yet because: • The Met Office hasn’t yet built the architecture that could be used to generate a finite-element benchmark. • The coupling technologies don’t yet support finite-element meshes.

23. Conclusions • The benchmarks are a useful tool. • They allow us to answer straightforward questions (both functional and non-functional) easily. • Care is needed when using the benchmarks to measure performance.

24. Questions?