1 / 27

So you want to use GCHP?

Learn about GCHP, its features, requirements, and workflow. Get started with GCHP and explore hardware and software needs. Presented by Sebastian Eastham and Lizzie Lundgren on May 6, 2019.

pelaez
Download Presentation

So you want to use GCHP?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. So you want to use GCHP? IGC9 GEOS-Chem High Performance Model Clinic Sebastian Eastham and Lizzie Lundgren May 6, 2019

  2. A note about this presentation Check the notes! Anything said during this presentation which isn’t on the slides should be in the presentation notes. A glossary of terms is included at the end! This presentation assumes use of GCHP 12.3.2. Always check GCHP documentation for the most up-to-date information. http://wiki.seas.harvard.edu/geos-chem/index.php/GCHP_Main_Page

  3. GCHP works now! This presentation will give you information on where we are, where we are going, what you need, and how to get started. But remember: you can use GCHP today: • GCHP undergoes the same version control as GCC (stable) • GCHP is benchmarked by the GCST (every X.Y.0 version)

  4. GCHP Today is GCC but with... • Grid resolution flexibility,without recompiling • Cubed-sphere transport • Improved scalability/speed • Takes advantage of: • FV3 offline advection • ESMF for core infrastructure • MAPL layer between GEOS-Chem and ESMF Eastham et al. (2018, GMD)

  5. GCHP Tomorrow Now New MAPL: • gfortran compatibility • Faster I/O • Lower memory footprint Mid-2019 2020 Stretched (“nested”) grids TBD Cloud GCHP CMake Improved error handling Containers Flux-based transport 2021? Ongoing integration of GMAO improvements throughout...

  6. What we will cover in this clinic • What GCHP needs • Hardware • Software • Workflow: GCHP versus GCC • Download and create run directory • Compile • Configure run • Run • Develop • Analyze • Resources

  7. What we will NOT cover in this clinic • Software library installation but... • Environment setup • Basic information covered in the GCHP wiki tutorial (http://wiki.seas.harvard.edu/geos-chem/index.php/Getting_Started_With_GCHP) For any of this information:Stop by the GCST Help Desk during IGC9M/Tu/W 1-3 pm Geological Museum 103A

  8. What GCHP needs: hardware concepts • Memory requirements grow with resolution* • All requirements can be distributed across all nodes.. • ..but bigger simulations will want better network fabric • Think carefully about data needs - the closer your I/O, the better Node A24 cores128 GB mem. Infiniband56+ Gb/s TCP-IP1 Gb/s10 Gb/s Node B24 cores128 GB mem.

  9. What GCHP needs: hardware concepts • GCHP can be run across a network, using multiple nodes • Two main considerations: • Do I have enough cores and memory for this resolution? In current version of GCHP, memory requirements grow with both resolution and the number of cores. You might find yourself “memory limited” rather than “core limited” - asking for more cores than you need so that you can get access to more memory. This is expected to be improved with the update to the new version of MAPL. • How fast is my network? Your network fabric may be limiting you. If you are thinking about upgrading your cluster or buying a new one, consider a high-speed network fabric This could be a higher-performance ethernet (TCP-IP here), or one of the Infiniband versions. You really do want at least 10 Gb/s. • Note that faster networks are generally also good for GC-Classic! If your compute nodes can’t talk to your data nodes because of slow network speeds, your simulation speed could be affected. However, you are more likely to fall foul of slow read/write at the disk level.

  10. What GCHP needs: hardware examples *Recommended values based on 12.3.2 where memory requirements grow with core count

  11. What GCHP needs: software

  12. Workflow ComparisonStep 1: Download and create run directory • Two nested source code repositories • Create GCHP run directory from source code, not Unit Tester • Use interactive script, not edit a text file • GCC will do this in a future update geos-chem gchp (includes MAPL, ESMF, FV3)

  13. Workflow ComparisonStep 2: Compile • Like GCC, build from run directory • Multiple build options • Initial full build required; subsequent build is change-dependent • Type make or make help to display options and use cases

  14. Workflow ComparisonStep 3: Configure run • More config files in GCHP than GCC • Set common run-time settings in driver file runConfig.sh • Simulation start/end time • Cubed-sphere grid resolution • Diagnostics collection frequency, duration, mode • # cores • Configure emissions in HEMCO_Config.rc, but also ExtData.rc • Unlike GCC diagnostics: • GCHP HISTORY.rc includes emissions • Diagnostic frequency is HHmmSS, not YYYYMMDD HHmmSS

  15. Workflow ComparisonStep 4: Run • Assess resource needs - # cores, # nodes, memory • Adapt run script from runScriptSamples/ • Non-system-specific in gchp.local.run • SLURM-specific in gchp.run • Advanced multi-run option for monthly diagnostics • Resource management tips: • # cores used (runConfig.sh) can be less than # cores requested (gchp.run) to maximize memory per core

  16. Workflow ComparisonStep 5: Develop • Develop source code same as GCC • Key exceptions: • main.F: driver files in GCHP (Chem_GridCompMod, gigc_chunk_mod) • HEMCO I/O and History/: I/O handling in MAPL (ExtData, History) • tpcore: advection in FVdycoreCubed_GridComp/ • Key concepts: • Same GEOS-Chem states as GCC (State_Chm/Met/Diag) • State array grids different per core (regional subset) • GCHP also has MAPL states (Imports, Exports, Internal State) • GEOS-Chem and MAPL states exchange data via pointers

  17. Workflow ComparisonStep 6: Analyze • Diagnostics data in OutputDir/ • Restart and emissions output are vertically flipped • Use gcpy benchmark code for data comparisons • Same functions for GCC and GCHP • Plot on cubed-sphere, or regrid to any grid/resolution • See gcpy/examples/compare_diagnostics.ipynb • Use NASA GISS Panoply netcdf viewer for raw data • Format not compatible until after upcoming MAPL version update

  18. Relax, you are not alone! • Create issues on GitHub to get help from GCST and beyond https://github.com/geoschem/gchp/issues • Join the GCHP Working Group and post your project http://wiki.seas.harvard.edu/geos-chem/index.php/GEOS-Chem_High_Performance_Working_Group • Join the Slack workspace to chat with users • Read and contribute to the GCHP wiki http://wiki.seas.harvard.edu/geos-chem/index.php/GEOS-Chem_HP • Contact the GEOS-Chem Support Team http://wiki.seas.harvard.edu/geos-chem/index.php/GEOS-Chem_Support_Team

  19. Questions/Comments/Discussion

  20. Top 10 list of time-saving tips • Only recompile code you changed (use make help) • Changing simulations does not require recompilation (copy geos) • runConfig.sh overwrites settings in other config files • Do not simply run the executable; use or adapt run scripts provided • Run-time MAPL/ESMF errors are nearly always from bad config • CAP error is usually bad simulation dates; check runConfig.sh • ExtData error is input; set MAPL_DEBUG_LEVEL to 20 in runConfig.sh • MAPL History error is problem with diagnostics; check HISTORY.rc • Search for “making install” in compile.log to find MAPL build errors • Create GCHP GitHub issue if you think you need to edit MAPL or ESMF

  21. Glossary of terms

  22. Detailed Workflow ComparisonStep 1: Download and create run directory

  23. Detailed Workflow ComparisonStep 2: Compile

  24. Detailed Workflow ComparisonStep 3: Configure run

  25. Detailed Workflow ComparisonStep 4: Run

  26. Detailed Workflow ComparisonStep 5: Develop

  27. Detailed Workflow ComparisonStep 6: Analyze

More Related