1 / 25

Report from the NGGPS Overarching System Team

This report discusses the development and implementation of a portable, high-performance unified software infrastructure for operational prediction models at NCEP. It includes updates on the status of the unified software infrastructure, code, data, and documentation management, as well as the promotion of communication and coordination across teams.

earle
Download Presentation

Report from the NGGPS Overarching System Team

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Report from the NGGPS Overarching System Team Cecelia DeLuca/NESII, LigiaBernardet/DTC-GMTB, Mark Iredell/EMC, Jim Doyle/NRL, Mariana Vertenstein/NCAR, Jim Kinter/COLA, Larry Marx/COLA, Fei Liu/NRL-SAIC, Gerhard Theurich/NRL-SAIC, Patrick Tripp/EMC, Vijay Tallapragada/EMC, plus many involved with coupled system development NGGPS Annual Meeting August 5-6, 2016

  2. OAS Objectives • Provide a portable, high performance, unified software infrastructure for use in operational prediction models at NCEP. • Coordinate and provide to NCEP a guideline document for code, [data, and documentation] management for NEMS-based modeling applications and suites. • Promote communication and coordinated software development across the NGGPS teams and related efforts in NOAA and the broader community.

  3. Outline • What does a unified NEMS framework mean? • Status of: • Unified software infrastructure and modeling applications • Code, data, and documentation management • Promotion of communication and coordination across teams • More reason to pay attention:Physics Dynamics Coupling in Weather and Climate Models, Sept 20-22, 2016 PNNL – with computationally intensive physics and growing parallelism, researchers are increasingly exploring task parallel physics packages and parameterizations.

  4. NEMS Modeling Applications • NEMS can be assembled into a number of different modeling applications, each associated with: • a predictive target • a set of model components • a range of supported options, including physics, grids, resolutions, interpolation methods, etc.

  5. A Unified Software Framework forMultiple Applications • Above is a simplified version of the spreadsheet of NEMS applications:https://docs.google.com/spreadsheets/d/1RS-fTBYnfSIWrJYfalD2lAI-bUOGM0frNPEMIO_ND28/edit#gid=0 • This spreadsheet includes interim milestones for each application.

  6. What Does a Unified NEMS Framework Mean? • All NEMS applications are checked out the same way. For example, to checkout a revision of a coupled application from its development trunk: • svn co -r <REV> https://svnemc.ncep.noaa.gov/projects/nems/apps/<Application>/trunk <Application> • Currently, <Application> can be: • HYCOM-Ice • Regional • Regional-Nest • UGCS-Weather • UGCS-Subseasonal • UGCS-Seasonal • WAM-IPE • This single command checks out all the components associated with the application.

  7. What Does a Unified NEMS Framework Mean? All NEMS applications are built the same way: Change into the <Application> directory created by the SVN command during download, and invoke: ./NEMS/NEMSAppBuilder The NEMS AppBuildertool guides you through the build process for a coupled application. The end result is a NEMS executable (./NEMS/exe/NEMS.x) that is configured to run the application. The AppBuilder tool includes a knowledge base of how to build the components in each application.

  8. What Does a Unified NEMS Framework Mean? • All NEMS applications can be run the same way: • An application can support multiple configurations (e.g. different grids, different component implementations, multiple resolutions, one of the active components swapped for a prescribed data version, etc.). • “Compsets” are a concise label used to run standard cases for development and testing: • cfsr%20150401_1day_leapfrog_gsm%eul%T126_ww3%T188cfsr%20150401_30day_nems_gsm%eul%T126_cice%0.5_mom5%0.5 • Compsets are run using the command, • ./NEMS/NEMSCompsetRun • With the compset plus an application revision number, runsmade by different developers are reproducible.

  9. What Does a Unified NEMS Framework Mean? All NEMS applications have flexible run sequencesthat are specified at run-time when the NEMS executable accesses a file called nems.configure: runSeq::  @1800.0    MED MedPhase_slow    MED -> OCN    OCN    OCN -> MED    @600.0      MED MedPhase_fast_before      MED -> ATM      MED -> ICE      ATM      ICE      ATM -> MED      ICE -> MED      MED MedPhase_fast_after    @  @:: runSeq:: @7200.0 OCN -> MED MED MedPhase_slow MED -> OCN OCN @3600.0   MED MedPhase_fast_before   MED -> ATM   ATM   ATM -> MED   MED MedPhase_fast_after @ @ :: Atm-ocn sequence Global seasonal sequence This approach enables components and couplers (mediator or “med”) to be reconfigured easily to support multiple applications.

  10. What Does a Unified NEMS Framework Mean? All NEMS components interact with other components through NUOPC (National Unified Operational Prediction Capability) “caps” – these are wrappers that translate native model time, grids, and memory layouts into standard forms that the framework can understand. The same component source code is used in multiple applications.

  11. What Does a Unified NEMS Framework Mean? • NEMS components share Earth System Modeling Framework (ESMF) and other utilities • Includes fast, reliable, general interpolation weight generation • Weights are computed in parallel in 3D, using an underlying finite element mesh framework • Options for defining lines between points – straight, great circle lines, lat-lon • Supported grids: grids composed of logically rectangular patches (tripole, cubed sphere), unstructured meshes, global and regional grids, point clouds, in 2D and 3D • Supported interpolation methods: nearest neighbor, higher order patch recovery, bilinear and 1st order conservative methods, GFDL exchange grid

  12. Community Infrastructure Base The Earth System Modeling Framework (ESMF) was developed by multiple agencies and has a stable community governance structure. ESMF infrastructure is used by: NASA models, the Community Earth System Model (CESM), Navy global and regional models, the UK Met Office, analysis groups including the NCAR Command Language (NCL), the GrADS team, and DOE’s UV-CDAT, and many individual researchers (~7000 downloads). ESMF Metrics: ~7000 downloads ~3000 individuals on info mailing list ~40 platform/compilers regression tested nightly ~6400 regression tests ~800,000 lines of code NUOPC component interfaces are being used by NASA, Navy, CESM, and other centers – see Theurich et al., July 2016 BAMS https://www.earthsystemcog.org/projects/esmf/ https://www.earthsystemcog.org/projects/nuopc/

  13. NEMS Component Schematic Image courtesy of DTC.

  14. Unified Software Infrastructure • CHARGE: Deliver multiple coupled system applications following the NGGPS Implementation Plan1 and CPO MAPP awards, all running under a unified software framework, with EMC, broader NOAA, and community engagement. • ACHIEVEMENTS: • 1. Delivered initial global, regional, space weather, and land/hydrologycoupled system milestones running under the NEMS framework: • Delivered development version (0.2) of the Unified Global Coupled System – Seasonal (UGCS-Seasonal) running under NEMS, with fully coupled GSM, MOM5, and CICE components, initialized for a cold start and optimized for comparable performance with CFSv2. • 1 http://www.nws.noaa.gov/ost/nggps/NGGPS%20Implementation%20Plan%20v1.0.pdf

  15. Unified Software Infrastructure • ACHIEVEMENTS (cont): • Delivered development version (0.1) of a single domain regional configuration running under NEMS, with two-way coupled NMMB and HYCOM components. • Delivered development version (0.4) of WAM and the Ionosphere-Plasmasphere Electrodynamics (IPE) models validated under NEMS, with a one-way (WAM>IPE) 3D coupling exchange. • Delivered development version (0.2) of WRF-Hydro and LIS/Noah land validated running with GSM, MOM5, and CICE under NEMS, showing correct technical exchanges implemented among components, not yet scientifically viable. • Implemented WAVEWATCHIII coupled one-way to GSM (GSM>WWIII), including nesting, running under NEMS.

  16. Unified Software Infrastructure • ACHIEVEMENTS (cont): • 2. Delivered high performance modeling coupling tools capable of supporting a variety of coupled applications in a flexible and systematic manner, based on community standards ESMF/NUOPC, with options for 2D and 3D grid remapping. • Coordinated a merge of the NEMS development branch back to the NEMS trunk, which resulted in all development codes at EMC now running through a NUOPC/NEMS driver. • Established a unified way to systematically checkout, build, run, configure, and sequence many new coupled configurations using NEMS. • Provided ESMF/NUOPC/NEMS training for about 60 EMC staff, during multiple sessions.

  17. Unified Software Infrastructure ACHIEVEMENTS (cont): 3. Initiated a NGGPS-funded pilot project with the Community Earth System Model (CESM) in which the main NEMS coupler/mediator will become a community component. The proposal is to distribute a version of it through a shared repository created by NCAR called Common Infrastructure for Modeling Earth (CIME). NCAR has started building its next CESM implementation on coupler code shared with NEMS.

  18. Unified Software Infrastructure DOCUMENTATION: All coupled model deliveries are documented here: https://docs.google.com/spreadsheets/d/1RS-fTBYnfSIWrJYfalD2lAI-bUOGM0frNPEMIO_ND28/edit#gid=0 New NEMS User’s Guide:https://docs.google.com/document/d/1-kFhPBf7GBTUd5SaB5D_3OUGX_93pWKP21QgTh4y6ok/edit# TEAM: Weekly communication with distribution of about 80, including: ESRL/CU NESII team (Fei Liu, Gerhard Theurich, Anthony Craig, Cecelia DeLuca, others), EMC (Xingren Wu, Jiande Wang, Bin Li, Mark Iredell, SuruSaha, Patrick Tripp, Jun Wang, Jack Woollen, Jiarui Dong, others), NCAR (David Bailey), COLA (Larry Marx, Jim Kinter), GFDL (Niki Zadeh), AOML (Steve Diaz, Gopal), others

  19. Unified Software Infrastructure • GAPS: • EMC has not yet addressed several key system architecture questions.2 Developers would benefit from the input of researchers with deep knowledge of and proven skill in the scienceof building coupled systems with many model components. Questions including how best to seamlessly address a broad range of time and space scales. • EMC would benefit from the formalization of a science team that spans its applications, and is authorized to make decisions about topics like the system architecture (similar to the CESM Scientific Steering Committee). • EMC would benefit from research community experimentation and contribution to coupled model configurations. Currently GMTB’s expertise and focus is the atmosphere, so there is a need to augment that expertise through partnerships (like the CIME pilot with CESM). • 2https://docs.google.com/document/d/1ySqAUtJiw2PrsfW_8cC4pypwiNv1wQ4EFolv2jeBL3w/edit#heading=h.gor6vgpzooko

  20. Code, Data, and Documentation Management CHARGE: Deliver a document that outlines issues, requirements, and plans related to code, data and documentation management, with EMC, broader NOAA, and community engagement. ACHIEVEMENTS: 1. Delivered updated Code, Data, and Documentation Management document:https://docs.google.com/document/d/1bjnyJpJ7T3XeW3zCnhRLTL5a3m4_3XIAUeThUPWD9Tg/edit#heading=h.ku78qulk21xh Organized a meeting for September 1-2, 2016 at NCWCP, focused on EMC review and evolution of the document. Workshop prospectus:http://cog-esgf.esrl.noaa.gov/projects/nems-workshop/ 2. The draft code/data/documentation management document includes proposed operating principles, a proposed standard requirements format, and initial requirements for software, documentation, and input and output data. Technical requirements were also initiated for land surface and ensembles.

  21. Code, Data, and Documentation Management • ACHIEVEMENTS (cont) • 3. Initiated a Change Review Board (CRB) process for prioritizing NEMS-related work and allocating shared technical resources. First meeting was March 8, 2016. • TEAM: • Contributors/reviewers include DTC (Bernardet, Carson), ESRL (DeLuca, Theurich, Liu), EMC (Iredell, Tolman, Saha, Trahan), NCAR CESM (Large, Vertenstein), VLab (Sperow), COLA (Kinter, Marx), others. • GAPS: • Broader community review of the code/data/doc management document will be needed after the September 1-2 meeting at EMC. • EMC needs to establish a regular schedule for NEMS CRB meetings (at least twice a year).

  22. Communication and Coordination • CHARGE: Communication and coordination across NGGPS teams and related efforts in the community. • ACHIEVEMENTS: • Initiated documentation survey and gap analysis across NEMS components, model applications, and end-to-end suites. Contacted participants NGGPS-wide for input. Included general, scientific, technical, and user documentation categories per entry:https://docs.google.com/spreadsheets/d/1CLT66uzJrjrsY-um0jB5hU-Gfeh3_VCIJDA4-Ibmu5s/edit#gid=0Recently addedrequirements andtutorialcategories. • Began migratingall developerdocumentation toan EMC CommunityWorkspace on VLab. https://vlab.ncep.noaa.gov/web/environmental-modeling-center/home

  23. Communication and Coordination ACHIEVEMENTS (cont): 3. Agreed to a preferred technical documentation format (in a repository with code, set up so that it is automatically updated with code updates), and implemented physical parameterization, physics driver documentation, and other documentation in this format. TEAM:Organized by DTC (Bernardet), NESII (DeLuca, Liu), and EMC (Howard, Kunkel, Iredell); others have been contributing ideas and information. GAPS:Several components do not appear to have documentation, or are missing specific kinds of documentation. Much of the technical documentation is not in the preferred format, and it needs to be reviewed for clarity and correctness. EMC would benefit from redesign of their primary website (http://www.emc.ncep.noaa.gov/) and dedicated resources to maintain it. This process will be easier if developer documentation is moved off the primary website to a VLab community workspace.

  24. OAS Role and Leadership OVERALL GAPS: Topics the OAS has tried to address (including system-wide architecture, organizational coordination, and communication) would normally be aspects of organizational management. The leadership and membership of OAS is primarily technical. It would benefit from working more closely with EMC management and science leadership.

  25. Acknowledgements The work presented is supported by the NGGPS program, the CPO MAPP program, and the NSF EarthCube program.

More Related