230 likes | 548 Views
WRF ESMF Development. Tom Henderson, John Michalakes National Center for Atmospheric Research Mesoscale and Microscale Meteorology hender@ucar.edu 4 th ESMF Community Meeting -- 21 July 2005. Acknowledgements. Significant support Air Force Weather Agency, esp. Jerry Wegiel DoD HPCMO
E N D
WRF ESMF Development Tom Henderson, John Michalakes National Center for Atmospheric Research Mesoscale and Microscale Meteorology hender@ucar.edu 4th ESMF Community Meeting -- 21 July 2005
Acknowledgements • Significant support • Air Force Weather Agency, esp. Jerry Wegiel • DoD HPCMO • USWRP (NOAA and NSF) • Many collaborators/contributors • NCAR, NOAA, DoD, universities, vendors
Implementation of WRF Architecture Hierarchical organization Multiple dynamical cores Plug compatible physics Abstract interfaces (APIs) to external packages Performance-portable Metaprogramming “Registry” for managing model state Top-level Control, Memory Management, Nesting, Parallelism, External APIs driver ARW core NMM core Physics Interfaces mediation Plug-compatible physics Plug-compatible physics Plug-compatible physics Plug-compatible physics model Plug-compatible physics WRF Software Framework Overview
WRF Registry • Specify model state arrays, I/O, coupling, interprocess communication (when needed), nesting interpolation, etc. in ASCII text file called “Registry” • WRF build reads Registry file and automatically generates source code for: • Data members for WRF “domain” objects • I/O, coupling, and communication calls • Actual and dummy arguments in mediation-layer calls
New Feature: Moving Nests • John Michalakes augmented 2-way interactive nesting in WRF to allow reorientation of nested domain with respect to parent • Automatic movement algorithms such as U. Miami vortex following scheme • Automatic ingest of nested-resolution terrain and other lower boundary data to initialize leading edge of moving nest • HyCOM coupling (UNDER DEVELOPMENT) • Supports 2 or more nest levels in telescoping configuration • Parallel and efficient: small additional overhead (~2%) on top of 5-8% overhead for non-moving 2-way nesting. • With S. Chen, J. Cangialosi, W. Zhao (RSMAS, U. Miami) and S. Gopal at NCEP. Software infrastructure development supported by NOAA/NCEP (for use with NMM core). • Fully implemented in ARW Core for use in RAINEX and real-time hurricane forecasting this coming season… • Following animations done by John Michalakes
Five-day Hurricane Ivan 12km/4km Moving Nest Two-way interacting nest with high-resolution terrain ingest at leading edge 400 x 301 x 35, dt = 72 sec 331 x 352 x 35, dt = 24 sec Best track Run time: 8.6 hours on 64p IBM Power 4 (AFWA), including 20 minutes I/O
Five-day Hurricane Ivan 12km/4km Moving Nest Two-way interacting nest with high-resolution terrain ingest at leading edge 400 x 301 x 35, dt = 72 sec 331 x 352 x 35, dt = 24 sec Run time: 8.6 hours on 64p IBM Power 4 (AFWA), including 20 minutes I/O
Five-day Hurricane Ivan 12km/4km Moving Nest Two-way interacting nest with high-resolution terrain ingest at leading edge 400 x 301 x 35, dt = 72 sec 331 x 352 x 35, dt = 24 sec Run time: 8.6 hours on 64p IBM Power 4 (AFWA), including 20 minutes I/O
Goals • Generic interoperability with other ESMF components... • WRF + “ocean model” ( + “wave model”) • Hurricane applications – HWRF • WRF + 3DVAR / 4DVAR • Decouple requirements • WRF + “land model” for regional climate modeling • CLM == CCSM's land model • First cut: call CLM as WRF subroutine (CAM-like) • Jimy Dudhia (MMM), Mariana Vertenstein (CGD) • Prototype complete • Second cut: couple two ESMF gridded components • Need CLM ESMF component – early 2006 • Compare performance “apples-to-apples”
Work Completed • Completed draft of WRF-ESMF Integration Plan • Reviewed by WRF SE working group (WG-2) • Completed “stand-alone” component • Finished “init”, “run”, “final”, “set services” • Built trivial “AppDriver” • Tested vs. standard WRF • Included in WRF 2.1 release • Useful for testing, not useful otherwise…
Work In Progress • Create and populate ESMF “import” and “export” states • Attach references to ESMF_State objects to WRF “domain” objects at top level • Implement operations on ESMF_State objects as new external I/O package via WRF I/O & Coupling API • No changes to WRF software framework • ESMF-specific code only in new top-level driver and in external I/O package • Customize contents of ESMF_State objects via WRF Registry and WRF namelist • Prototype code complete, testing in progress
Work In Progress • Couple WRF with simple component via simple coupler • “Dummy ocean model” • Read SST from file and send to WRF • Receive SST from WRF and compare to file data for self-test • Coding under way • Test to validate prototype code
Future Plans • Couple to other ESMF components • HYCOM, CLM, 3D/4DVAR, LIS (NASA), … • Currently limited to “trivial” cases due to lack of support for general curvilinear coordinates in ESMF • Now a high priority for the core team • Will need “moving nests” for coupled hurricane applications • Also, need for “single executable” makes use of ESMF more difficult than existing WRF coupling methods for some models • Extension to “PSMF” • Planetary WRF • NASA, CalTech, etc.
Future Plans • Maximize coupler re-use by minimizing dependencies on gridded components • Climate and Forecast (CF) metadata conventions • www.cgd.ucar.edu/cms/eaton/cf-metadata/index.html • Use for all WRF I/O and coupling • Specify CF “standard_name” via WRF Registry • Use Balaji’s solution for Arakawa grids • Model metadata • Time, etc. • GO-ESSP: go-essp.gfdl.noaa.gov
A Few Issues • Orthodox Canonical Form • Time management • Attaching ESMF objects to pre-existing component objects • Destruction of ESMF objects
Orthodox Canonical Form • Key best practice for statically-typed object-oriented languages that do not support native garbage collection like C++ (and Fortran90) • Coplien: Advanced C++ Programming Styles and Idioms • Meyers: Effective C++ and More Effective C++ • Objects must have an assignment operator and copy constructor • ESMF “deep” objects do not have assignment operators • See ESMF Reference Manual Section 8.2 • Some do not have copy constructors, but will eventually
Orthodox Canonical Form • Lack of assignment operators causes confusion even in ESMF demo code! • CoupledFlowDemo.F90 ! Make our own local copy of the clock localclock = clock • Thanks to Nancy for fixing this after I whined… • All ESMF deep objects must have assignment operators with uniformly well-defined behavior!
Options for Deep Object Assignment • Option 1: Assignment behaves like standard Fortran assignment – it looks like a deep copy • Easy to explain and use • Difference between “deep” and “shallow” is performance • Future changes from shallow to deep will not break user code • Difficult to implement efficiently • Optimize using copy-on-write idiom via reference counting (Meyers, and many others too) • Is it possible to re-use an existing implementation?
Options for Deep Object Assignment • Option 2: Assignment behaves like reference-copy (assignment of Fortran logical unit numbers) • Easier to implement • Not very helpful to users – this can already be done via Fortran pointers! • Option 3: Assignment fails • Unless the assignment is a return value from a “Create” call • Insane for sure! • But still better than nothing
Initialization of “master clock” • Gridded components may already have expertise to initialize their own clocks • Dependencies • Time step: horizontal resolution, physical processes, etc. • Start and stop time: input and boundary data sets, time step, … • Avoid duplicating expertise in couplers/drivers • No “empty” ESMF_Clock (like “empty” ESMF_State) • Instead, pass time information back to driver/coupler through “init” interfaces as exportState metadata • Driver/coupler resolves inconsistencies between components and creates top-level clock (or aborts) • “Model metadata” conventions useful here
A Few More Issues • Attaching ESMF objects to pre-existing component objects • ESMF_VMGetCurrent() – extend to other types • Attach pointers, not “copies” • Proposal: pass pointers into “init”, “run”, “final” and let user decide via dummy argument declaration • Destruction of ESMF objects • Need inquiry ability: who must delete this? • Need ability to say “it’s OK to let ESMF delete this” (delegation) • Must pass in pointers to “Create” or “Set”