1 / 75

Monday January 26, 2004

Welcome To the Inaugural Meeting of the WRF Software Training and Documentation Team Jan. 26-28, 2004 NCAR, MMM Division. Monday January 26, 2004. Introduction Software Overview WRF Software Tutorial, June 2003 Data and data structures Parallel infrastructure. Introduction. History

kylie-alvarado
Download Presentation

Monday January 26, 2004

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Welcome To the Inaugural Meeting of the WRF Software Training and Documentation TeamJan. 26-28, 2004NCAR, MMM Division

  2. Monday January 26, 2004 • Introduction • Software Overview • WRF Software Tutorial, June 2003 • Data and data structures • Parallel infrastructure

  3. Introduction • History • Requirements emphasize flexibility over a range of platforms, applications, users • WRF develops rapidly. First released Dec 2000; Last beta release, 1.3, in May 2003. Official 2.0 release coming in May 2004 • Circa 2003: "Arcane" used to describe WRF: • Adj. Known or understood by only a few. Mysterious.

  4. Introduction • Purpose of WRF Tiger Team effort • Extend knowledge of WRF software to wider base of software developers • Create comprehensive developer document • Team approach to both objectives • Streamlining and code improvement as byproductsin the coming months but not the subject for this meeting

  5. Introduction • This meeting • Review of WRF software structure and function: • Phenomenological – this is the code as it exists • Incomplete –time to prepare and present is limiter, but • What we're looking for now is a roadmap through the code for producing the comprehensive documentation • Develop an outline for the developer documentation • Writing assignments and work plan over next 9 months

  6. Some terms • WRF Architecture – scheme of software layers and interface definitions • WRF Framework – the software infrastructure, also "driver layer" in the WRF architecture • WRF Model Layer – the computational routines that are specifically WRF • WRF Model – a realization of the WRF architecture comprising the WRF model layer with some framework • WRF – a set of WRF architecture-compliant applications, of which the WRF Model is one

  7. WRF Software Overview

  8. Weather Research and Forecast Model Goals: Develop an advanced mesoscale forecast and assimilation system, and accelerate research advances into operations 12km WRF simulation of large-scale baroclinic cyclone, Oct. 24, 2001

  9. WRF Software Requirements… ;-) • Fully support user community's needs: nesting, coupling, contributed physics code and multiple dynamical cores – but keep it simple • Support every computer but make sure scaling and performance is optimal on the computer we use • Leverage community infrastructure, computational frameworks, contributed software, but please no opaque code • Implement by committee of geographically remote developers • Adhere to union of all software process models • Fully test, document, support • Free

  10. Goals Community Model Good performance Portable across a range of architectures Flexible, maintainable, understandable Facilitate code reuse Multiple dynamics/ physics options Run-time configurable Nested Package independent Aspects of Design Single-source code Fortran90 modules, dynamic memory, structures, recursion Hierarchical software architecture Multi-level parallelism CASE: Registry Package-neutral APIs I/O, data formats Communication Scalable nesting/coupling infrastructure WRF Software Requirements (for real)

  11. Aspects of WRF Software Design

  12. Aspects of WRF Software Design

  13. Model Coupling

  14. ESMF

  15. Performance

  16. Structural Aspects • Directory Structure and relationship to Software Hierarchy • File nomenclature and conventions • Use Association

  17. Directory Structure

  18. driver mediation model page 5, WRF D&I Document WRF Model Directory Structure

  19. WRF File Taxonomy and Nomenclature

  20. Modules are named module_something Name of file containing module is module_something.F If a module includes an initialization routine, that routine should be named init_module_something() Typically: Driver and model layers are made up of modules, Mediation layer is not (rather, bare subroutines), except for physics drivers in phys directory Gives benefit of modules while avoiding cycles in the use association graph Module Conventions and USE Association MODULE module_this MODULE module_that … driver USE module_this USE module_that USE module_whatcha USE module_macallit … mediation MODULE module_whatcha MODULE module_macallit USE module_this USE module_that … model

  21. WRF S/W Tutorial, June 2003

  22. Tutorial Presentation (click here) • Parallel Infrastructure • Registry details • I/O architecture and mechanism • Example of coding new package into framework

  23. Data and Data Structures

  24. Session 3: Data Structures • Overview • Representation of domain • Special representations • Lateral Boundary Conditions • 4D Tracer Arrays

  25. Data Overview • WRF Data Taxonomy • State data • Intermediate data type 1 (I1) • Intermediate data type 2 (I2) • Heap storage (COMMON or Module data)

  26. State Data • Persist for the duration of a domain • Represented as fields in domain data structure • Arrays are represented as dynamically allocated pointer arrays in the domain data structure • Declared in Registry using state keyword • Always memory dimensioned; always thread shared • Only state arrays can be subject to I/O and Interprocessor communication

  27. I1 Data • Data that persists for the duration of 1 time step on a domain and then released • Declared in Registry using i1 keyword • Typically automatic storage (program stack) in solve routine • Typical usage is for tendency arrays in solver • Always memory dimensioned and thread shared • Typically not communicated or I/O

  28. I2 Data • I2 data are local arrays that exist only in model-layer subroutines and exist only for the duration of the call to the subroutine • I2 data is not declared in Registry, never communicated and never input or output • I2 data is tile dimensioned and thread local; over-dimensioning within the routine for redundant computation is allowed • the responsibility of the model layer programmer • should always be limited to thread-local data

  29. Heap Storage • Data stored on the process heap is not thread- safe and is generally forbidden anywhere in WRF • COMMON declarations • Module data • Exception: If the data object is: • Completely contained and private within a Model Layer module, and • Set once and then read-only ever after, and • No decomposed dimensions.

  30. Grid Representation in Arrays • Increasing indices in WRF arrays run • West to East (X, or I-dimension) • South to North (Y, or J-dimension) • Bottom to Top (Z, or K-dimension) • Storage order in WRF is IKJ but this is a WRF Model convention, not a restriction of the WRF Software Framework

  31. Grid Representation in Arrays • The extent of the logical or domain dimensions is always the "staggered" grid dimension. That is, from the point of view of a non-staggered dimension, there is always an extra cell on the end of the domain dimension.

  32. v1,5 v2,5 v3,5 v4,5 u1,4 u2,4 u3,4 u4,4 u4,5 m1,4 m2,4 m3,4 m4,4 v1,4 v2,4 v3,4 v4,4 u1,3 u2,3 u3,3 u4,3 u5,3 m1,3 m2,3 m3,3 m4,3 v1,3 v2,3 v3,3 v4,3 u1,2 u2,2 u3,2 u4,2 u5,2 m1,2 m2,2 m3,2 m4,2 v1,2 v2,2 v3,2 v4,2 u1,1 u2,1 u3,1 u4,1 u5,1 m1,1 m2,1 m3,1 m4,1 v1,1 v2,1 v3,1 v4,1 Grid Indices Mapped onto Array Indices (C-grid example) jde = 5 Computation over mass points runs only ids..ide-1 and jds..jde-1 Likewise, vertical computation over unstaggered fields run kds..kde-1 jds = 1 ids = 1 ide = 5

  33. LBC Arrays • State arrays, declared in Registry using the b modifier in the dimension field of the entry • Store specified forcing data on domain 1, or forcing data from parent on a nest • All four boundaries are stored in the array; last index is over: P_XSB (western) P_XEB (eastern) P_YSB (southern) P_YEB (northern) These are defined in module_state_description.F

  34. LBC Arrays • LBC arrays are declared as follows: em_u_b(max(ide,jde),kde,spec_bdy_width,4) • Globally dimensioned in first index as the maximum of x and y dimensions • Second index is over vertical dimension • Third index is the width of the boundary (namelist) • Fourth index is which boundary • Note: LBC arrays are globally dimensioned • not fully dimensioned so still scalable in memory • preserves global address space for dealing with LBCs • makes input trivial (just read and broadcast)

  35. LBC Arrays P_YEB unused unused jde P_YEB A Given Domain P_XEB P_YSB jds spec_bdy_width ids ide

  36. LBC Arrays P_YEB unused unused jde P_XEB A given subdomain that includes a domain boundary P_YEB P_YSB jds spec_bdy_width ids ide

  37. Four Dimensional Tracer Arrays • State arrays, used to store arrays of 3D fields such as moisture tracers, chemical species, ensemble members, etc. • First 3 indices are over grid dimensions; last dimension is the tracer index • Each tracer is declared in the Registry as a separate state array but with f and optionally also t modifiers to the dimension field of the entry • The field is then added to the 4D array whose name is given by the use field of the Registry entry

  38. Four Dimensional Tracer Arrays • Fields of a 4D array are input and output separately and appear as any other 3D field in a WRF dataset • The extent of the last dimension of a tracer array is from PARAM_FIRST_SCALAR to num_tracername • Both defined in Registry-generated frame/module_state_description.F • PARAM_FIRST_SCALAR is a defined constant (2) • Num_tracername is computed at run-time in set_scalar_indices_from_config (module_configure) • Calculation is based on which of the tracer arrays are associated with which specific packages in the Registryand on which of those packages is active at run time (namelist.input)

  39. Four Dimensional Tracer Arrays • Each tracer index (e.g. P_QV) into the 4D array is also defined in module_state_description and set in set_scalar_indices_from_config • Code should always test that a tracer index greater than or equal to PARAM_FIRST_SCALAR before referencing the tracer (inactive tracers have an index of 1) • Loops over tracer indices should always run from PARAM_FIRST_SCALAR to num_tracername -- EXAMPLE

  40. Parallel Infrastructure

  41. Parallel Infrastructure • Distributed memory parallelism • Some basics • API • module_dm.F routines • Registry interface (gen_comms.c) • Data decomposition • Communications • Shared memory parallelism • Tiling • Threading directives • Thread safety

  42. Some Basics on DM Parallelism • Principal types of explicit communication • Halo exchanges • Periodic boundary updates • Parallel transposes • Special purpose scatter gather for nesting • Also • Broadcasts • Reductions (missing, using MPI directly) • Patch-to-global and global-to-patch • Built in I/O server mechanism

  43. Some Basics on DM Parallelism • All DM comm operations are collective • Semantics for specifying halos exchanges, periodic bdy updates, and transposes allow message agglomeration (bundling) • Halos and periods allow fields to have varying width stencils within the same operation • Efficient implementation is up to the external package implementing communications

  44. DM Comms API • External package provides a number of subroutines in module_dm.F Click here for partial API specification • Actual invocation of halos, periods, transposes provided by a specific external package is through #include files in the inc directory. This provides greater flexibility and latitude to the implementer than a subroutine interface • Package implementer may define comm-invocation include files manually or they can be generated automatically by the Registry by providing a routine external/package/gen_comms.c for inclusion in the Registry program

  45. RSL maintains descriptors for domains and the operations on the domains An operation such as a halo exchange is a collection of logical "messages", one per point on the halo's stencil Each message is a collection of fields that should be exchanged for that point RSL stores up this information in tables then compiles an efficient communication schedule the first time the operation is invoked for a domain msg2{ t, w, ps } . . . 24 pt stencil A few notes on RSL implementation msg1{ u, v }

  46. Example HALO_EM_D2_5 USED in dyn_em/solve_em.F #ifdef DM_PARALLEL IF ( h_mom_adv_order <= 4 ) THEN # include "HALO_EM_D2_3.inc" ELSE IF ( h_mom_adv_order <= 6 ) THEN # include "HALO_EM_D2_5.inc" ELSE WRITE(wrf_err_message,*)'solve_em: invalid h_mom_adv_order ' CALL wrf_error_fatal(TRIM(wrf_err_message)) ENDIF # include "PERIOD_BDY_EM_D.inc" # include "PERIOD_BDY_EM_MOIST2.inc" # include "PERIOD_BDY_EM_CHEM2.inc" #endif Defined in Registry halo HALO_EM_D2_5 dyn_em 48:u_2,v_2,w_2,t_2,ph_2;\ 24:moist_2,chem_2;\ 4:mu_2,al

  47. Example HALO_EM_D2_5 !STARTOFREGISTRYGENERATEDINCLUDE 'inc/HALO_EM_D2_5.inc' ! ! WARNING This file is generated automatically by use_registry ! using the data base in the file named Registry. ! Do not edit. Your changes to this file will be lost. ! IF ( grid%comms( HALO_EM_D2_5 ) == invalid_message_value ) THEN CALL wrf_debug ( 50 , 'set up halo HALO_EM_D2_5' ) CALL setup_halo_rsl( grid ) CALL reset_msgs_48pt CALL add_msg_48pt_real ( u_2 , (glen(2)) ) CALL add_msg_48pt_real ( v_2 , (glen(2)) ) CALL add_msg_48pt_real ( w_2 , (glen(2)) ) CALL add_msg_48pt_real ( t_2 , (glen(2)) ) CALL add_msg_48pt_real ( ph_2 , (glen(2)) ) if ( P_qv .GT. 1 ) CALL add_msg_24pt_real ( moist_2 ( grid%sm31,grid%sm32,grid%sm33,P_qv), glen(2) ) if ( P_qc .GT. 1 ) CALL add_msg_24pt_real ( moist_2 ( grid%sm31,grid%sm32,grid%sm33,P_qc), glen(2) ) if ( P_qr .GT. 1 ) CALL add_msg_24pt_real ( moist_2 ( grid%sm31,grid%sm32,grid%sm33,P_qr), glen(2) ) if ( P_qi .GT. 1 ) CALL add_msg_24pt_real ( moist_2 ( grid%sm31,grid%sm32,grid%sm33,P_qi), glen(2) ) if ( P_qs .GT. 1 ) CALL add_msg_24pt_real ( moist_2 ( grid%sm31,grid%sm32,grid%sm33,P_qs), glen(2) ) if ( P_qg .GT. 1 ) CALL add_msg_24pt_real ( moist_2 ( grid%sm31,grid%sm32,grid%sm33,P_qg), glen(2) ) CALL add_msg_4pt_real ( mu_2 , 1 ) CALL add_msg_4pt_real ( al , (glen(2)) ) CALL stencil_48pt ( grid%domdesc , grid%comms ( HALO_EM_D2_5 ) ) ENDIF CALL rsl_exch_stencil ( grid%domdesc , grid%comms( HALO_EM_D2_5 ) ) Defined in Registry halo HALO_EM_D2_5 dyn_em 48:u_2,v_2,w_2,t_2,ph_2;\ 24:moist_2,chem_2;\ 4:mu_2,al

  48. v1,5 v2,5 v3,5 v4,5 u1,4 u2,4 u3,4 u4,4 u4,5 v1,4 v2,4 v3,4 v4,4 u1,3 u2,3 u3,3 u4,3 u5,3 v1,3 v2,3 v3,3 v4,3 u1,2 u2,2 u3,2 u4,2 u5,2 v1,2 v2,2 v3,2 v4,2 u1,1 u2,1 u3,1 u4,1 u5,1 v1,1 v2,1 v3,1 v4,1 Notes on Period Communication m1,4 m2,4 m3,4 m4,4 updating Mass Point periodic boundary m1,3 m2,3 m3,3 m4,3 m1,2 m2,2 m3,2 m4,2 m1,1 m2,1 m3,1 m4,1

  49. v1,5 v2,5 v3,5 v4,5 u1,4 u2,4 u3,4 u4,4 u4,5 v1,4 v2,4 v3,4 v4,4 u1,3 u2,3 u3,3 u4,3 u5,3 v1,3 v2,3 v3,3 v4,3 u1,2 u2,2 u3,2 u4,2 u5,2 v1,2 v2,2 v3,2 v4,2 u1,1 u2,1 u3,1 u4,1 u5,1 v1,1 v2,1 v3,1 v4,1 Notes on Period Communication m4,4 m1,4 m1,4 m2,4 m3,4 m4,4 updating Mass Point periodic boundary m4,3 m1,3 m1,3 m2,3 m3,3 m4,3 m4,2 m1,2 m1,2 m2,2 m3,2 m4,2 m4,1 m1,1 m1,1 m2,1 m3,1 m4,1

  50. note: replicated v1,5 v2,5 v3,5 v4,5 v1,4 v2,4 v3,4 v4,4 v1,3 v2,3 v3,3 v4,3 v1,2 v2,2 v3,2 v4,2 v1,1 v2,1 v3,1 v4,1 Notes on Period Communication u1,4 u2,4 u3,4 u4,4 u1,5 m1,4 m2,4 m3,4 m4,4 updating U Staggered periodic boundary u1,3 u2,3 u3,3 u4,3 u1,3 m1,3 m2,3 m3,3 m4,3 u1,2 u2,2 u3,2 u4,2 u1,2 m1,2 m2,2 m3,2 m4,2 u1,1 u2,1 u3,1 u4,1 u1,1 m1,1 m2,1 m3,1 m4,1

More Related