1 / 25

WRF Model: Software Architecture

WRF Model: Software Architecture. WRF Tutorial, 8/16/01 John Michalakes, NCAR. Overview. Purpose Familiarization with concept and structure behind WRF software from the point of view of a scientific user/developer Outline Code organization Framework goals and design aspects

kirkan
Download Presentation

WRF Model: Software Architecture

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WRF Model: Software Architecture WRF Tutorial, 8/16/01 John Michalakes, NCAR

  2. Overview • Purpose • Familiarization with concept and structure behind WRF software from the point of view of a scientific user/developer • Outline • Code organization • Framework goals and design aspects • Software hierarchy, model interface, and memory model • Parallelism • Data structures • The WRF Registry

  3. Directory Structure (WRF 1.1) clean* clean script compile* compile script configure* configure script Registry/ Registry registry file arch/ configure.defaults arch-specific compile options external/ IOAPI WRF I/O API specification RSL/ external comm package: RSL io_netcdf/ external I/O package: NetCDF inc/ holds intermediate include files src/ source directory (.F files) test/ directory containing test cases b_wave/ hill2d_x/ quarter_ss/ real/ squall2d_x/ squall2d_y/ tools/ use_registry Perl scripts implementing Registry

  4. Directory Structure (WRF 1.2) clean* clean script compile* compile script configure* configure script Registry/ Registry registry file arch/ configure.defaults arch-specific compile options dyn_eh/ Eulerian-height dyncore source files dyn_em/ Eulerian-mass dyncore source files dyn_slt/ semi-implicit semi-Lagrangian dyncore external/ same as 1.1 frame/ driver layer (framework) source files inc/ holds intermediate include files main/ wrf.F main WRF source file phys/ physics directory share/ files shared across dyncores test/ test cases tools/ registry* new implementation of registry mechanism

  5. Goals Good performance Portable across a range of architectures Flexible, maintainable, understandable Facilitate code reuse Multiple dynamics/physics options Run-time configurability Package independence Aspects of Design Single-source code Fortran90 modules, dynamic memory, structures, recursion Hierarchical design Multi-level parallelism CASE: Registry Package APIs WRF Model Software

  6. WRF main: • Top-level flow of control • - Initialize packages • - Input config info • - Allocate, initialize, • decompose main domain • Start integration • Shutdown • Integrate: • Time loop • Nesting (recursive) • Calls to I/O driver • Solve_eh: • Sequence through tile loops calling model layer routines for dynamics and physics • Multi-threading (OpenMP) • Interprocessor communi- cation • Solve_interface: • 1 step on 1 domain • Select dyncore • Dereference ADT • Solve_eh: • Sequence through tile loops calling model layer routines for dynamics and physics • Multi-threading (OpenMP) • Interprocessor communi- cation mediation • Top-level model layersubroutines: • One tile computation • Boundary conditions/avoidance • May select physics opt. • May dereference 4D fields • Top-level model layersubroutines: • One tile computation • Boundary conditions/avoidance • May select physics opt. • May dereference 4D fields • Top-level model layersubroutines: • One tile computation • Boundary conditions/avoidance • May select physics opt. • May dereference 4D fields model Code structure

  7. WRF main: • Top-level flow of control • - Initialize packages • - Input config info • - Allocate, initialize, • decompose main domain • Start integration • Shutdown WRF domain derived data type (frame/module_domain.F) MODULE module_domain TYPE (domain) REAL, DIMENSION(:,:,:), POINTER :: eh_ru_1 REAL, DIMENSION(:,:,:), POINTER :: eh_ru_2 REAL, DIMENSION(:,:,:), POINTER :: eh_rv_1 . . . END TYPE (domain) CONTAINS SUBROUTINE allocate_space_field ( grid , . . . ) TYPE (domain), POINTER :: grid IF ( dyn_opt == DYN_EH ) THEN ALLOCATE( grid%eh_ru_1(ims:ime,kms:kme,jms:jme) ALLOCATE( grid%eh_ru_1(ims:ime,kms:kme,jms:jme) . . . ELSE . . . • Integrate: • Time loop • Nesting (recursive) • Calls to I/O driver generated by Registry • Solve_eh: • Sequence through tile loops calling model layer routines for dynamics and physics • Multi-threading (OpenMP) • Interprocessor communi- cation • Solve_interface: • 1 step on 1 domain • Select dyncore • Dereference ADT • Solve_eh: • Sequence through tile loops calling model layer routines for dynamics and physics • Multi-threading (OpenMP) • Interprocessor communi- cation mediation • Top-level model layersubroutines: • One tile computation • Boundary conditions/avoidance • May select physics opt. • May dereference 4D fields • Top-level model layersubroutines: • One tile computation • Boundary conditions/avoidance • May select physics opt. • May dereference 4D fields • Top-level model layersubroutines: • One tile computation • Boundary conditions/avoidance • May select physics opt. • May dereference 4D fields model Code structure

  8. WRF main: • Top-level flow of control • - Initialize packages • - Input config info • - Allocate, initialize, • decompose main domain • Start integration • Shutdown • Integrate: • Time loop • Nesting (recursive) • Calls to I/O driver • Solve_eh: • Sequence through tile loops calling model layer routines for dynamics and physics • Multi-threading (OpenMP) • Interprocessor communi- cation • Solve_interface: • 1 step on 1 domain • Select dyncore • Dereference ADT • Solve_eh: • Sequence through tile loops calling model layer routines for dynamics and physics • Multi-threading (OpenMP) • Interprocessor communi- cation mediation • Top-level model layersubroutines: • One tile computation • Boundary conditions/avoidance • May select physics opt. • May dereference 4D fields • Top-level model layersubroutines: • One tile computation • Boundary conditions/avoidance • May select physics opt. • May dereference 4D fields • Top-level model layersubroutines: • One tile computation • Boundary conditions/avoidance • May select physics opt. • May dereference 4D fields model Code structure pseudo code for integration(frame/module_integrate.F) RECURSIVE SUBROUTINE integrate ( grid, start_step, end_step ) DO step = start_step , end_step CALL solver ( grid ) ! Advance 1 step WHILE ( active nests of grid ) CALL force_nest( grid, grid->child(i), mapping, subset, interpolator ) CALL integrate ( grid->child(i), (step-1)*(nest_ratio**nest_level), (step)*(nest_ratio**nest_level) CALL fdbck_nest( grid->child(i), grid, mapping, subset, interpolator ) END WHILE END DO END SUBROUTINE

  9. WRF main: • Top-level flow of control • - Initialize packages • - Input config info • - Allocate, initialize, • decompose main domain • Start integration • Shutdown pseudo code for solve interface (share/solve_interface.F) SUBROUTINE interface ( grid ) IF ( dyn_opt == DYN_EH ) THEN CALL solve_eh ( grid->eh_ru, grid->eh_rv, grid->eh_rtb, grid->eh_rtb, grid->eh_rw, . . . ) ELSE IF ( dyn_opt == DYN_EM ) THEN CALL solve_em ( . . . ) ELSE IF ( dyn_opt == DYN_SL ) THEN CALL solve_sl ( . . . ) ELSE IF ( dyn_opt . . . ENDIF END SUBROUTINE • Integrate: • Time loop • Nesting (recursive) • Calls to I/O driver • Solve_eh: • Sequence through tile loops calling model layer routines for dynamics and physics • Multi-threading (OpenMP) • Interprocessor communi- cation • Solve_interface: • 1 step on 1 domain • Select dyncore • Dereference ADT • Solve_eh: • Sequence through tile loops calling model layer routines for dynamics and physics • Multi-threading (OpenMP) • Interprocessor communi- cation generated by Registry mediation • Top-level model layersubroutines: • One tile computation • Boundary conditions/avoidance • May select physics opt. • May dereference 4D fields • Top-level model layersubroutines: • One tile computation • Boundary conditions/avoidance • May select physics opt. • May dereference 4D fields • Top-level model layersubroutines: • One tile computation • Boundary conditions/avoidance • May select physics opt. • May dereference 4D fields model Code structure

  10. pseudo code for eh solver (dyn_eh/solve_eh.F) SUBROUTINE solve_eh ( ru, rv, rtb, rtb, rw, . . . ) dummy argument declarations i1 data declarations INTEGER, DIMENSION(max_tiles) :: i_start, i_end, j_start, j_end CALL get_tiles ( numtiles, i_start, i_end , j_start, j_end ) . . . #include “HALO_EH_A.inc” !$OMP DO PARALLEL DO ij = 1, numtiles its = i_start(ij) ; ite = i_end(ij) jts = j_start(ij) ; jte = j_end(ij) CALL model_subroutine( arg1, arg2, . . . ids , ide , jds , jde , kds , kde , ims , ime , jms , jme , kms , kme , its , ite , jts , jte , kts , kte ) END DO . . . END SUBROUTINE • WRF main: • Top-level flow of control • - Initialize packages • - Input config info • - Allocate, initialize, • decompose main domain • Start integration • Shutdown • Integrate: • Time loop • Nesting (recursive) • Calls to I/O driver generated by Registry • Solve_eh: • Sequence through tile loops calling model layer routines for dynamics and physics • Multi-threading (OpenMP) • Interprocessor communi- cation • Solve_interface: • 1 step on 1 domain • Select dyncore • Dereference ADT • Solve_eh: • Sequence through tile loops calling model layer routines for dynamics and physics • Multi-threading (OpenMP) • Interprocessor communi- cation mediation • Top-level model layersubroutines: • One tile computation • Boundary conditions/avoidance • May select physics opt. • May dereference 4D fields • Top-level model layersubroutines: • One tile computation • Boundary conditions/avoidance • May select physics opt. • May dereference 4D fields • Top-level model layersubroutines: • One tile computation • Boundary conditions/avoidance • May select physics opt. • May dereference 4D fields model Code structure

  11. template for model layer subroutine SUBROUTINE model ( & arg1, arg2, arg3, … , argn, & ids, ide, jds, jde, kds, kde, & ! Domain dims ims, ime, jms, jme, kms, kme, & ! Memory dims its, ite, jts, jte, kts, kte ) ! Tile dims IMPLICIT NONE ! Define Arguments (S and I1) data REAL, DIMENSION (kms:kme,ims:ime,jms:jme) :: arg1, . . . REAL, DIMENSION (ims:ime,jms:jme) :: arg7, . . . . . . ! Define Local Data (I2) REAL, DIMENSION (kts:kte,its:ite,jts:jte) :: loc1, . . . . . . ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO i = its, ite DO k = kts, kte IF ( i > ids .AND. I < ide ) THEN loc(k,i,j) = arg1(k,i,j) + … ENDIF END DO END DO END DO • Domain dimensions • Size of logical domain • Used for bdy tests, etc. • Memory dimensions • Used to dimension dummy arguments • Do not use for local arrays • Tile dimensions • Local loop ranges • Local array dimensions

  12. Single version of code for efficient execution on: Distributed-memory Shared-memory Clusters of SMPs Vector and microprocessors WRF Multi-Layer Domain Decomposition Logical domain 1 Patch, divided into multiple tiles Model domains are decomposed for parallelism on two-levels • Patch: section of model domain allocated to a distributed memory node • Tile: section of a patch allocated to a shared-memory processor within a node; this is also the scope of a model layer subroutine. • Distributed memory parallelism is over patches; shared memory parallelism is over tiles within patches Inter-processor communication

  13. Halo updates Periodic boundary updates Parallel transposes Interface to code through the mediation layer dyn_eh/solve_eh.F SUBROUTINE solve_eh ( & . . . ) IMPLICIT NONE . . . code before communication #include “HALO_EH_A.incl” code after communication . . . Generated by Registry Distributed Memory Communications

  14. pseudo code for eh solver (dyn_eh/solve_eh.F) SUBROUTINE solve_eh ( . . . ) USE module_tiles . . . INTEGER, DIMENSION(max_tiles) :: i_start, i_end, j_start, j_end CALL get_tiles ( numtiles, i_start, i_end , j_start, j_end ) . . . !$OMP DO PARALLEL DO ij = 1, numtiles its = i_start(ij) ; ite = i_end(ij) jts = j_start(ij) ; jte = j_end(ij) CALL model_subroutine( arg1, arg2, . . . ids , ide , jds , jde , kds , kde , ims , ime , jms , jme , kms , kme , its , ite , jts , jte , kts , kte ) END DO . . . END SUBROUTINE SharedMemory Parallelism

  15. Controlling parallelism at runtime • Shared memory number of tiles • 1 tile if not compiled for OpenMP or if there is only one thread available • unless: numtiles > 1 in namelist • unless: tile size is specified in namelist as tile_sz_x > 0 and tile_size_y > 0 • The tiling dimension is specified at compile time in frame/module_machine.F as either TILE_Y, TILE_X, or TILE_XY (this is overridden by tile_sz_x and tile_sz_y)

  16. Controlling parallelism at runtime • Distributed memory patches • The number of patches is always the same as the number of MPI processes, which WRF learns by interrogating the particular comm package (e.g. RSL, which returns the value of MPI_Comm_size) • The patching algorithm is built in and choses a decomposition that is closest to square (should be more controllable) • unless: numtiles > 1 in namelist • unless: tile size is specified in namelist as tile_sz_x > 0 and tile_size_y > 0 • The tiling dimension is specified at compile time in frame/module_machine.F as either TILE_Y, TILE_X, or TILE_XY (this is overridden by tile_sz_x and tile_sz_y)

  17. WRF model data structures • State data • Fields in domain data type, defined in Registry • Decomposed 2- and 3-D arrays; dimensions correspond to physical domain dimensions • 4-D “scalar” arrays (e.g. moist); accessible individually or en masse • Boundary arrays • Misc. un-decomposed arrays and 0-dimensional variables • Dimensioned using “memory” dimensions • Allocated using F90 ALLOCATE • I1 (local to solver) data • Also defined in Registry but not in domain data type • Dimensioned using memory dimensions • Automatic allocation (usually on main program stack) • I2 (local to model layer subroutine) data • Defined only in subroutine • Defined using “tile” dimensions • Automatic allocation (usually on thread-local stacks)

  18. WRF Registry • CASE mechanism for managing complexity of WRF code: • Data base (ASCII flat file) of information about code • State data and attributes • Dimensionality, time levels, core association, other metadata • I/O dataset membership • Configuration data • Packages (including multiple dyncores) with data associations • Communication definitions • Mechanism for compile-time generation of • Data definitions, allocations • Driver/mediation layer interfaces • I/O mechanisms • Communication mechanisms • WRF 1.2 Registry rewritten • Added Abstract Data Types (3DVAR) • Additional communication options (e.g. transposes) • More general data definition semantics

  19. Registry Data Base • ACSCII flat file: Registry/Registry • Types of entry: • Dimspec -- Describes dimensions that are used to define arrays in the model • State – Describes state variables and arrays in the domain DDT • I1 – Describes local variables and arrays in solve • Typedef -- Describes derived types that are subtypes of the domain DDT • Rconfig – Describes a configuration (e.g. namelist) variable or array • Package – Describes attributes of a package (e.g. physics) • Halo -- Describes halo update interprocessor communications • Period -- Describes communications for periodic boundary updates • Xpose -- Describes communications for parallel matrix transposes • Mechanism in tools directory; program name is “registry”; used by WRF build procedure

  20. Dimspec entry • Elements • Entry: The keyword “dimspec” • DimName: The name of the dimension (single character) • Order: The order of the dimension in the WRF framework (1, 2, 3, or ‘-‘) • HowDefined: specification of how the range of the dimension is defined • CoordAxis: which axis the dimension corresponds to, if any (X, Y, Z, or C) • DatName: metadata name of dimension • Example #<Table> <Dim> <Order> <How defined> <Coord-axis> <DatName> dimspec i 1 standard_domain x west_east dimspec j 3 standard_domain y south_north dimspec k 2 standard_domain z bottom_top dimspec l 2 namelist=num_soil_layers z soil_layers

  21. State entry • Elements • Entry: The keyword “state” • Type: The type of the state variable or array (real, double, integer, logical, character, or derived) • Sym: The symbolic name of the variable or array • Dims: A string denoting the dimensionality of the array or a hyphen (-) • Use: A string denoting association with a solver or 4D scalar array, or a hyphen • NumTLev: An integer indicating the number of time levels (for arrays) or hypen (for variables) • Stagger: String indicating staggered dimensions of variable (X, Y, Z, or hyphen for no staggering) • IO: String indicating whether and how the variable is subject to I/O • DName: Metadata name for the variable • Units: Metadata units of the variable • Descrip: Metadata description of the variable • Example # Type Sym Dims Use Tlev Stag IO Dname Descrip # definition of a 3D, two-time level, staggered state array state real ru ikj dyn_eh 2 X irh "RHO_U" "X WIND COMPONENT“ # definition of fields in 4D scalar array “moist” state real qv ikjft moist 2 - irh "QVAPOR" "Water vapor mix ratio“ state real qc ikjft moist 2 - irh "QCLOUD" "Cloud water mixing ratio"

  22. Rconfig entry • Elements • Entry: the keyword “rconfig” • Type: the type of the namelist variable (integer, real, logical – no strings yet) • Sym: the name of the namelist variable or array • Howset: indicates how the variable is set: e.g. namelist or derived, and if namelist, which block of the namelist it is set in • Nentries: specifies the dimensionality of the namelist variable or array. If 1 (one) it is a variable and applies domain-wide; otherwise specify max_domains (which is an integer parameter defined in module_driver_constants.F). • Default: the default value of the variable to be used if none is specified in the namelist; hyphen (-) for no default • Example # Type Sym How set Nentries Default rconfig integer dyn_opt namelist,namelist_01 1 1

  23. Package Entry • Elements • Entry: the keyword “package”, • Packagename: the name of the package: e.g. “kesslerscheme” • Associatedrconfigchoice: the name of a rconfig variable and the value of that variable that choses this package • Packagestatevars: unused at present; specify hyphen (-) • Associated 4D scalars: the names of 4D scalar arrays and the fields within those arrays this package uses • Example # namelist entry that controls microphysics option rconfig integer mp_physics namelist,namelist_04 max_domains 0 # specification of microphysics options package passiveqv mp_physics==0 - moist:qv package kesslerscheme mp_physics==1 - moist:qv,qc,qr package linscheme mp_physics==2 - moist:qv,qc,qr,qi,qs,qg package ncepcloud3 mp_physics==3 - moist:qv,qc,qr package ncepcloud5 mp_physics==4 - moist:qv,qc,qr,qi,qs

  24. Comm entries: halo and period • Elements • Entry: keywords “halo” or “period” • Commname: name of comm operation • Description: defines the halo or period operation • For halo: npts:f1,f2,...[;npts:f1,f2,...]* • For period: width:f1,f2,...[;width:f1,f2,...]* • Example # first exchange in eh solver halo HALO_EH_A 24:u_2,v_2,ru_1,ru_2,rv_1,rv_2,w_2,t_2;4:pp,pip # a periodic boundary update period PERIOD_EH_A 2:u_1,u_2,ru_1,ru_2,v_1,v_2,rv_1,rv_2,rw_1,rw_2

  25. Additional Information • wrfhelp@ucar.edu • www.wrf-model.org • WRF Design and Implementation (draft) • Tomorrow: • How to make changes in WRF code

More Related