1 / 44

Technical details on how to use NICAM

22Oct2009 COLA seminar. Technical details on how to use NICAM. Chihiro Kodama Research Institute for Global Change (RIGC) Japan Agency for Marine-Earth Science and Technology (JAMSTEC). Contents of my talk. How to use NICAM (technical details) compile

marci
Download Presentation

Technical details on how to use NICAM

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 22Oct2009 COLA seminar Technical details on how to use NICAM Chihiro Kodama Research Institute for Global Change (RIGC) Japan Agency for Marine-Earth Science and Technology (JAMSTEC)

  2. Contents of my talk • How to use NICAM (technical details) • compile • pre-processes (horizontal grid, external condition, initial condition) • run (configuration) • post-processes (lat-lon data) • trouble shooting (runtime error) • Following tutorial package (updated 2009.10.16) • http://nicam.jp/tutorial/ (password needed) • Please feel free to ask questions!

  3. Compile NICAM % ln –s Mkinclude.ES2 Mkinclude <- link Mkinclude.[arch] % emacs Mkinclude <- change $TOPPATH to NICAM-dir. % make TOPPATH: NICAM source code path NOTE: Please mind that unit length of record must be 1 byte. In addition, input/output files are big-endian. Intel ifort:-assume byterecl (compiler option) setenv F_UFMTENDIAN big Earth Simulator 2: setenv F_RECLUNIT BYTE

  4. Managing Information Files (mnginfo) • They contain: • number of regions • linkages between one and other regions • number of processes • process-region mapping (Which process control a region?) • mkmnginfo.cnf necessary to run MPI-based program with 320 processes, rlevel=5. &MKMNGINFO rlevel = 5, prc_num = 320, output_fname = ‘rl5-prc320.info’, / % ${TOPPATH}/bin/mkmnginfo Concept of rlevel will be explained later.

  5. Concept of Horizontal Grid g: glevel number of rhomboses = 10×4g S: Earth Surface [m2] resolution ~ sqrt [S / (10×4g)] • What is “glevel” ? • Total Grid Division Level glevel 3 893 km glevel 0 7141 km glevel resolution 6 112 km 7 56 km 8 28 km 9 14 km 10 7 km 11 3.5 km glevel 4 446 km glevel 1 3570 km glevel 5 223 km glevel 2 1785 km Figures from http://www.ccsr.u-tokyo.ac.jp/~satoh/nicam/ico.html

  6. Concept of Horizontal Grid rlevel # of regions 0 10 1 40 2 160 3 640 4 2560 5 10240 • What is “rlevel” ? • Region Division Level • Designed for MPI program • File name: var-name.rgn????? e.g. sst.rgn00000 ~ sst.rgn10239 • Note: As rlevel increases, total memory consumption is increased, especially when glevel-rlevel is small. • due to overlapped grid along edge of the region. region region If using 20 processes, 128 regions per process rlevel 1 40 regions / globe rlevel 0 10 regions / globe

  7. Flow of Horizontal Grid Generation generate grid mkgrid _grid.rgn????? (for rlevel=0) provided in the package if rlevel=0 splitgrid increase rlevel _grid.rgn????? (for rlevel > 0) mkgcgrid modify grid (gravitational-centered) Horizontal Grid Files grid.rgn?????

  8. Generating Horizontal Grid (1/2) splitgrid • splitgrid.cnf (e.g. glevel=10, rlevel=5) &ADMPARAM glevel = 10, rlevel = 0, vlayer = 1, rgnmngfname ='../../mnginfo/data/rl00-prc05.info', / &SPLITGRIDPARAM input_base = '../../../NICAM_DATABASE/_hgrid/gl10/rl00/grid', output_base = 'data/_grid', alt_rlevel = 5, / % mpirun -np 5 ${TOPPATH}/bin/splitgrid Output: _grid.rgn00000, _grid.rgn00001, ..., _grid.rgn10239

  9. Generating Horizontal Grid (2/2) mkgcgrid • mkgcgrid.cnf (e.g. glevel=10, rlevel=5) &ADMPARAM glevel = 10, rlevel = 5, vlayer = 1, rgnmngfname = '../../mnginfo/data/rl05-prc05.info', / &MKGCGRIDPARAM input_base = '../splitgrid/data/_grid', output_base = 'data/grid', / % mpirun -np 5 ${TOPPATH}/bin/mkgcgrid Output: grid.rgn00000, grid.rgn00001, ..., grid.rgn10239

  10. External Condition Files (external_data/) • MATSIRO : Minimal Advanced Treatments of Surface Interaction and RunOff • MATSIRO/ • topog: topography • topog_std: standard deviation of topography • albedo_*: surface albedo • gradz: tangent of mean surface slope • lai: leaf area index • slidx: soilindex • veget_mat: vegetation index • O3/ : ozone • OCN/ (for nudging slab ocean) • sst: sea surface temperature [K] • ice: sea ice mass [kg/m2] • ice is NEITHER sea ice concentration NOR sea ice fraction.

  11. Topography Files (MATSIRO/) • To avoid numerical instability, use topography files included in the tutorial package. • For higher rlevel run, split the above files using cnvrlev, which increases rlevel of general data • cnvrlev.cnf &ADMPARAM ... / &SPLITGRIDPARAM num_data = 1, input_base = '../../NICAM_DATABASE/sfcdata/gl10/rl04/topog', input_direct = .false., output_base = 'data/topog', output_direct = .false., output_rlevel = 5, / % mpirun -np 5 ${TOPPATH}/bin/cnvrlev Output: topog.rgn00000, topog.rgn00001, ..., topog.rgn10239

  12. Other MATSIRO Files / O3 Files • See tutorial package for details. Probably, all you have to do is modify glevel & rlevel. • mklanddata2.cnf • mko3.cnf

  13. OCN Files (1/2) • SST: sea surface temperature [K] • mkgisst.cnf &ADMPARAM ... / &GRDPARAM ... / &MKGISSTPARAM sst_lldata_fname = '../../../dataset/oisst_v2/sst.grd', output_sst_base = './data/sst', num_data = 8, data_periodic = .false., imax = 360, jmax = 180, slat = 89.5, slon = 0.5, dlat = 1.0, dlon = 1.0, opt_southward = .false., direct_access = .true., / % mpirun -np 5 ${TOPPATH}/bin/mkgisst

  14. OCN Files (2/2) • ICE: sea ice mass [kg/m2] • In our previous package, sea ice fraction is used. It’s wrong! • mkgisst.cnf &ADMPARAM ... / &GRDPARAM ... / &MKGISSTPARAM sst_lldata_fname = '../../../NICAM_DATABASE/rawdata/hadsst1979-1999_clm.dat', ice_lldata_fname = '../../../NICAM_DATABASE/rawdata/cmip3sit1979-1999_clm.dat', output_sst_base = './data/hadsst1979-1999_clm', output_ice_base = './data/cmip3ice1979-1999_clm', num_data = 12, data_periodic = .false., direct_access = .true., offset = 0.0, ice_fact = 1000.0, / NOTE: SST above (HadSST) is used only for modifying ice.

  15. Initial Condition Files (1/2) Most variables can be obtained from NCEP Final Analysis • Atmosphere • 3D velocity • temperature • water vapor mixing ratio • cloud water mixing ratio • Ocean (for slab ocean) • SST • sea ice fraction & mass & surface temperature • snow on sea ice • Land (for MATSIRO) • soil moisture & temperature • surface accumulated snow

  16. Initial Condition Files (2/2) • Atmosphere • mkinit_atm.cnf &MKINIT_ATM_PARAM glevel = 10, rlevel = 5, rain_type = 'COLD', plus_tke = .true., hgrid_basename = '../../../NICAM_DATABASE/hgrid/gl10/rl05/grid.', vgrid_filename = '../../grid/vgrid/data/vgrid40.dat', topo_basename = '../../../NICAM_DATABASE/sfcdata/gl10/rl05/topog', ncep_atm_filename = '../atm.dat', init_atm_basename = 'init_atm', / After you prepare atm.dat (see tutorial for data format), % ${TOPPATH}/bin/mkinit_atm Preparing ocean and land conditions is similar.

  17. Prep. for Icosahedral Data  Lat-Lon Data • Lat-Lon mapping files (llmap) are necessary • mkllmap.cnf &ADMPARAM glevel = 10, rlevel = 5, vlayer = 1, rgnmngfname = '../../mnginfo/data/rl05-prc05.info', / &GRDPARAM hgrid_fname = '../../../NICAM_DATABASE/hgrid/gl10/rl04/grid‘, / &LATLONPARAM imax = 5120, jmax = 2560, / &MKLLMAP_PARAM output_dir = './data/‘, / % mpirun -np 5 ${TOPPATH}/bin/mkllmap Output: llmap.rgn00000, llmap.rgn00001, ..., llmap.rgn10239

  18. Final Step for NICAM simulation % mpirun -np 320 ${TOPPATH}/bin/nhm_driver • You have to prepare nhm_driver.cnf. • See tutorial package for more detailed descriptions of nhm_driver.cnf

  19. nhm_driver.cnf (Run Type Settings) &RUNCONFPARAM EIN_TYPE = 'SIMPLE', RAIN_TYPE = 'COLD', NDIFF_LOCATION = 'IN_LARGE_STEP2' RAD_TYPE = 'ISCCP', TRC_ADV_TYPE = 'MIURA2004', TRC_NEG_FIX = 'OFF', AF_TYPE = 'NONE', MP_TYPE = 'NSW6', RD_TYPE = 'MSTRNX_AR5', ND_TYPE = 'NONE', FR_TYPE = 'NONE', SV_TYPE_T = 'GIVEN', SV_TYPE_QV = 'SATURATION', SF_TYPE = 'LOUIS', TB_TYPE = 'MY2MOIST', CP_TYPE = 'NONE', GWD_TYPE = 'NONE', LAND_TYPE = 'MATSIRO', OCEAN_TYPE = 'MIXEDLAYER', TB_DIV_NUM = 1, SFC_DIV_NUM = 1, / &RUNCONFPARAM EIN_TYPE = 'SIMPLE', RAIN_TYPE = 'CLOUD_PARAM', NDIFF_LOCATION = 'IN_LARGE_STEP2', RAD_TYPE = 'ISCCP', TRC_ADV_TYPE = 'MIURA2004', TRC_NEG_FIX = 'OFF', AF_TYPE = 'NONE', MP_TYPE = 'LSC', RD_TYPE = 'MSTRNX_AR5', ND_TYPE = 'NONE', FR_TYPE = 'NONE', SV_TYPE_T = 'GIVEN', SV_TYPE_QV = 'SATURATION', SF_TYPE = 'LOUIS', TB_TYPE = 'MY2MOIST', CP_TYPE = 'PAS', GWD_TYPE = 'NONE', LAND_TYPE = 'MATSIRO', OCEAN_TYPE = 'MIXEDLAYER', TB_DIV_NUM = 24, SFC_DIV_NUM = 12, / For high resolution run For low resolution run

  20. nhm_driver.cnf (Time Step Control) &TIMEPARAM INTEG_TYPE = 'RK2', LSTEP_MAX = 8640, SSTEP_MAX = 4, DTL = 30.0D0, SPLIT = .TRUE., start_year = 2004, start_month = 6, start_day = 1, / ##### <--- default : RK2 with SSTEP_MAX = 4; ##### <--- But, if unstable, use RK3 with SSTEP_MAX=6. number of time step Δt [s] start data/time

  21. nhm_driver.cnf (Initial Conditions 1/2) &RESTARTPARAM input_basename = '../../initial_data/ATM/init_atm' input_direct_access = .true., output_direct_access = .true., trc_vmax_input = 3, / ##### <--- when restart, delete “TRC_VMAX_INPUT=3”, &NM_OCEAN_INIT merged_rst_in = .true., merged_rst_out = .true., merged_input_restfname = '../../initial_data/OCN/init_ocn', merged_output_restfname = 'restart_ocn', / &NM_LAND_INIT merged_rst_in = .true., merged_rst_out = .true., merged_input_restfname = '../../initial_data/LND/init_lnd', merged_output_restfname = 'restart_lnd', opt_albsfc_in = .false., opt_albsfc_out = .true., / ###### <--- When restart, opt_albsfc_in = .true. Atmosphere Ocean Land

  22. nhm_driver.cnf (Initial Conditions 2/2) &DIAGVARPARAM tb_fname = 'NONE', input_direct_access = .true., output_direct_access = .true., / ###### <--- When restart, set TB_FNAME (restart_diagvar) Turbulence • When you restart run, • &RESTARTPARAM • change input_basename • delete trc_vmax_input • &NM_OCEAN_INIT • change merged_input_restfname • &NM_LAND_INIT • change merged_input_restfname • let opt_albsfc_in = .true. • &DIAGVARPARAM • set tb_fname Normally, no initial conditions for turbulence

  23. nhm_driver.cnf (External Conditions 1/2) • Case 1: Time invariant field • e.g. ‘SLIDX’, ‘GRTANS’, ‘GRALB’, ‘GRALBN’ • Case 2: Monthly-constant field • e.g. ‘OZONE’ &NM_EXTDATA dataname = 'GRTANS', fname = '../../external_data/MATSIRO/data/gradz/gradz', num_of_data = 1, layer_type = 'SFC', / &NM_EXTDATA dataname ='OZONE', fname = '../../external_data/O3/data/amip2o3', num_of_data = 12, opt_monthly_cnst = .true., layer_type = 'ATM', /

  24. nhm_driver.cnf (External Conditions 2/2) • Case 3: Step-by-step field (with time interpolation) • e.g. ‘SST’, ‘ICE’, ‘GRLAI’ &NM_EXTDATA dataname = 'SST', fname = '../../external_data/OCN/data/sst', num_of_data = 8, data_date(:,1) = 2004, 5,26,0,0,0, data_date(:,2) = 2004, 6,02,0,0,0, data_date(:,3) = 2004, 6,09,0,0,0, data_date(:,4) = 2004, 6,16,0,0,0, data_date(:,5) = 2004, 6,23,0,0,0, data_date(:,6) = 2004, 6,30,0,0,0, data_date(:,7) = 2004, 7,07,0,0,0, data_date(:,8) = 2004, 7,14,0,0,0, layer_type = 'SFC', opt_periodic_year = .false., / ###### <--- PLEASE CHECK THE DATES BEFORE RUNNING!!!

  25. nhm_driver.cnf (Output Variables) &NMHIST ktype = ‘3D’, item = ‘ml_tem’, file = ‘ms_tem’, output_type = ‘SNAPSHOT’, step = 360, / &NMHIST ktype = ‘2D’, item = ‘sl_slp’, file = ‘sa_slp’, output_type = ‘AVERAGE’, step = 120, / variable name file name ( ms_tem.rgn?????) • ktype = ‘3D’, ‘2D’, ‘GO’ (ocean), ‘GL’(land), ‘ISCCP’ ‘SNAPSHOT’ or ‘AVERAGE’ timestep interval of output if DTL=30s, step = 120: 1 hourly step = 360: 3 hourly

  26. nhm_driver.cnf (Output Variables) &NMHISD direct_access = .TRUE., no_vintrpl = .false., / • &NMHISD: default setting of &NMHIST • Output with pressure levels .true. for z* coordinate .false. for z coordinate &NMHISD ... npreslev = 23,   pres_levs = 1000, 925, 850, 700, 600, 500, 400, 300, 250, 200, 150, 100, 70, 50, 30, 20, 10, 7, 5, 3, 2, 1, 0.4, / &NMHIST ktype = ‘3D’, item = ‘ml_tem’, file = ‘ms_tem’, output_type = ‘SNAPSHOT’, step = 360, out_prelev = .true., /

  27. Run NICAM ! % mpirun -np 320 ${TOPPATH}/bin/nhm_driver • See tutorial package for more detailed descriptions of nhm_driver.cnf Output: ms_tem.rgn?????, sa_slp.rgn?????, ..... (huge number of files are created)

  28. Icosahedral Data  Lat-Lon Data &ICO2LL_PARAM glevel = 10, rlevel = 5, input_dir = './icodata/', output_dir = './lldata/' info_fname = 'history.info', llmap_dir = '../../grid/llmap/data/' llmap_base = 'llmap', lon_swap = .true., input_size = 4, init_year = 2004, init_month = 6, init_day = 1, / • ico2ll.cnf automatically generated by nhm_driver % ${TOPPATH}/bin/ico2ll Output: ms_tem.ctl, ms_tem.grd, sa_slp.ctl, sa_slp.grd, .....

  29. Restarting Run • Followings should be changed in nhm_driver.cnf: • &TIMEPARAM: start_* • External condition • &RESTARTPARAM • &NM_OCEAN_INIT • &NM_LANDINIT • &DIAGVARPARAM

  30. Changing Glevel • Followings should be changed in nhm_driver.cnf: • &RUNCONFPARAM (e.g. cloud resolving / cloud param.) • &ADMPARAM • glevel, rlevel, rgnmngfname • &NUMFILTERPARAM (as glevel increase by 1,) • Kh_coef_maxlim , Kh_coef_minlim, alpha_d : divided by 8 • gamma_h_lap1 : divided by 2 • &TIMEPARAM • dtl, lstep_max (if necessary) • &NM_RD_TIME • TINTV= 10 mins for gl9, but 5 mins for gl10 and 11 • File names

  31. Handling Runtime Error • Model bug? Configuration mistake? Numerical instability? • Model run diagnosis • msg.????? : message from NICAM • standard out/err • Check List • Where is the run fail? What module/subroutine/line? • Are the prescribed files such as external_data and initial_data correct? • check them using ico2ll • unit of variables • glevel/rlevel • file name, little/big endian • compiler option (such as “-assume byterecl” in intelifort) • memory enough for the run? • parameters in nhm_driver.cnf correct? • Does the error also occur in the lower glevel/rlevel? If the runtime error seems to occur due to numerical instability, ...

  32. Numerical Filter in nhm_driver.cnf &NUMFILTERPARAM dep_hgrid = .false. DEEP_EFFECT = .false., hdiff_fact_rho = 0.01D0, hdiff_fact_q = 0.0d0, hdiff_type = 'NONLINEAR1', cfact = 2.0D0, Kh_coef_maxlim = 3.125D+12, Kh_coef_minlim = 3.125D+11, ZD_hdiff_nl = 20000D0, gamma_h_lap1 = 6.25D+5, ZD_hdiff_lap1 = 20000.0D0, divdamp_type = 'DIRECT', lap_order_divdamp = 2, alpha_d = 3.125D+11, alpha_dv = 0.00D0, gamma_v = 0.0D0, ZD = 25000.0d0, alpha_r = 0.0D0, / In addition, 2D divergence damping horizontal diffusion Laplacian-type horizontal diffusion 3D divergence damping vertical diffusion Rayleigh friction

  33. Other Configurations in nhm_driver.cnf &nm_tb_my2moist ICE = .true., LENGTH_TYPE = '3KM', NU_MIN = 0.1D0, KH_MIN = 0.1D0, N2max = 0.0016, / ##### <--- If unstable, reduce the N2max. &TIMEPARAM INTEG_TYPE = 'RK2', LSTEP_MAX = 8640, SSTEP_MAX = 4, DTL = 30.0D0, SPLIT = .TRUE., start_year = 2004, start_month = 6, start_day = 1, / ##### <--- default : RK2 with SSTEP_MAX = 4; ##### <--- But, if unstable, use RK3 with SSTEP_MAX=6.

  34. Adjusting Initial Conditions • Tool from Tatsuya Seiki (CCSR, Univ. Tokyo) • modify pressure to achieve hydrostatic balance • modify vertical velocity using continuity equation

  35. Summary • Please inform us if you find bugs, misleading points, suggestions, etc in NICAM tutorial package. Thank you

  36. ##### Appendix / Memo #####

  37. 0. Who am I ? • Research interest • storm track • stratosphere dynamics • global warming • role of cloud in climate system • Misc • GrADS script library

  38. Memory Consumption as rlevel increase

  39. Generating Vertical Grid 0.000000000000000E+000 161.683000000000 335.958000000000 523.806000000000 726.285000000000 944.534000000000 1179.78100000000 1433.34900000000 1706.66700000000 2001.27200000000 2318.82200000000 2661.10400000000 3030.04500000000 3427.72000000000 3856.36800000000 4318.40000000000 4816.41800000000 5353.22300000000 5931.83700000000 6555.51600000000 7227.76900000000 7952.38000000000 8733.42800000000 9575.30600000000 10482.7530000000 11460.8760000000 12515.1790000000 13651.5960000000 14876.5210000000 16196.8500000000 17620.0100000000 19154.0120000000 20807.4880000000 22589.7440000000 24510.8110000000 26581.4960000000 28813.4570000000 31219.2520000000 33812.4180000000 36607.5550000000 39620.3830000000 • mkvlayer.cnf &MKVLAYER_CNF num_of_layer = 40, layer_type = 'GIVEN', infname = './vgrid40.txt', outfname = './data/vgrid40.dat', / TODO: how to create refstate or No explanation of vertical grid

  40. refstate

  41. nhm_driver.cnf &nm_rd_time TINTV=10.0D0, TUNIT='MIN' / ###### <--- TINTV = 10min for gl9, but 5min for gl10 and 11.

  42. nhm_driver.cnf (Resolution, Grid) &ADMPARAM glevel = 10, rlevel = 5, vlayer = 40, rgnmngfname = '../../mnginfo/data/rl05-prc320.info', / &GRDPARAM hgrid_fname = '../../grid/hgrid/data/grid', vgrid_fname = '../../grid/vgrid/data/vgrid40.dat', topo_fname = '../../external_data/MATSIRO/data/topog/topog', toposd_fname = '../../external_data/MATSIRO/data/topog_std/topog_std', vegeindex_fname = '../../external_data/MATSIRO/data/veget_mat/veget_mat' /

  43. nhm_driver.cnf (Ocean) &nm_ocean_mixedlayer osstfx = .false., oicefx = .false., oicrfx = .false., TNUGST = 432000.0D0, TNUGSI = 432000.0D0, / ###### <--- The relaxation times of SST and Sea ICE are 5 days.

More Related