1 / 34

NOAA’s Microwave Integrated Retrieval System Overview of Installation and Operation

NOAA’s Microwave Integrated Retrieval System Overview of Installation and Operation. Kevin Garrett 1 , Christopher Grassotti 1 , Leslie Moy 1 , Flavio Iturbide-Sanchez 1 , Wanchun Chen 2 , and Sid-Ahmed Boukabara 3 CPTEC/INPE 24 March, 2011

fruma
Download Presentation

NOAA’s Microwave Integrated Retrieval System Overview of Installation and Operation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NOAA’s Microwave Integrated Retrieval SystemOverview of Installation and Operation Kevin Garrett1, Christopher Grassotti1, Leslie Moy1, Flavio Iturbide-Sanchez1, Wanchun Chen2, and Sid-Ahmed Boukabara3 CPTEC/INPE 24 March, 2011 1. I. M. Systems Group 2. Dell, Inc. 3. NOAA/NESDIS/STAR, JCSDA

  2. Agenda • Overview of MiRS • Supported platforms/compilers • Installation procedures • Configuration, compilation and testing • Directory structure and processing units • MiRS operation • The MiRS GUI • Running from the command line • Running individual programs • Extension to currently unsupported sensors • Code requirements • Required ancillary files • Summary

  3. Overview • NOAA/NESDIS/STAR has developed a flexible physical algorithm: the Microwave Integrated Retrieval System (MiRS) • Can be applied to any microwave sounder/imager • 1DVAR approach using CRTM as forward and jacobian operators • Retrieves sounding and surface parameters simultaneously, including hydrometeor profiles and surface emissivity • Applicable over all surfaces and in all-weather conditions • Run operationally at NOAA for NOAA-18, NOAA-19, Metop-A, DMSP F-16 and DMSP F-18. • Extended to support NPP ATMS (proxy), AMSR-E, FY3, TRMM TMI (next release), and GPM GMI (next release)

  4. IT Requirements Hardware Requirements Not necessary for standard outputs BUT required for QC DAP Software Requirements *This is minimum required. Number of CPUs will affect the speed of execution. 8 CPUs are used in STAR to run three sensors daily.

  5. Download and Extract • Download the Delivery Algorithm Package (DAP) from the STAR website. Username and Password required. mirs_dap_rel7.0_v2229_Jun30_2010.tar.gz • Move to target installation directory on Linux or Unix platform. • Run • gunzip mirs_dap_rel7.0_v2229_Jun30_2010.tar.gz • tar –xvf mirs_dap_rel7.0_v2229_Jun30_2010.tar This will extract the MiRS DAP

  6. Configure and Install Directory Structure Overview MIRS • /setup Contains configuration files for IDL and Fortran installation paths Also contains Process Configuration File (PCF) to configure processing options. • /scripts Contains the driver scripts to execute MiRS (manually or from GUI). • /logs Contains MiRS output log files to track errors/warnings. • /srcContains all MiRS source code and libraries. • /gui Contains the MiRS GUI program to configure and run MiRS. • /doc Contains the MiRS documentation. • /data Contains all Static/Semi Static data files, namelist files, and directories for storing input, intermediary and output MiRS data. • /bin Contains the MiRS executables after code has been compiled. /setup /scripts /logs /src /gui /doc /data /bin

  7. Configure and Install“paths” files • In /setup, open the “paths” file • Configure MIRS_ROOT to the MiRS installation directory • Configure HDF and NetCDF library paths (required for FY3 and NPP • Select compiler options depending on compiler and platform • Configure JAVA_HOME path if GUI use is desired • In /setup, open the paths_idl.pro • Configure paths to point to local MiRS installation (IDL required for figure generation) paths_idl.pro file paths file

  8. Configure and InstallCompiler Options • The compiler options are available in the make.macros file located in src/crtm/CRTMp_twostream_1024/src • Compatible options are Intel, g95, and xlf95 (gfortran supported with next release) • For Intel, comment or uncomment Linux_FLAGS_Intel for optimised or regular flags • Select Linux_FLAGS which reflect the desired compiler to use Choose compiler Intel flag options

  9. Configure and InstallMake Binaries • With paths set and the compiler selected, MiRS is ready to be built. • In the src directory, type “make”. This will compile all source code and install executables in the bin directory. • To compile the GUI, type “make” in the gui directory

  10. Test ProceduresConfiguration • Testing can be done running sample Sequence Control Script (SCS) and sample Process Configuration File (PCF) • First, configure the PCF in the setup directory, n18_pcf_orbit.bash. Modify the rootPath to match the MiRS installation directory and possibly LD_LIBRARY_PATH if the fortran library is not in default system location Portion of PCF file

  11. Test ProceduresExecution • From the scripts directory, run the script for n18 orbit • ./n18_scs_orbit.bash NSS.AMAX.NN.D06032.S0122.E0317.B0361819.WI Where NSS* is the N18 AMSU orbit file name • If the script is successful, it will end gracefully after the EDR & DEP IMAGE GENERATION STEP.

  12. Test ProceduresCompare to Benchmark • The next step is to verify against BenchMarkData • From src/qcDelivery, run the IDL code qc_lr.pro • Choose options for N18 and sub orbit 1 when prompted

  13. Test ProceduresCompare to Benchmark • Once the IDL has finished, choose to view the comparison scatter plots between your retrieval and the benchmark. PNG files are located in the qcDelivery directory • For reference, the benchmark data is located in data/BenchMarkData

  14. This confirms the successful implementation of MiRS. Now let’s get more details on fine tuning configuration and operation of the algorithm

  15. Running MiRSHigh Level • The highest level of MiRS operation is from the Graphical User Interface • To compile and run the GUI, run “make” and then “make run” in the gui directory (requires JAVA 1.6)

  16. Running MiRSHigh Level • The “Sensor” dropdown menu allows for the selection of desired sensor • The “Task Selection” panel shows all processing steps that can be • Some steps require the previous step • “Process Mode” may be either Orbital or Daily Mode if processing more than 1 orbit • “Generate Script” will generate the SCS file along with the PCF file with selected parameters. “Run” will run the SCS file automatically (as was done manually during the testing shown previously) • “Execution Monitoring” window shows the status of processing

  17. Running MiRSHigh Level • In this case, all necessary steps are checked to process N18 data from level 1B input, through the 1DVAR, and figure generation steps. • All subsequent steps selected rely on previous selected step. • Steps may be run in different jobs as long as intermediate data from previous step had been created from previous job. • “Chopp Orbit” and “Merge Retrieved Granules” are optional steps which take advantage of multi-CPU environments (they are interdependent)

  18. Running MiRSHigh Level • After MiRS has run and created products for a particular sensor, the system has the ability to compare to NWP data • GDAS, ECMWF, and GFS supported • Comparison is between Geophysical parameters and Radiometric observations (compared to simulations) • “NWP Collocation” step requires “Footprint Matching” step previously run • “Bias Figs Generation” requires all previous steps have been run (except for Chopping and Merging) • Optionally, all steps may be run in sequence.

  19. Running MiRSHigh Level • Configuration of the system is available through the GUI • Paths: Change binary and data paths for input, intermediate, output data • Preferences: Change operating parameters such as resolution, bias correction, processing area, single or multi-CPU, etc.

  20. Running MiRSMid Level • Alternatively, MiRS may be executed directly from the Bash script level at the command line. • These scripts (SCS) are generated originally by the GUI, along with the PCF containing Paths Settings and Preference Settings information. • Running from the script allows for flexibility in tuning algorithm parameters. • This type of interaction was demonstrated running the test script to compare with benchmark data.

  21. Running MiRSMid Level • First an SCS and PCF should be generated by the GUI for a desired sensor, with all potential steps to run (“Generate Script”). • This generates the sensorId_scs.bash script in the scripts directory, and the sensorId_pcf.bash PCF file in the setup directory. • MiRS is then executed by running the script in the scripts directory • If the script is generated with Daily Process selected in GUI, it is executed by supplying the date of the data as argument. n18_scs.bash 2011-03-21 • If the script is generated with Orbit Process selected, it is run supplying an orbit file as argument. n18_scs.bash NSS.AMAX.NN…..

  22. Running MiRSMid Level • The steps in the GUI “Task Selection” correspond 1-to-1 with function calls in the SCS file, and also to flags in the PCF (to manually turn on or off the task).

  23. Running MiRS MiRS Sequence from Level-1b Processing To EDR/DEP Output Main Inputs RDR2TDR Intermediary Data Storage NEDT Level 1B data SCS crontab TDR2SDR Error Handling FM First Guess Bash script PCF 1dvar Monitoring Formatting Validation QC F95 / C++ VIPP MiRS Data Main Outputs SCS is the driver bash script that calls all other scripts functions and programs. Each step requires that the previous step’s output data is available.

  24. Running MiRSLow Level • Alternatively, the individual MiRS Fortran and IDL codes may be executed manually from command line. • This is useful if the scripts and PCF parameters remain static, but codes have been modified and units need to be compiled. • For example, if the 1dvar.f90 program has been modified, it may be recompiled and run from the bin directory, with it’s namelist as input. • The SCS must be run at least once to generate necessary namelists for all Fortran and IDL codes for the program/step of interest (located in data/ControlData).

  25. MiRS OverviewDirectory Structure Detail MIRS /setup /scripts /logs /src /gui /doc /data /bin /1dvar /fwd /crtm /lib /lib_idl /qcDelivery /testbed • /1dvar - core 1dvar program • /fwd - interface to forward operator CRTM • /crtm - CRTM source code • /lib - MiRSlibrarys (utilities, I/O subroutines, data type definitions, inversion subroutines) • /lib_idl - IDL codes (for figures/plots/maps) • /lib_cpp - C module for NPP data reader • /lib_java - library for JAVA • /qcDelivery - IDL codes for comparing test to benchmark data • /testbed - program units for most MiRS processing /lib_cpp /lib_java

  26. MiRS OverviewDirectory Structure /src MIRS /setup /scripts /logs /src /gui /doc /data /bin /1dvar /fwd /crtm /lib /lib_idl /qcDelivery /testbed /lib_cpp /lib_java /biasGenerAndMonit /chopp /fm /grid /mergeEDR /mergeNEDTofDiffInstr /nedtMonitoring /nwp /rdr2tdr /retrHeritage /retrRegress /tdr2sdr /vipp

  27. MiRS OverviewDirectory Structure /data MIRS /setup /scripts /logs /src /gui /doc /data /bin /BenchmarkData /ControlData /ExternalData /StaticData /TestbedData • /BenchmarkData - Benchmark output for testing MiRS • /ControlData - All namelists for Fortran/IDL • /ExternalData - Directories for sensor L1B and NWP data • /InputsData - List files referenced by namelists • /SemiStaticData- regression coefficient, bias correction, and modeling error files • /StaticData- Tuning Files, CRTM coeff. Files, Instrument config. Files, 1DVAR Tuning Files, LUTs • /TestbedData - Stores all MiRS intermediate and output data including EDR and DEP product files, gridded data, and figures in /Outputs and processed sensor data (SDRs, FMSDRs, collcated NWP data) in /DynamicData. /InputsData /SemiStaticData

  28. MiRS OverviewDirectory Structure /data MIRS /setup /scripts /logs /src /gui /doc /data /bin /BenchmarkData /ControlData /ExternalData /StaticData /TestbedData /InputsData /SemiStaticData /DynamicData /nedt /grid /figs /Outputs /edr /PerfsMonitoring /dep /edr_heritage

  29. MiRSExtending to Other Sensors • This section seeks to identify the components need to extend MiRS to new sensors. • The MiRS code was developed in a modular fashion to facilitate the extension of MiRS capability to new sensors. • Some MiRS components (Static, SemiStatic, and codes) are sensor dependent. • Primary dependency on extension is whether the sensor is supported by CRTM.

  30. Extension to SensorCode Components • src/testbed/rdr2tdr/rdr2tdr_sensor.f90 – Level 1B reader • src/testbed/fm/fm_sensor.f90 – footprint matching code • src/lib/misc/Consts.f90 – addition of sensor ID constant • Some codes may need conditionals built in that reference the sensor ID for specific action • src/testbed/biasGenerAndMonit/Calib_generic_rad.pro • src/testbed/vipp/vipp.f90

  31. Extension to SensorSemiStatic Components • data/SemiStaticData/biasCorrec/biasCorrec_sensor.dat– radiometric bias correction coefficients • data/SemiStaticData/biasCorrec/ModelErrFile_sensor.dat– RTM uncertainty file • data/SemiStaticData/regressAlgors/Sfc_regressCoeffs_sensor.dat– regression coefficients for first-guess retrieval (optional).

  32. Extension to SensorStatic Components • data/StaticData/CovBkgStats/CovBkgMatrxTotSfc_all_sensor.dat– surface emissivity covariance matrix • data/StaticData/CRTMFiles/sensor_SpcCoeff.bin– CRTM spectral coefficient file • data/StaticData/CRTMFiles/sensor_TauCoeff.bin– CRTM absorption coefficient file • data/StaticData/InstrConfigInfo/InstrConfig_sensor.dat– Instrument frequency/polarization definitions • data/StaticData/TuningData/TunParams_sensor.dat– Tuning parameters for 1dvar retrieval (1st attempt) • data/StaticData/TuningData/TunParams_sensor_2.dat– Tuning parameters for 1dvar retrieval (2nd attempt) • data/StaticData/EmissCatalog/SeaIceEmissCatalog_sensor.dat – sea ice concentration look up table • data/StaticData/EmissCatalog/SnowEmissCatalog_sensor.dat– snow properties look up table

  33. Summary • MiRS is an operational algorithm at NOAA/NESDIS which provides operations sounding products • Near term improvements to sounding in heavy precipitating conditions • MiRS algorithm being extended to NPP ATMS, TRMM TMI, GPM GMI, M-T in the short term • Website http://mirs.nedsis.noaa.gov

  34. MIRS /setup /scripts /logs /src /gui /doc /data /bin /1dvar /fwd /crtm /lib /lib_idl /qcDelivery /testbed /lib_cpp /lib_java

More Related