Observation Screening

1. Observation Screening Kertész Sándor 3D-VAR/ODB Training Budapest, 7 June, 2006

2. OUTLINE • ODB primer • Observation data flow • Screening QC and decisions in ODB • Screening inputs • How is screening working? • Output listings • Exercises

3. ODB primer • ODB is Observational Database • Basic data structure is called table. Tables is made up of data columns • Set of tables (with relations) forms a database type. E.g. ECMA that contains all the observations to be used in a DA system • Information on tables, columns etc. • http://www.cnrm.meteo.fr/gmapdoc/meshtml/DOC_odb/odb.html • In „Doc” directory on regatta: Assimilation_Chapter9_v2.3.pdf • ODB training!

4. ODB primer • An ODB database for a given analyis date is a set of files organized into a directory structure (check your BATOR output!) • ODB data is divided into pools for parallel runs (to ensure load balance) • To look into ODB you need to run the viewer with your view! • The view define the retrieval!

5. ODB primer • View definition (e.g. to retrieve AIREP data) • CREATE VIEW myview AS • SELECT • obstype, lat, lon, varno, obsvalue, status.blacklist@body • FROM hdr, body • WHERE obstype == 2 View name Retrieved columns Involved tables Selection criteria

6. ODB primer • The concept of sub-bases was invented to have the choice for handling observation types separately. E.g.: ECMA.conv, ECMA.amsua, ECMA.amsub etc… • These ODB sub-bases must be merged into a new ODB • Many applications are working on this kind of ODB that is called virtual basis

7. Observation Screening • The observation screening is the last step in the pre-processing chain to produce observation inputs for the variational analysis • Screening performs various quality control checks on data • ALADIN screening is very similar to the global (ARPEGE) one • Screening uses an ECMA ODB as input • The production of this ODB is rather complicated, we have to step back to BATOR to see the complete data flow …

8. Obs dataflow: BATOR • BATOR is producing separate ECMA ODB sub-bases containing the required number of ODB pools • We need: #pools = # CPUs for further applications! • These ODB sub-bases must be merged into a new ECMA ODB using programmeSHUFFLE • Further applications are working on this ODB virtual basis

9. Obs dataflow: LAMFLAG • The resulting ODB is not suitable forALADIN screening since it is not working with observations outside the C+I zoneAbort in GEFGER! • Solution: the input ODB must be restricted to the C+I zoneprogrammeLAMFLAG E LAMFLAG writes status flags into ODB C+I zone@hdr=0 zone@hdr=1

10. Obs dataflow: LAMFLAG namelists Observation filter If .TRUE.: selection is based only on observation filter! &NAMFCNT LOBSONLY=.FALSE. /END &NAMFGEOM ELAT0=46.2447006399999978, ELON0=17.0000000000000000, ELATC=46.2447005948822678, ELONC=16.9999999982550491, EDELX=12176.1563281414165, EDELY=12176.1563281414165, NDLUN=1, NDGUN=1, NDLUX=229, NDGUX=205, Z_CANZONE=1500., LVAR=.T., LNEWGEOM=.F., /END &NAMFOBS LSYNOP=.T., LAIREP=.T., LSATOB=.T., LDRIBU=.F., LTEMP=.T., LPILOT=.T., LSATEM=.T. LPAOB=.F., LSCATT=.F., /END Model geometry settings Used for screening

11. Obs dataflow: SHUFFLE reduction • LAMFLAG puts flags into ODB but does not reduce it! • SHUFFLE must run with setting TO_ODB_REDUC=1 for each sub-bases in the ODB separately to produce a reduced ODB • Then another SHUFFLE run is needed to put sub-bases together again (merging) • This results in a virtual base ECMA ODB again that is now ready for used as screening input

12. Obs dataflow: Screening • Screening puts its quality control flags and decisions into the input ECMA ODB • QC flags, decisions for reports (in table hdr): • statusstatus_t • event1report_event1_t • blacklistreport_blacklist_t report_event1_t     no_data                  bit1     all_rejected             bit1     bad_practice           bit1     rdb_rejected            bit1     rdb_activated          bit1     whitelist_activated   bit1     horipos_outrange    bit1     vertpos_outrange    bit1     time_outrange         bit1     redundant                bit1     land                     bit1     sea                      bit1….. report_blacklist_t     obstype      bit1     statid       bit1     codetype     bit1     instype      bit1     date         bit1     time         bit1     lat          bit1     lon          bit1     stalt        bit1     scanpos      bit1     retrtype     bit1 ….. status_t activebit1 passivebit1rejectedbit1blacklistedbit1 …. Passive and blacklisted are a subset of rejected!

13. Obs dataflow: Screening • QC flags, decisions and additional infromation for observed values (in table body): • status status_t • event1 report_event1_t • blacklist datum_blacklist_t • fg_depar = obs-guess datum_event1_t vertco_missing        bit1      obsvalue_missing    bit1      fg_missing              bit1      rdb_rejected            bit1      rdb_activated           bit1      whitelist_activated    bit1      bad_practice            bit1      vertpos_outrange     bit1      reflevel_outrange      bit1      fg2big                  bit1      depar2big              bit1      obs_error2big           bit1      datum_redundant      bit1      level_redundant         bit1 …. datum_blacklist_t     obstype      bit1     statid       bit1     codetype     bit1     instype      bit1     date         bit1     time         bit1     lat          bit1     lon          bit1 …. status_t activebit1 passivebit1rejectedbit1blacklistedbit1 ….

14. Obs dataflow: Compression • After Screening another SHUFFLE is running, this time to perform an ECMA CCMA conversion • A CCMA ODB consists of only those tables and column that are needed for the analysis • The resulting CCMA ODB after SHUFFLE contains only the active reports and observations! • This CCMA ODB is the input of the minimization (e131) • The observation data flow is finished!

15. ECMA (Full) LAMFLAG BATOR SHUFFLE (Reduc) ECMA.sb1 ECMA.sb2 … ECMA.sb1 ECMA.sb2 … SHUFFLE (Merge) SHUFFLE (Merge) ECMA (C+I) Screening SHUFFLE (ECMA->CCMA) CCMA Obs dataflow summary

16. How to run screening? • It is configuration 002: • MASTER –c002 –maladin –vmeteo –t001 –aeul –eMIN1 • Inputs: • ECMA ODB (observation, sigma-o, blacklisting) • First guess file under 5 different names: • ICMSHMIN1INIT,ICMSHMIN1IMIN,ICMRFMIN10000,ELSCFMIN1ALBC000,ELSCFMIN1ALBC • Constants, statistics • Namelists

17. Constants • errgrib: GRIB file containing grid-point space sigma-B values (lat-lon grid, several levels recieved from MF) • rt_coef_atovs_neepred_ieee.dat: coefficients for the RTM (received from MF), binary file • rszcoef_fmt.dat: the same the previous file but in ASCII format • bcor_noaa.dat: bias correction file • chanspec_noaa: obsolete? • rmtberr_noaa.dat: obsolete? • cstlim_noaa.dat: obsolete?

18. Screening specific namelists • These parameters should be set like this for 3D-VAR: • NUPTRA = -1 , number of updates of the trajectory (NAMVAR) • NOBSHOR = 201, bilinear horizontal interpolation in H (NAMVAR) • LOBS = .TRUE., real obs (NAMCT0) • LOBSC1 = .FALSE., (NAMCT0), traj. is not computed • LSIMOB = .FALSE., no simelated obs (NAMCT0) • LSCREEN = .TRUE., (NAMCT0), screening is on! • LSCRE4D = .FALSE. , 3D screening (NAMSCC) • Tunable parameters are in NAMSCC and NAMOBS (discussed later)!

19. How is screening working? Decisions are taken in subroutine: Independent decisions: performed separately for each report, RDB flag is assigned to reports DECIS rdbflag@hdr rdbflag@body Update of flags (FLGTST). Flags are converted into status: active, passive, blacklisted, rejected Dependent decisions: dependency on other reports or obs, or the previous decisions

20. Independent decisions • Preliminary checks (PRECH): Completeness of report missing station altitude for SYNOP, TEMP and PILOT… • Blacklisting (BLACKCLN, BALCKSAT) • Background quality control (FIRST)

21. Blacklisting • Blacklisting information is placed into ODB by BATOR • In the screening a separate selection is applied for satellite data (BLACKSAT) • Blacklisting is also performed in BLACKCLN: • Blacklisted obs will be rejected • Various blacklisting for SATOB (e.g. QI < 85) • Land-sea rejection (SYNOP, TEMP, PILOT). Controlled via NAMOBS LSLREJ (default is .T.) • LSLRW10: Wind 10m, LSLRT2: T 2m, LSLRRH2: THU 2m (default is .T. meaning rejection over land. Will get a passive status!)

22. Blacklisting • Also in BLACKCLN: • Height-based rejection. Controlled via NAMOBS LHDLREJ (default is .T., rejected obs will get passive status!) • Orographic rejection limit is applied: • LHDRW10: Wind 10m, LHDRT2: T 2m, LHDRRH2: RHU 2m (default is .T.) • AIREP data and significant levels for TEMP and PILOT are not used below the model surface! • RHU and Q is rejected above 300 hPa • AIREP is not used below a certain pressure

23. Background QC (FIRST) • From the BLUE theory we know: • For a given location we expect: Values are read from file errgrib and interpolated to obs locations Values are read from ODB

24. Background QC (FIRST) • Flags are assigned to observations by checking: Based on predefined flaglimits 4 flag values can be assigned to obs 0: correct 1: probably correct 2: probably incorrect 3: incorrect Flaglimits are defined in DEFRUN

25. Dependent decisions • Remove redundant surface levels from TEMP/PILOT (REDSL) • Vertical consistency of profiles (VERCO) • Removal of duplicated reports (DUPLI): two reports with the same ID, time an location … • Redundancy check (REDUN) • Thinning for AIREP (THIAIR) • Specific scatterometer quality control • Thinning of satellite data (THINN)

26. Redundancy check (REDUN) • Applied for all active reports which are co-located and originate from the same station. Separate treatment for: • Land SYNOP: more reports for the same place but with different time, the one with closest to the analysis date with the most active obs are selected • Ship SYNOP, DRIBU: redundant if moving platforms are within a circle of 1° radius • TEMP and PILOT • SYNOP mass observation is redundant if TEMP obs is available and the diff < 50 hPa

27. Thinning of AIREP data (THIAIR) • One flight consist of a set of reports • Thinning is performed for each flight separately • 3D boxes are constructed around model levels • In each box the report closest to the analysis date with the most active observations are selected • Box size is set in metres via RFIND_AIREP in NAMSCC, the default is ~70 km! • By default box size less than 50 km cannot be used in ARPEGE, code modification is needed

28. Thinning of satellite data (THINNER) • For radiances a horizontal thinning is performed in two steps: • First a minimum distance is enforced (default ~ 70 km, can be changed in NAMSCC via RMIND_RAD1C(sensor)) • Then a repeated thinning is performed to reach the final separation (default ~ 140 km, can be changed in NAMSCC via RFIND_RAD1C(sensor)) • Where sensor is: 0=HIRS, 1=MSU, 2=SSU, 3=AMSU-A, 4=AMSU-B, 6=SSM/I, 20=METEOSAT, 21=MSG_HR, … • Selection criterion: • Sea is preferred over land • Clear sky pixel is preferred over a cloudy one • Obs closer to the analysis date is preferred

29. Thinning of satellite data (THINNER) • For SATOB a the thinning is performed: • Horizontally in two steps similarly to radiances: controlled in NAMSCC via RMIND_SATOB, RFIND_SATOB (default is ~70 and ~140 km) • Vertically around model levels • In each box the report closest to the analysis date with the most active observations are selected • QI values are also used in the selection

30. Sreening output • Valuable summary about screening decisions can be found in the NODE file: • Try „grep SCREENING STATISTICS” to get: • STATUS summary • EVENT summary • Number of variables, departures and missing departures • Diagnostic JO-table

31. Sreening output • Summary at „grep SCREENING STATISTICS” • Report and data status summary OB.TYP REPORTS ACTIVE PASSIVE REJECTED BLACKLISTED 1 725 44 0 681 0 2 0 0 0 0 0 3 0 0 0 0 0 4 0 0 0 0 0 5 61 60 0 1 0 6 61 59 0 2 0 7 0 0 0 0 0 8 0 0 0 0 0 9 0 0 0 0 0 10 0 0 0 0 0 ---------------------------------------------------------------------------------------- TOT 847 163 0 684 0 OB.TYP DATA ACTIVE PASSIVE REJECTED BLACKLISTED 1 4576 164 1846 4412 3862 2 0 0 0 0 0 3 0 0 0 0 0 4 0 0 0 0 0 5 9186 7271 1218 1915 705 6 3438 2050 64 1388 72 7 0 0 0 0 0 8 0 0 0 0 0 9 0 0 0 0 0 10 0 0 0 0 0 ------------------------------------------------------------------------- TOT 17200 9485 3128 7715 4639

32. Sreening output • Summary at „grepSCREENING STATISTICS” • Diagnostic Jo table Obstype 1 === SYNOP, Land stations and ships -------------------------------------------------- Codetype 11 === SYNOP Land Manual Report Variable DataCount Jo_Costfunction JO/n ObsErr BgErr U10 1380 2250.937197524 1.63 0.200E+01 0.000E+00 H2 721 834.1196058351 1.16 0.100E+00 0.000E+00 Z 567 892.5077244762 1.57 0.785E+02 0.000E+00 T2 721 2138.950482967 2.97 0.140E+01 0.000E+00 Q 721 1577.382611689 2.19 0.998E-03 0.000E+00---------- --------------------------- -------- ObsType 1 Total: 4110 7693.897622492 1.87

33. Sreening output • Observation monitoring available at: • http://pc2088.met.hu/monitor/start.php

34. Thank you for your attention! Any question?

35. Exercises • Run screening for all the prepared observations with script ~/workdir/3dvar/Assim and study the output listing! (You have to put an exit after screening in the script).

36. Exercises • Examine in detail the SYNOP and TEMP report for Budapest, see if surface obs (except Z) are really rejected, and find the reason for it. Try this view first: • DEFINE VIEW bp AS • SELECT • obstype, status.active@hdr,varno, • obsvalue, fg_depar, press, status.active@body, • status.passive@body, status.blacklisted@body, status.rejected@body, • event1.land@body, event1.datum_redundant@body • FROM hdr, body • WHERE (obstype == 1 OR obstype == 5) && statid = ‘12843 ‘ • Guidance: A prepared viewer par file and sql file can be found in ~wshop01/workdir/OdbViewer as bp.par bp.sql. Copy them to your directory, set the sql path fo your file and type „viewer –p bp.par” to run viewer. • If you are brave enough try to write the same view for AIREP (obstype == 2) and explore bit fields of event@body!

37. Exercises • Re-run screenig with a 10 km thinning distance for AIREP. To do this you have to use the exe you created yesterday (by modifying arp/obs_preporc/thiair.F90to reduce ISCALE to 1000!) Modify PROC_LAM_ODB and d_RUN in include.inc! Compare the default and the new airep thinning in the output listings or in the ODB!