Observations Preprocessing Carla Cardinali

1. Observations PreprocessingCarla Cardinali

2. Observations preprocessing MakeCMA BUFR ECMA ODB Screening CCMA Minimization Forecast MatchUp ECMA FeedBack BUFR

3. ECMA/ODB CCMA/ODB Output BUFRs Observations preprocessing

4. Observations preprocessing What is ODB? Developed software at ECMWF to manage large amounts of satellite data. ODB/SQL language is a subset of ANSI/SQL query language. Compiler translates ODB/SQL in C-codes files How do use it? Access to ODB database is through Fortran90 modules functions. A ECMA data base contains table with column entries lat, lon, date, time observation type whichis used to locate observation at a given moment Obs. Ident ( sat. id, lat, lon, st.alt., date, time) Obs. V. (wind, temp.,…per pressure) and (radiances per ch., inst. type) Various flag: active, blacklisted…. Departures (obs-background, obs-analysis)

5. Observations preprocessing Incoming ObservationThe observations arrive at ECMWF through GTS and they are stored in a decoded format in Report Data Base after some rudimentary quality control e.g. observation format and position, climatological and hydrostatic limit and temporal consistency. An observation file suitable for assimilation is created 6-hour Preprocessing Observation ArrayFormat conversions, change of some observed variables (relative humidity from dry and wet bulb temperature) and assignment of observation error statistics

6. Observations Screening Select the “best” observationAt the first trajectory run the model counterparts for all the observations are calculated through the nonlinear observation operators. For the observation screening, the background errors are interpolated to the locations of the observed variables. The "extended" observation array (ECMA) contains observations complemented by the background departures and quality control information for most of the observations. This array is stored for later feedback. After the screening a "compressed" array is passed to the minimization (CCMA).

7. Independent Observations Screening The screening logic is to make first those decisions that are not depending on any other ones Preliminary checks: completeness of the report Blacklist Data Selection: which observation types, variables, vertical ranges will be used in the assimilation. Monthly selection: discarding stations that have been reporting excessively noisy or biased compare to the background field. Impact studies.

8. Independent Screening Decision: Conventional Observations Background Quality ControlThe BgQC is applied to all the variables that are intended to be used in the assimilation BgQC of wind observations is done simultaneously for both wind components. For wind direction the error limits of 60, 90 and 120 apply for flags 1,2 and 3 uo ub

9. Independent Screening Decision: Conventional Observations

10. When is not working

11. Independent Screening Decision: Satellite Observation Bias CorrectionBias correction coefficients are recomputed from the past 2/4 weeks of departure statistics. The feedback files are used for monitoring the performance of the observing and assimilation system: e.g. ATOVS and SSMI radiances, SCATT wind. If the removed bias is a model forecast bias, the subsequent assimilation will enforce it. Usually, only half bias is removed. Gross checkMeasured and background brightness temperatures are present for all required channels. Bias correction coefficients, satellite id, and scan position are all valid before proceeding. For all channels a cloud detection is performed 230K 290K

12. SSMI HIRS 3 2 1 ocean: LWP=f(ch3/4) If LWP >  cloud+rain land +ocean Channels below rejected Dependent Screening Decision: Satellite Observation

13. AMSU-A Tb 6 no cloud 5 cloud 4 rain 3 1 3  2 1 ocean: LWP=f(ch1,ch2,ch3) If LWP >  cloud+rain land: if (ch1-ch3) >  rain  because of large emissivity over land no cloud contamination detection Dependent Screening Decision: Satellite Observation

14. Independent Screening Decision: Satellite Observation Background Quality control: ATOVS, SSMI, METEOSATBackground temperature, specific humidity and ozone profiles are checked to make sure they are close to or within the range for which the radiative transfer model is valid. Temperature is within the range 150-350 K, specific humidity is positive and not supersaturated and the ozone is within climate extremes. Radiance at the 2 extreme edge positions of the swath are not used in 4D-Var

15. Independent Screening Decision: Satellite Observation

16. Independent Screening Decision: Satellite Observation

17. Dependent Screening Decision: Conventional Observation Before performing the dependent screening decisions, the flag information gathered so far is converted into a report status, namely active, passive, rejected or blacklisted i.e the RDB datum flag. Vertical consistency of multilevel reportsDuplicated levels are removed. If some consecutive layers are of suspicious quality they are rejected and for geopotential obs. also the layer above Removal of duplicated report Search globally for duplicated (AIREP) data. Usually, there are aircraft reports that have the same date/time, roughly the same location and, at least partially, the same data. Usually, the station ID is slightly different.

18. Dependent Screening Decision: Conventional Observation Redundancy check Removes redundant SYNOP/PAOB : co-located reports with the same station ID are searched for and only the closest active report to the analysis time is retained. For simultaneous reports, the one with more active data is retained. The same is done for reports that have equal time difference to the analysis time. Co-located SYNOP (mass obs) is redundant to TEMP (geop.) that is within 50 hPa Removes redundant SHIP/DRIBU: moving platforms within a circle of 1 ° are considered as potentially redundant: reports closest to the centre of the screening time with most active date are retained.

19. Dependent Screening Decision: SYNOP 4D-Var Screening for 4D-Var The effect of the observation screening on SYNOP surface pressure observations. Column height gives the number of observations available, while the shaded part displays those actually used in the assimilation. 3D-Var Screening for 3D/4D-Var

20. Dependent Screening Decision: Conventional Observation Level selection for TEMP/PILOT redundancies: for one time window rejection in layers around std-levels according the following priorities maximum datum flag time difference to analysis time distance to the std-level significant level (turning points of the sounding) temp over pilot

21. Dependent Screening Decision: Thinning AIREP thinning horizontal thinning from the same platform, a minimum distance between the nearby reports is enforced: Box 125 62 km vertical thinning is performed around the model vertical levels: one aircraft measurement per model level.

22. Dependent Screening Decision: Thinning + VERT increased number of AIREP from 15 to 60 vertical levels +HOR also increased number of AIREP in horizontal

23. GTS Obs thin Screening Decision: Thinning TEMP

24. Dependent Screening Decision: Thinning ATOVS thinning: a repeated scan is performed to get the observation resolution of 70 km (it depends on sensors/channels) a sea sounding is preferred to a land one clear sounding to a cloudy one closeness of observation time to the centre of the screening time window A second thinning takes place that selects one observation every 140km

25. Dependent Screening Decision: Thinning The usage of ATOVS reports in the assimilation on the North Eastern Atlantic. Filled rings mark reports contain one or more channels used in the assimilation, whereas the empty rings denote rejected reports. Most of the rejections are due to the horizontal thinning and much less due to the quality reasons.Note that both edges of the swath are rejected.

26. Dependent Screening Decision: Thinning SCAT:ERS-2 The process is defined with respect to the particular measurement geometry of the instrument. The backscatter data are acquired within individual cells related to a 450 km wide grid with a mesh of 25 km in the across and along track directions. 19 measurement nodes are thus defined across the scatterometer´s swath, numbered from 1 to 19 as the incidence angle increases, while 19 rows are also considered in the along track direction to gather the data in squares of 19 by 19 points. The thinning is then achieved by keeping only every fourth point within these squares. The data are thus used at a resolution of 100 km instead of the original 25 km sampling distance. Sea-ice contamination < 273K High wind rejection test > 25m/s Normalized distance to the cone or wind residual. During the assimilation this quantity is computed and data rejected if a large value is found.

27. 1-step ..        .. ..     .. ..     ..        .. …….... 2-step check the distance between observations Triplet obs .. 3.. .. .. 7 Dependent Screening Decision: Thinning

28. Quadruplet obs Courtesy of Hans Hersbach Lars Isaksen Mark Leidner QuikSCAT 50 km

29. Dependent Screening Decision: Thinning High resolution trajectory to choose only one

30. Dependent Screening Decision: Thinning Rej: only 1 beam measur. active Minima are not well defined Var-QC White areas: insuf. info for MLE inversion e.g rain contamination

31. Blacklisted Obs. Type Reports Rejected 0 0 100699 0 0 7 154867 76 0 3933 22899 176518 3094 3 75 193416 76 0 28151 52868 217018 4802 605 805 249279 214 0 Synop Airep Satob Dribu Temp Pilot Satem Paob Scat Screening Statistics Active Passive 24218 29969 40500 1708 602 730 55863 138 0

32. After Screening Compression of CMA-file After screening only 15% of all observations are active 10-20% TOVS left 40% Conventional left The observation are resorted among the processors for a more optimal load balancing of the parallel computer

33. After Screening Parallel computing environment The observation screening should result in exactly the same selection of observations when different number of processors are used. Independent decision can be made at different processor fully in parallel. But global view of observation array is needed when a dependent decision has to be taken which implies that some communication between the processors is required. The observation array is too large to be copied in each individual processor then only a minimum necessary information is globally communicated 1 Global Array Location ID 2 Time