Quality control

1. Quality control John Derber NCEP/EMC Reference slide at end

2. Quality Control - General • Quality control in data assimilation is used to ensure that the analysis is not degraded by the inclusion of certain observations • Good quality control also often requires knowledge of the details the instrument design and processing system • Quality control can be applied by removing observations from analysis or by adjusting observation errors for observations • An infinite observation error in the same as removing an observation • Monitoring is enhanced if observation is used in analysis, but weight is zero. • Quality control requires monitoring of data and has significant impacts on the design of the database. • Quality control is extremely important, but not recognized.

3. Observation errors • For random observation errors should be Gaussian. • Reasons for non-random errors • Representativeness error • Inability of forward model to properly model observation • Instrument Difficulties

4. Observation errors • For random observation errors should be Gaussian.

6. Representativeness error • Models do not resolve all scales of the atmosphere. • Observations often are point measurements • When structures are observed which cannot be resolved – representativeness error. • Observation is correct – but does not represent the structures in the analysis • Example – Radiosonde launched in local thunderstorm used for large scale analysis

7. Inability of forward model to properly model observation • If forward model transforming analysis variables into observation variables cannot properly simulate observation • Example – cloudy radiances from satellite when only simulating clear radiances • Example 2 – not including CO2 in retrieval process – errors created at scales of CO2 variation • Observations correct, but errors introduced which can be aliased into analysis

8. Satellite data • Most simulation problems with satellite data come from 5 sources: • Instrument problems. • Inadequate characterization • Degrading instrument • Inadequately measured/modelled components of forward model. • Inadequately understood physics

9. Satellite radiances • IR cannot see through clouds. • Cloud height can be difficult to determine – especially with mixed FOVs. • Since measures deep layers, not many channels completely above clouds. • Microwave impacted by clouds and precipitation but signal is smaller from thinner clouds and can be modeled. • Surface emissivity and temperature characteristics not well known for land/snow/ice. • Also makes detection of clouds/precip. more difficult over these surfaces. • Error distribution may be asymmetric due to clouds and processing errors.

10. Observation Weight after QC

11. Observation Weight after QC

12. Observation Weight after QC

13. Observation Weight after QC

14. Instrument Difficulties • If the instrument is fails – observations usually not good (by definition) • Instrument design • ascending/descending aircraft • Non-Gaussian errors – Scatterometer ambiguity • Insufficient sensitivity – GPS RO • Monitoring vital to detect problems • Surface data examples

15. Mismatch between top planetary boundary layer Model BA exceeds limit for observation Below, obs rejected by QC

16. Instrument Difficulties • If the instrument is fails – observations usually not good (by definition) • Instrument design • ascending/descending aircraft • Non-Gaussian errors – Scatterometer ambiguity • Insufficient sensitivity – GPS RO • Monitoring vital to detect problems • Surface data examples

17. Surface pressure data examples

19. Quality control usually has multiple steps • Choice of observations used in analysis • Physically realistic? • Pressure > 0, Moisture > 0, Surface observation < 500mb, etc. • Complex quality control – attempting to save observations • Station monitoring – Blacklist • Must have capability to add/remove stations from list • Manual QC – Human intervention • Greatly complicates operational system • Instrument specific checks based on instrument design • Variational QC (replaces buddy check)

20. Variational quality control • Modifies weight given observation based on deviation from guess/analysis • Assumes that observation error is not Gaussian. • Examples – Assume error PDF is Gaussian + constant • Examples – Assume error PDF is sum of Gaussians • Will make minimization non-linear – will generally slow convergence and may create multiple minima. • Generally makes weight a function of how close the data is to current solution – solution should be close to final solution

21. Correlated errors • If observational errors are correlated – all parts of quality control must be rethought • Nearby obs may have same signal because of real signal or correlated errors. • Can be extremely difficult to distinguish correlated error • Correlated errors occur because of • Instrument/forward model biases • Retrievals from multiple observations spreads errors across all components of retrieval

22. Data Monitoring • It is essential to have good data monitoring. • Usually the NWP centres see problems with instruments prior to notification by provider (Met Office especially). • The data monitoring can also show problems with assimilation systems. • Needs to be ongoing/real time. • https://groups.ssec.wisc.edu/groups/itwg/nwp/monitoring

23. AIRS Channel 453 26 March 2007 Quality Monitoring of Satellite Data Increase in SD Fits to Guess

24. Summary • Quality control is a very important (but under-recognized) component of data assimilation • Often performed in multiple steps – platform dependent and cross-platform components • Inclusion of quality control within analysis can be done (variational QC) and is being done at operational centres. • However, variational QC significantly complicates minimization • Much QC work not documented in literature

25. References Andersson, E., and H. Jarvinen, 1999: Variational quality control Quart. J. Reoy. Meteor. Soc., 125, 697-722 – See also ECMWF tech memo #250. Baker, Nancy L., 1994: Quality control of meteorological observations at Fleet Numerical Meteorology and Oceanography Center, U.S. Navy, Naval Research Laboratory,23p. Collins, W. G., 2001: The operational complex quality control of radiosonde heights and temperatures at the National Centers for Environmental Prediction. Part I: Description of the Method, Jour. of App. Meteor.,40.2, 137-151. European Centre for Medium Range Weather Forecasts, WMO/ECMWF Workshop on Data Quality Control Procedures, Reading, England, March 6-10, 1989, Papers., Reading, England, European Centre for Medium-Range Weather Forecasts (ECMWF), 1989 Gandin, Lev S., 1988: Complex quality control of meteorological observations, Mon.Wea.Rev.116. 1137-1156. Ingleby, Nbruce, Lorenc, Andrew C., 1993, Bayesian quality control using multivariate normal distributions, Quart. J. Roy. Met. Soc, 119, 1195-1225. Kelly, G; Andersson, E; Hollingsworth, A; Loennberg, P; Pailleux, J; et al.; 1991: Quality control of operational physical retrievals of satellite sounding data, Mon. Wea. Rev.,1866-1880. Lorenc, A.C., and Hammon, O., 1988: Objective quality control of observations using Bayesian methods – Theory, and a practical implementation Quart. J. Roy. Met. Soc. 114, 515-543. Purser, R.J., 2011: Mathematical principles of the construction and characterization of a parameterized family of Gaussian mixture distributions suitable to serve as models for the probability distributions of measurement errors in nonlinear quality control, NCEP Office Note #468. Various ECMWF lectures and training courses.