1 / 18

VERA-QC, a new Data Quality Control based on Self-Consistency

VERA-QC, a new Data Quality Control based on Self-Consistency. Dieter Mayer , Reinhold Steinacker, Andrea Steiner University of Vienna, Department of Meteorology and Geophysics, Vienna, Austria. Presentation at the 10th European Conference on Applications of Meteorology (ECAM )

tamah
Download Presentation

VERA-QC, a new Data Quality Control based on Self-Consistency

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. VERA-QC, a new Data Quality Control based on Self-Consistency Dieter Mayer, Reinhold Steinacker, Andrea Steiner University of Vienna, Department of Meteorology and Geophysics, Vienna, Austria Presentation at the 10th European Conference on Applications of Meteorology (ECAM) Berlin, 14 September 2011

  2. Outline • Motivation for VERA-QC • Applicability and basis of VERA-QC • Mathematical background of VERA-QC • Deviations and error detection • Handling special station alignments • Conclusion and availability of VERA-QC 10th European Conference on Applications of Meteorology (ECAM) Berlin, 12-16 September 2011 IMG Vienna Mayer et.al.

  3. Motivation for VERA-QC High quality data is needed as input for VERA • What is VERA? • Analysing observations to grid points (complex topography) • Combining interpolation (TPS) & downscaling (Fingerprints) • Features of VERA • Modelindependent • No need for first guess fields • Works on real time & operational basis • Applications of VERA & VERA-QC • Real time model verification • Basis for nowcasting • Evaluation of case & field studies • Computation of analysis ensembles High quality data is needed as input for VERA • What is VERA? • Analysing observations to grid points (complex topography) • Combining interpolation (TPS) & downscaling (Fingerprints) • Features of VERA • Model independent • No need for first guess fields • Works on real time & operational basis • Applications of VERA & VERA-QC • Real time model verification • Basis for nowcasting • Evaluation of case & field studies • Computation of analysis ensembles 10th European Conference on Applications of Meteorology (ECAM) Berlin, 12-16 September 2011 IMG Vienna Mayer et.al.

  4. Selecting or designing a QC? Requirements to select / design QC Existing QC- methods Properties of VERA & its applications no back- ground fields Bayesian QC model independent Variational QC fast (not iterative) complex topography QC using OI real time handle inhomogeneous station distribution QC using ID model verification QC using SR field studies propose deviations Internal consistency checks analysis ensembles no statistical information Limit checks Answer: there is a need for a new QC-method 10th European Conference on Applications of Meteorology (ECAM) Berlin, 12-16 September 2011 IMG Vienna Mayer et.al.

  5. Applicability of VERA-QC • Basis: spatio and / or temporal consistency of data • Requirement: High degree of redundancy in observations • Depending on station density & scale • of phenomenon • Expressed as station distance and decorrelation length • QC applicable if / >> 1 (GTS:pMSL,Q,Qe) Example: VERA-Analysis for precipitation (green) & MSL-pressure (black) Dots and stars: Observations for precip. & pressure 10th European Conference on Applications of Meteorology (ECAM) Berlin, 12-16 September 2011 IMG Vienna Mayer et.al.

  6. Basis of VERA-QC • Error affected observations  (rough) observation field Yo • Corrected observations  (smoother) analysis field Ya=Yo+ DY • Main task is to receive deviations DY Example: South-West to North East pressure-gradient with some artificial errors: • Note: DY is not a simple difference between observation and interpolation 10th European Conference on Applications of Meteorology (ECAM) Berlin, 12-16 September 2011 IMG Vienna Mayer et.al.

  7. Mathematical Coreof VERA-QC • Goal: receive deviations to obtain smooth analysis field. • Defining cost function J as squared curvature of analysis field: d1,d2, D: dimensions n, N: grid points • Curvature of analysis field Cya is not known  Taylor series expansion: P: prim. neighbors • - Building global cost function: (taking into account all stations and grid points) m,M: main stations s,S: second. neighbors • - Solving optimization problem for deviations : 10th European Conference on Applications of Meteorology (ECAM) Berlin, 12-16 September 2011 IMG Vienna Mayer et.al.

  8. Concept of natural neighbors • Questions regarding the cost function: • Q1: Where should the cost function be evaluated? A1: Regular grid is too expensive, take station points • Q2: What are main stations, primary and secondary neighbors? A2: m: Main station: one station after another s: (secondary) neighbors of m p: (primary) direct neighbors of m • Q3: ? Which stations contribute to the Taylor series expansion? A3: A certain station and its natural neighbors.  More than one station is allowed to be erroneous! • Method connecting stations: • Delaunay Triangulation 10th European Conference on Applications of Meteorology (ECAM) Berlin, 12-16 September 2011 IMG Vienna Mayer et.al.

  9. Triangulation / Computing curvatures • Computing curvatures Typical example for realistic station distribution and Delaunay Triangulation • Defining local grids around stations • Interpolate station values YS to grid points n: (Inverse distance interpolation) 10th European Conference on Applications of Meteorology (ECAM) Berlin, 12-16 September 2011 IMG Vienna Mayer et.al.

  10. Weighting Deviations • Simplest example: 1D, 1 spike • Outlier corrected partially, but counter swinging at neighbors • Solution: correcting erroneous observation should reduce cost function. Compute weighted deviations: • with 10th European Conference on Applications of Meteorology (ECAM) Berlin, 12-16 September 2011 IMG Vienna Mayer et.al.

  11. Deviations and Gross Errors • Three possibilities to handle an observation yes yes yes no No gross error Obs. accepted no Gross error Obs. rejected No gross error Obs. corrected no • a, b and c: parameter dependent, user defined thresholds • VERA-QC is repeated without rejected observations 10th European Conference on Applications of Meteorology (ECAM) Berlin, 12-16 September 2011 IMG Vienna Mayer et.al.

  12. Cluster Treatment • Error propagation possible at close by stations • Example: circles with stations, cluster in center • Both stations obtain significant deviations • Combine both stations to one fictive cluster station • Compute deviation for cluster station • Add deviation to both stations • Repeat VERA-QC for modified observations 10th European Conference on Applications of Meteorology (ECAM) Berlin, 12-16 September 2011 IMG Vienna Mayer et.al.

  13. Conclusions • Properties of VERA-QC: • Applicable to 1, 2, 3 and 4 dimensional problems • High efficiency in detecting errors compared to other QC methods • No simple averaging algorithm • Can handle very inhomogeneous station distributions • Modelindependent, fast, no iterations necessary • Deviations can be stored to compute bias • Implemented as Matlab stand alone application, runs on Server & PC • FurtherInformations: • Publication: Steinacker, R., D. Mayer, and A. Steiner 2011, Data Quality • Control Based on Self Consistensy. Accepted in Monthly Weather Review. • Poster Presentation: A. Steiner, Operational Application of VERA-QC, • Challenges and how to cope with them. Poster Hall, Thursday 16-17:00. 10th European Conference on Applications of Meteorology (ECAM) Berlin, 12-16 September 2011 IMG Vienna Mayer et.al.

  14. Availability of VERA-QC VERA-QC is freely available for non-commercial use Homepage: http://www.univie.ac.at/amk/veraflex/test/intern/ 10th European Conference on Applications of Meteorology (ECAM) Berlin, 12-16 September 2011 IMG Vienna Mayer et.al.

  15. The End Thank you for your attention Contact: dieter.mayer@univie.ac.at http://www.univie.ac.at/amk/veraflex/test/intern/ Acknowledgments: Austrian Science Fund (FWF), support under grant number P19658

  16. Is VERA-QC an averaging technique? • Considering a signal at only 3 stations (unlikely to be a gross error) • Unweighted deviations smooth signal • Weighted deviations only soften contrast 10th European Conference on Applications of Meteorology (ECAM) Berlin, 12-16 September 2011 IMG Vienna Mayer et.al.

  17. VERA-QC in higher dimensions 10th European Conference on Applications of Meteorology (ECAM) Berlin, 12-16 September 2011 IMG Vienna Mayer et.al.

  18. VERA in a nut shell Fingerprint Solution • Interpolate irregularly distributed station values to regular grid (Thin plate spline) • Downscaling with the help of idealized physically motivated patterns Unexplained field Explained field Weight 10th European Conference on Applications of Meteorology (ECAM) Berlin, 12-16 September 2011 IMG Vienna Mayer et.al.

More Related