1 / 43

Quality Control and Cross Validation in Real-Time Mesoscale Analysis

Learn about the application of cross validation for quality control in the Real-Time Mesoscale Analysis system. Explore topics like gross error checks, variational QC, and the use of reject lists. Understand how mesonet data is verified and incorporated into analysis.

mollylindsey
Download Presentation

Quality Control and Cross Validation in Real-Time Mesoscale Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. L Quality Control and the application of Cross Validation in the Real-Time Mesoscale Analysis(RTMA) system Sei-Young Park Sei-Young Park KMA/NWPD, NCEP/EMC Manuel Pondeca, Jim Purser, David Parrish, Geoff Dimego John Derber, Xiujuan Su,Wan-Shu Wu,Geoff Manikin NCEP/EMC sypark@kma.go.kr

  2. Contents • Introduction of RTMA • Quality control in RTMA • Gross error check • Variational QC • Use list vs. Reject list • Cross Validation • Hilbert curves • Summary and conclusion

  3. The RTMA is a fast-track, proof-of-concept effort intended to: leverage and enhance existing analysis capabilities in order to generate experimental CONUS-scale hourly NDFD-matching analyses establish a real-time process that delivers a sub-set of fields to allow preliminary comparisons to NDFD forecast grids also provide estimates of analysis uncertainty establish benchmark for future AOR (Analysis Of Record) efforts build constituency for subsequent AOR development activities Real-Time Mesoscale Analysis (RTMA)

  4. Procedure Temperature & dew point at 2 m & wind at 10 m RUC forecast/analysis (13 km) is downscaled by GSD to 5 km NDFD grid Downscaled RUC used as first-guess in NCEP’s 2DVar analysis of ALL surface observations Estimate of analysis error/uncertainty Precipitation – NCEP Stage II analysis Sky cover – NESDIS GOES sounder effective cloud amount Logistics Hourly within ~30 minutes 5 km NDFD grid in GRIB2 Operational at NCEP Q3 FY2006 Distribution of analyses and estimate of analysis error/uncertainty via AWIPS SBN as part of OB7.2 upgrade – end of CY2006 Archived at NCDC Real-Time Mesoscale Analysis (RTMA)

  5. Quality control in RTMA QC is very important when using the high density and unverified new data. This is one of the reasons why the Mesonets have not been used, despite their high data density. Therefore, applying QC with reasonable methods is the first step to using these data in the analysis system. 1. Gross error check : decided by the observation increment (residual) 2. Variational QC 3. Use list vs. Reject list of Mesonet Wind Analysis will be concentrated on the Mesonet wind. temperature: wind:

  6. 1. Gross Error Check Obs vs. Anal Obs vs. Guess Limit : (o-a)/R = 5 Limit : (o-a)/R = 10

  7. 2. Variational QC (Y-Hxb ) The distributions of departure often reveal a more frequent occurrence of large departures than expected from the corresponding Gaussian (normal) distribution with the same mean and standard deviation-showing as wide “Tails”. By Erik Andersson, 1999,2006

  8. Variational QC By Erik Andersson, 1999,2006

  9. Variational QC By Erik Andersson, 1999,2006

  10. Var QC weight function vs. IV( A :0.08 for Metar, Synoptic sea and land ) A : 0.1(288) A : 0.08(288) A: 0.06(288)

  11. Distribution of the innovation (VarQC) Obs vs. Aanl Obs vs. Guess Limit : (o-a)/R = 5 Limit : (o-a)/R = 10

  12. Mesonets comprise majority of obs but they are not as good as other conventional sfc obs sources 5/6 of all Mesonet data are from AWS which includes most school sites and APRSWXNE(citizen’s network) No mesonet winds used in current RUC (or NAM) due to slow wind bias. GSD has constructed a “Uselist” of acceptable networks based on overall siting strategies etc. : It depends on the Mesonet provider name. GSD Uselist was applied in the RTMA and has been running on the parallel system. Continuing need for scrutiny of mesonet quality 3. Uselist of Mesonet wind Provider name OK-Meso : Oklahoma Mesonet WT-Meso : West Texas Mesonet APG : U.S. Army Aberdeen Proving Grounds CODOT : Colorado Department of Transportation FLDOT : Florida Dep of Transportation INDOT : Indiana Dep of Transportation MNDOT : Minnesota Dep of Transportation DCNet : DCNet GoMOOS : Gulf of Maine Ocean Observing System GPSMET : ESRL/GSD Ground-Based GPS NOS-PORT : National Ocean Service Physical Oceanographic Real-Time System RAWS : Remote Automated Weather Stations MesoWestAGRIMET : U.S. Bureau of Reclamation MesoWestAQ : NOAA Air Resources Laboratory Special Operations and Resource Division MesoWestARL FRD : NOAA Air Resources Laboratory Field Research Division MesoWestARL SORD : NOAA Air Resources Laboratory Special Operations and Resource Division MesoWestDOERD : Department of Energy Office of Repository Development MesoWestDUGWAY : U.S. Army Dugway Proving Grounds MesoWestITD : Idaho Transportation Department MesoWestMT DOT : Montana Dep. of Transportation MesoWestTOOELE : U.S. Army Desert Chemical Depot, Tooele County

  13. Number distribution of wind data (U) For Var QC Var_pg=0.05, wgtlim=0.25, Gross=10 m/s With uselist Without uselist 250 4000 1000 4500

  14. Number distribution of wind data (V) For Var QC Var_pg=0.05, wgtlim=0.25, Gross=10 m/s With uselist Withoutuselist 250 4000 1000 4500

  15. Verification of the Uselist 2006.5.23.00.~2006.6.14.23. (23days, hourly)

  16. # Mesonet: 11,148 # Metar: 1761 # Mesonet: 1,617(14.5%) # Metar:1766 VarQC Uselist of Mesonet wind CASE 1 : 2006.3.14.15 UTC Withoutuselist With uselist All obs data All obs data

  17. VarQC Uselist of Mesonet wind CASE 1 :2006.3.14.15 UTC Withoutuselist With uselist

  18. # Mesonet: 11,775 # Metar: 1725 # Mesonet: 1,812 (15%) # Metar:1727 VarQC Uselist of Mesonet wind CASE 2 : 2006.11.25.12 UTC Withoutuselist With uselist All obs data All obs data

  19. VarQC Uselist of Mesonet wind CASE 2 : 2006.11.25.12 UTC Withoutuselist With uselist

  20. Rejest list : constructed by the rejected data in gross error check and VarQC - hourly made and updated - It depends on the station name. 4. Reject list of Mesonet wind station name lat lon 1 MLGC1 x 32.880 243.570 2 FHCC1 x 32.990 243.930 3 LTHC1 34.020 243.810 4 BPNC1 x 34.380 242.310 5 PIVC1 x 35.450 241.720 6 INTC1 x 36.120 242.910 7 QTWA3 x 36.580 246.270 8 TS037 x 36.620 241.790 9 QBRA3 x 36.790 246.240 10 BADU1 x 37.150 246.050 11 HP001 32.890 243.580 12 GDSN2 x 35.810 244.530 13 A36 x 36.540 244.460 14 AR221 x 34.190 243.290 15 AR745 34.500 242.680 16 C6728 34.840 240.920 17 H0099 x 34.380 242.400 18 PHELN 34.450 242.370 19 HSPRA 34.450 242.680 20 APPLE 34.510 242.820 …

  21. Distribution of rejected data 2006.6.8.~2006.6.20. (13 days) 0 ~ 25% (81.4%) 25 ~ 50% (16.4%) 50 ~ 75% (1.8%) 75 ~ 100% (0.6%)

  22. Distribution of rejected data 2006.11.21.12. 2006.11.21.19.

  23. Verification of the reject list

  24. Verification of the reject list 2006.11.10.~2006.11.17. (8 days)

  25. Verification of the reject list 2006.11.10.~2006.11.17. (8 days)

  26. NWP data assimilation gauges the quality of initial conditions via model forecast skill. Cross-validation is really only way to verify analysis for analysis sake : Withhold small percentage observations from analysis (10%) Validate analysis at those withheld obs Measure ability of analysis system to reproduce their values Now built into GSI Can withhold and internally compare analysis Baseline CV also computed internally based on a simple single-pass Cressman analysis scheme Future performance metrics will be based on improvement over this Baseline 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 … 1st test set 2nd test set … … 10th test set Estimates of RTMA Analysis “Accuracy”Cross-Validation (CV) Ordering of the each type : ps, t, q, uv, spd 10 % withhold

  27. Surface Obs Stations

  28. A. CTRL Number of test set = 10 (10%) : 1st set Number of training set = 10 (10%) : 1st set

  29. It is an example of a "space-filling" curve discovered by David Hilbert in the early 1900's. : It literally covers every point in a square. Like all good fractals, it is generated in iterations. What is the Hilbert Curves? Iteration 0 iteration 1 iteration 2

  30. "A space-filling Hilbert curve provides an efficient and convenient tool for arranging randomly located data in a serial ordering from which it is then possible to draw multiple non-overlapping subsets of data, each subset tending to be more evenly distributed in space than the complete dataset. Each such subset can be used as independent validating data for a corresponding analysis that uses only the complement of that subset. In this way, a cross-validation of the parameters defining the covariance models can be carried out and the parameters optimized" Why should be the Hilbert curves for Cross Validation?

  31. Random and tanhx distribution By Jim Purser

  32. B. Hilbert_curve Number of test set = 10 (10%) : 1st set Number of training set = 10 (10%) : 1st set A. CTRL

  33. Comparison of the three test sets A. CTRL B. Hilbert_curve

  34. Var QC vs. No Var QC for CV

  35. Ctrl: without reject list Exp: with reject list 2006.11.20.~2006.11.26. (7 days) Reject list vs. no-list ( No CV) -0.275 -0.266 2.6973 2.7005

  36. Ctrl: without reject list Exp: with reject list 2006.11.20.~2006.11.26. (7 days) Reject list with CV -0.099 -0.093 1.8941 2.0049

  37. Anisotropic background error parameter Rltop : Function correlation length In the anisotropic background error covariance model, the pattern is very sensitive to the parameter. Rltop = 500 Rltop = 250 Rltop = 900 isotropic

  38. Anisotropic background error parameter tests BIAS distribution for 10 test sets (WIND) • Considered Parameters • Scale length • Function correlation length RMSE distribution for 10 test sets (WIND)

  39. Anisotropic background error parameter tests Mean bias of 10 test sets in experiments 1: isotropic 2: w1.0_t1.0_w900_t100 3: w1.3_t1.0_w900_t100 4: w1.6_t1.0_w900_t100 5: w1.0_t1.0_w500_t100 6: w1.3_t1.0_w500_t100 7: w1.6_t1.0_w500_t100 8: w1.0_t1.0_w500_t500 9: w1.3_t1.0_w500_t500 10: w1.6_t1.0_w500_t500 iso exp9 exp7

  40. Analysis Increment (U-wind) anl-ges (iso) anl-ges (exp9) anl(iso)-anl(exp9) Shaded : smoothed terrain Solid : analysis increment

  41. Analysis Increment (U-wind) anl-ges (iso) Shaded : smoothed terrain Solid : analysis increment anl-ges (exp9) anl-ges (exp7) anl(iso)-anl(exp9) anl(iso)-anl(exp7)

  42. RTMA - Phase I of AOR - leverage and enhance existing analysis capabilities in order to generate experimental CONUS-scale hourly NDFD-matching analyses - establish a real-time process that delivers a sub-set of fields to allow preliminary comparisons to NDFD forecast grids QC in RTMA: By gross error check, VarQC and reject list, the efficient QC could be done for Mesonet wind data. Cross Validation : As the accurate validation methods, the Cross validation is built in GSI and tested in RTMA. Hilbert curves : For getting the homogenous test sets, Hilber curves was introduced and successfully implemented in the RTMA system. Anisotropic background error : It was shown that the CV could be carried out to define the proper parameters. Summary and conclusion

  43. Thank you !!

More Related