1 / 27

L3MTO QC and Accuracy

L3MTO QC and Accuracy. Marina Timofeyeva Contributors: Annette Hollingshead, Dave Unger and Andrea Bair. General QC Flow. CPC: CD forecast. LFPS: Make forecast Primary QC Fix troubles. RHQ: Collecting trouble tickets. Archive. WS: Post product on web Fix Troubles. WFO:

juliet
Download Presentation

L3MTO QC and Accuracy

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. L3MTO QC and Accuracy Marina Timofeyeva Contributors: Annette Hollingshead, Dave Unger and Andrea Bair

  2. General QC Flow CPC: CD forecast • LFPS: • Make forecast • Primary QC • Fix troubles RHQ: Collecting trouble tickets Archive • WS: • Post product on web • Fix Troubles • WFO: • Ensure link to the product • Secondary QC of the product • Local value added • Customer services and outreach

  3. L3MTO Primary QC Interrogate CPC site and check for new forecast data 1. National 2. Alaska & Hawaii Early am ET (~ 7am- Noon) Parse data: extract mean & standard deviation Import data into Splus Troubleshoot and fix problems Run Splus scripts 6-8 hours Export data • Parse Splus output • Create XML formatted files Run QC scripts Fail Pass 45 minutes Push data to web team

  4. Secondary QC (WFOs) Start Web & Functionality Error Check Fail Send trouble ticket to WS Section A Pass Erroneous Data Check Fail Send trouble ticket to Regional HQ Section B Pass Conspicuous Data Check Fail Send trouble ticket to Regional HQ Section C Pass Other Discrepancy: Detailed QC Fail Send trouble ticket to Regional HQ Approve data Pass

  5. WFO Technical Support Team • Who are they? Regional representatives available to help answer any L3MTO questions • What is their role? Answer customer and partner questions and inquiries that NWS personnel are unable to address (see Guideline doc 3.2.3) Provide technical expertise & guidance: - interpreting L3MTO - comparing L3MTO to CPC 3-Month Temperature Outlook • How to find your regional rep? Page 13 in the L3MTO Guidelines document

  6. Web problem Station metadata Content and methodology QC failure l3mto.cms@noaa.gov Contact Regional Technical Support Team Trouble Ticket Procedure Check CSSA, if error report to Regional CSPM and RCPM WFO Quick Guide on Getting Help with L3MTO L3MTO Frustration

  7. WFO L3MTO Problem Reporting Protocol ( ) Web QC Secondary QC Detailed QC Add Stations Metadata Methodology Issue Trouble Ticket Compare to CSSA Discussion Technical Support Team Report to CSPM / RCPM Web Team TT collectors Buffer issues 1) Add Station (if not in CMS): report to CSPM; 2) Metadata Errors: - If CSSA is up-to-date, then submit change to CSPM - If CSSA is NOT up-to-date submit change to CSPM and RCPM who will expedite CSSA change Technical Issues Inconsistencies, Methodology Technical Support Teams Protocol in Guidelines document Marina Timofeyeva Annette Hollingshead

  8. QC Tool: L3MTO Analysis Resource Center - ARC • Section A allows checking short leads L3MTO “sanity” with current climate trends • Section B allows checking L3MTO agreement with some of CPC tools • Section C allows checking L3MTO presentation and methodology correctness • Section D presents L3MTO long term performance evaluation http://ww2.wrh.noaa.gov/hsd/climate/L3MTO/ARC/index.php

  9. QC Tool: L3MTO Analysis Resource Center - ARC

  10. QC Tool: L3MTO Analysis Resource Center - ARC

  11. QC Tool: L3MTO Analysis Resource Center - ARC

  12. QC Tool: L3MTO Analysis Resource Center - ARC

  13. QC Tool: L3MTO Analysis Resource Center - ARC Season: MJJ A CPC forecast of 49% (or greater) BELOW NORMAL probability is required to produce a significant* forecast for BELOW Normal category at the station. This automatically reduces the forecast for ABOVE Normal category to 28.3% or less. A CPC forecast of 36% (or greater) ABOVENORMAL probability is required to produce a significant* forecast for ABOVE Normal category at the station. This automatically reduces the forecast for BELOW Normal category to 28.3% or less.

  14. L3MTO forecast archive Additional scores to coordinate with NWS verification practices Extended selection choices for years and leads L3MTO Verification Univ. of AZ verification tool expanded to include:

  15. QC Tool: L3MTO Analysis Resource Center - ARC ARC section D verification intended for WFO use, it shows results of 3 verification methods: • Heidke Skill Score verifies 3 categorical forecast (pie chart format), indicates % improvement over climatology • Reliability Diagram verifies long term performance of individual %-tile of POE • Continuous Rank Probability Skill Score verifies accuracy, skill and resolution of POE, the most stringent score of the 3

  16. QC Tool: L3MTO Analysis Resource Center - ARC DJF JJA • Most stations have correct forecasts 30% (or more) of the time, and have little variability between the 13 leads • For all years analyzed together, shorter leads in the winter seasons have higher scores than the same in the warm seasons • Larger spread between stations’ performance in the warm seasons than cold

  17. QC Tool: L3MTO Analysis Resource Center - ARC

  18. QC Tool: L3MTO Analysis Resource Center - ARC

  19. Reliability Diagram for San Jose Reliability Diagram for San Jose for the Years 1994-2005 for the Season JJA for the Years 1994-2005 for the Season DJF 1.0 1.0 Perfect Reliability Perfect Reliability Forecast Forecast Climatology Climatology 0.8 0.8 0.6 0.6 Observed Frequency Observed Frequency 0.4 0.4 0.2 0.2 0.0 0.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 POE Percentile POE Percentile QC Tool: L3MTO Analysis Resource Center - ARC • For each station and forecast target season a reliability diagram was created to analyze each percentile’s performance. • Confidence limits around the perfect forecast line, were drawn from a binomial distribution at 95% confidence. • The forecast was compared with 1971-2000 climatology performance

  20. QC Tool: L3MTO Analysis Resource Center - ARC

  21. QC Tool: L3MTO Analysis Resource Center - ARC

  22. Verification Summary • Using multiple verification scores in climate forecast evaluation provides the most comprehensive assessment of long term forecast performance • Verification analysis identified locations and 3-month periods with satisfactory forecast performance, and their spatial and temporal variability • Overall, for the majority of stations and seasons, the L3MTO indicates higher skill than the 1971-2000 climatology

  23. Verification Summary • Although L3MTO in general proves to be more skillful than 1971-2000 climatology, it is still important to identify station/season combinations where L3MTO does not have long term skill. At such stations/seasons a Verification Filter will be applied. • The verification filter is the test that compares the station Heidke Skill Score with cutoff value at 75% confidence level that depends on number of independent forecast made for the stations. If the HSS is greater than passing, then the forecast is reported, if less, then 1971-2000 climatology is reported.

  24. Monthly Routine: (5) CPC Produces the CD Forecast (7) Verification Filter Verification Filter 5. CPC produces CD forecasts monthly 6. CD outlook together with bi-variate statistics reported from the regression analysis, station forecast is produced 7. Verification filter is the test that compares the station Heidke Skill Score with cutoff value at 75% confidence level. 8. Data distribution and display are provided at the L3MTO web site (6) Station Forecast Produced Sufficient Score? Yes No Produce Forecast Data Provide Climatology Data (8) Distribution and Display

  25. Additional Verification Information Additional Verification Information Additional Verification Information

  26. Additional Verification Information Additional Verification Information

  27. Additional Verification Information

More Related