1 / 26

AHPS River Data Quality Control Techniques and Tools

AHPS River Data Quality Control Techniques and Tools. OCWWS/HSD Hydrologic Support Branch December 15, 2011. Outline. Introduction Solution Review of Quality Control (QC) Operations Operational Discussion Identifying ...Problem Data Reporting ...Problem Data Correcting ...Problem Data

charvin
Download Presentation

AHPS River Data Quality Control Techniques and Tools

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AHPS River DataQuality ControlTechniques and Tools OCWWS/HSD Hydrologic Support Branch December 15, 2011 Note: This presentation is being recorded for training purposes.

  2. Outline • Introduction • Solution • Review of Quality Control (QC) Operations • Operational Discussion • Identifying ...Problem Data • Reporting ...Problem Data • Correcting ...Problem Data • Conclusion Note: This presentation is being recorded for training purposes.

  3. Intro: Problem • Bad observed data at water.weather.gov • Data is very visible • Evident on AHPS maps/graphs/tables • Increased use of web info brings greater scrutiny • Data used by public, countless partners at all levels • Bad data characteristics • Usually involves data spikes • Focus is on observed river data • Forecast and precipitation data on web not covered • Current WHFS tools provide only limited support Note: This presentation is being recorded for training purposes.

  4. Intro: Goals • Eliminate bad data presented to public • Ensure reliable observations, best forecasts • Provide decision support at local, regional, and national levels using web info • Minimize burden on WFO staff - provide automated tools where possible Note: This presentation is being recorded for training purposes.

  5. Solution: Summary QC process modeled as three fundamental steps: • Identify bad data automatically - Minimizes burden of manual review • Report/notice detected bad data • Reported to forecaster • Automated, still requires user to take action • Noticed by forecaster - Manual review can simultaneously identify/”report”... • Correct reported bad data - For AHPS, manually set data to missing (not delete) Note: This presentation is being recorded for training purposes.

  6. Solution: Principles • Focus on data spike problems • Most “bad” data spikes detectable via rate-of-change (ROC) check • But not all ROC-failed data are actually bad or questionable • Focus on spikes assumes range checks locally configured • Maintain separation of QC and Alert-Alarm (AA) functions • ROC thresholds exist for both QC and AA • QC judges data quality, AA judges threat • Difficulty in trusting precise thresholds for ROC complicates automated QC process Note: This presentation is being recorded for training purposes.

  7. Solution: Details • Identify using rate-of-change (ROC) check • Configure WHFS ROC_checker • Ensure thresholds are appropriate – Done by local office • Set token so ROC-failed data set as questionable, not bad (qc_questorbad) – HSD Support will do • Report using new QC report utility • Complements Alert-Alarm report utility • Configure new utility operations • Set up new utility in existing cron -HSD Support will do • Set up trigger in textdb operations - Done by local office • Define QC product id in report script - Done by local office • Correct using WHFS TimeSeries application Note: This presentation is being recorded for training purposes.

  8. QC Review: Info Available • Web QC part of general hydro data QC and Alert-Alarm operations • Previous WHFS QC/Alert-Alarm webinar • Given October 6, 2009 • Powerpoint and recording available at:  http://www.nwstc.noaa.gov/Hydrology/hydromain/webinars/index.html • Available WHFS documentation https://ocwws.weather.gov/intranet/whfs Note: This presentation is being recorded for training purposes.

  9. QC Review: QC Checks • All value set to one of three QC levels: • Good, Questionable, Bad (G, Q, B) • QC Checks in WHFS • Rate-of-change • Only one ROC threshold • If exceeded, set to Q or B (token controlled) • Range • Separate thresholds for Q and B • Obs-Forecast Difference - set to Q • Station Consistency • New report utility on LAD • Compares data from multiple sources at same station • Does not set QC flag New focus Assume already setup ok Note: This presentation is being recorded for training purposes.

  10. QC Review: Processing • Hydro data organized by type (obs, fcst, other) and physical element (PE) • River Data Access for AHPS • Height / FcstHeight tables (stage) • Discharge / FcstDischarge tables • RejectedData table *not* read • Written by SHEFdecode for out-of-range data (per token shef_post_baddata) • ROC Bad data not moved there! (regardless of token) • HydroGen Data Products • Tables read by genXML function within HydroGen operations • HML product file generated and sent every 15 minutes! • HydroGen does NOT filter Bad data [say what!? change???] Note: This presentation is being recorded for training purposes.

  11. QC Review: HydroBase Controls • QC thresholds defined as: • Default values for physical element • Location-specific values override defaults • ROC thresholds • ROC QC threshold should be set • ROC Alert-Alarm thresholds are separate • Help identify hydromet threats, not bad data • If sensor continually sends bad data, consider disabling the station or specific data elements  • Location->Modify Location-> Additional Info: disable station for AHPS and WHFS • Data Ingest->IngestFilter: disable specific element for AHPS and WHFS • Setup->HydroGen Configuration: disable station • Can also use AHPS CMS to... • Define annotated hydro note to describe data QC problem  • Disable station able using “Sensor in Service” option Note: This presentation is being recorded for training purposes.

  12. Operational Discussion Identify: Hydro Database ---- Good Questionable Bad NEW ROC_checker Report: QC report product report_hydroQC Text Database Alert/Alarm report product report_alarm Correct: trigger TimeSeries Web HML product HydroGen Note: This presentation is being recorded for training purposes.

  13. Identify Bad Data: ROC Checks • Automated ROC checks • Detects data spikes (QC) or quick changes indicating threat (AA) • Compares successive time series values • Only second value in value pair being checked is flagged • Ignores B data, ignores Q data if program argument set • Data exceeding QC ROC threshold • Flagged as Q or B based on token qc_questorbad • Default behavior is to set to B • Even if B, not moved to the RejectedData table so AHPS will still use it since AHPS does not filter bad data!!! • HSD Support will change token to set to Q Note: This presentation is being recorded for training purposes.

  14. Report Bad Data: Options Options for becoming aware of data QC problems: • Internal text reports using textdb notification: • New QC summary product • Existing AA summary product • Could set up dedicated report for only report ROC alert-alarms • Use –fRMT in run_roc_checker script • Must consider this since some “bad” data may actually only be flagged for alert-alarm purposes • Manual review using WHFS applications • Manual review using AHPS web pages themselves • Manual review using local CRH app • Other ideas? Note: This presentation is being recorded for training purposes.

  15. Report Bad Data: New QC Report • Name: report_hydroQC • Arguments: • Table – Height, Discharge, or both • Lookback time – number of minutes (MM) • Location filter – all, HSA only, HSA AHPS only • Other switches/filters to be added per request • Function: • Reads database table(s) • Considers all questionable or bad values • Filters on location filter • Filters on time period ending at current time and beginning MM minutes earlier • Formats text product • Sends text product to textdb if Q/B values found Note: This presentation is being recorded for training purposes.

  16. Report Bad Data: New QC Report Sample QC Report: ** Report of Questionable/Bad Hydro Obs Data **Period (Z): Tue 12-13 06:04:09 to Tue Dec 13 18:04:09BOGW1 HG RP Dec 13 10:30 [ROC 1/3] (Bogachiel River near La Push)    CENW1 HG RP Dec 13 12:00 [RNG 1/2] (Chehalis River at Centralia)CRNW1 HG RP Dec 13 10:30 [ROC 1/1] (Snoqualmie River near Carnation)UIL   HZ RZ Dec 13 12:00 [SHF 0/1] (Quillayute)ALRW1 QR RZ Dec 13 15:00 [ROC 1/0] (Alder Dam)Created: 12-13-2011 18:04:09 GMTReading Height, Discharge data.Shows latest obs time [QCreason questionable/bad value count] Note: This presentation is being recorded for training purposes.

  17. Notice Bad Data: WHFS Applications • HydroView LiveData-> • Questionable/Bad window • Rejected Data (‘trash”) window • TimeSeries • Tabular/graphical display • River Monitor • Automatically updates Note: This presentation is being recorded for training purposes.

  18. Sample HydroView Questionable/Bad Window Note: This presentation is being recorded for training purposes.

  19. “A” means alert/alarm In effect Sample RiverMonitor Main Window Note: This presentation is being recorded for training purposes.

  20. Sample RiverMon AlertAlarm Window Note: This presentation is being recorded for training purposes.

  21. Notice Bad Data: AHPS Pages • AHPS Page features • “Area HydroGraph” feature • Under “Additional Resources” heading • Allows quick, multi-station review • Monitor AHPS map on screen in office • Snapshots only - must refresh • Local CRH app is compact, auto-refresh of “area hydrograph: feature • http://intra.crh.noaa.gov/metdat/DDC/AHPS/ahps_qc.html • On METDAT server, under DDC/AHPS/ Note: This presentation is being recorded for training purposes.

  22. Correcting Bad Data:TimeSeries • General functions  • Primary tool to edit data • Also very helpful for reviewing data • Using the "pre-defined groups" mode, can be pre-configured to show select station data • Correcting of data • Data already sent to AHPS must be set to missing to correct it at the web farm database • Next HydroGen run will retransmit data to web • Presentation, titled "Data QC Procedural Change for WHFS", describing this:  <http://www.werh.noaa.gov/Hyd/ahps/ahps_resource_phase6_final.htm#9._Data_QC_Procedural_change_for_WHFS_> Note: This presentation is being recorded for training purposes.

  23. Sample TimeSeries Tabular Window Note: This presentation is being recorded for training purposes.

  24. Conclusion: HQ To-Do • HSD Support To-Do • Will update token • Will update script to create hydroQC product • Will provide instructions for local changes • May do this as early as next week • Future Changes • Will push for future AWIPS work to improve displays and tools for QC and situational awareness Note: This presentation is being recorded for training purposes.

  25. Conclusion: WFO To-Do • Configure • Review QC/AA thresholds in HydroBase • Define triggers in textDB • Define product id in report_hydroQC script • Review data regularly • RiverMon/PrecipMon, AA reports, HydroView, TimeSeries… • Correct • Set AHPS data to missing...don’t delete Note: This presentation is being recorded for training purposes.

  26. Conclusion • Questions? • Contact: whfs@noaa.gov • Thanks Note: This presentation is being recorded for training purposes.

More Related