Web Seminar 08 June 2009

1. Web Seminar 08 June 2009 Reprocessed HADS Hourly Precipitation Data and Quality Control - Progress Dongsoo Kim National Climatic Data Center NOAA/NESDIS

2. Acknowledgement • NWS/OHD HADS Team led by Larry Cedrone • NCDC: T. Karl, S. Leduc, J. Bates, E. Kearns, S. DelGreco • MPR Working Group members: H. Fuelberg , E. Tollerud, S. Vasiloff, J. Bradberry, D.-J Seo, R. Kuligowski, C. Kondragunta, R. Boyles, Y. Zhang, G. Creager, E. Habib and B. Nelson. • SERFC: Bradberry and others • LMRFC: Caldwell and others • And many others

3. Outline of Presentation Background on HADS reprocessing Comparison with Real-time PP Use of Repro_HADS Web Development of Automated Spatial QC Summary

4. HADS Processing Suite NWS/OHD DCP Original SHEF HADS PC, Derived PP USGS Gauges RAWS Some NWS/RFC USACE Others Some NWS/WFO Edited NWS/NCEP/EMC Original ASOS PP UCAR NESDIS/NCDC Real-time flow Archival flow SRRS

5. HADS Reprocessing Suite NWS/OHD NCDC Original SHEF HADS Preprocessing Decoding for PC Station List Recovery of -99 Baseline PP Spikes check Level-3 PP, QF Level-1 PP Multi-sensor check Spatial check Level-2 PP Similar to Real-time Stored in UCAR/EOL Repro HADS LVL-1 Evaluated against Real-time Repro HADS LVL-2 Under evaluation against manual QC; LOUZIE and HPD

6. Comparison of two HADS Products Repro vs Real-time • Fraction of number of missing values by month for 2003-2005 in CONUS domain • Fraction of top-of-the-hour observations by month for 2003-2005 in CONUS domain • Dark bars are Repro HADS, gray bars are Real-time HADS • (NOV 2003 and JAN 2005 are missing months of original SHEF data – No comparison) (Accepted to WAF, 2009)

7. Impact of off-top-hour on Analysis Analysis with Repro HADS input Analysis with Realtime HADS input Difference of two Analyses

8. Impact of Missing Values on Diurnal Cycle of Precipitation Frequency • Number of missing values by hour for Apr-Sep, 2003-2005 in Pilot (NC & SC) domain. • Number of wet-hours by hour for Apr-Sep, 2003-2005 in Pilot (NC & SC) domain. • Solid circles are for Real-time HADS, open circles are for Repro HADS. (Accepted to WAF, 2009)

9. Repro HADS (Ver1.0) Web-page http://www.ncdc.noaa.gov/hads/ • Displays time-series of hourly precipitation • Displays neighbor HADS stations • Displays neighboring COOP stations and comparative statistics • Mass analyses: • Overlay with neighboring stations • User input of threshold value • User may select storm period • For instruction “How to Repro_HADS.doc”, click icon after slideshow

10. An example of paired comparison between hourly HADS (TRAN7) & daily COOP (319100) during Dec. 2004. – In preparation

11. An Example of Mass Analysis at DQDA4, June 2007

12. Development of Spatial QC that Emulates Manual QC of RFC http://www.srh.noaa.gov/lmrfc/?n=technicalpapers By Jason Caldwell and Joshua Palmer • LMRFC has been generating Quality Flags on hourly precipitation data, called “LOUZIE” data since May 2007. • SERFC began “LOUZIE” process in October 2008. • Unique dataset that has quality info at each hour at every station. • NCDC collaborates with Tollerud (ESRL) and Vasiloff (NSSL) to assess automated QC algorithm with LOUZIE codes (Kim et al., 23rd AMS Hydrology Conf. 2009)

13. Legends of LOUZIE Flags (LMRFC Practice) QUALITY CONTROL FLAGS L Light precipitation amounts, generally in regions with no precipitation reported. Also applies to dripping gauges following rainfall events and snowmelt O Overestimated gauge reports. The gauge is over-estimating the amount of precipitation observed on radar and surrounding reports. This is rare but accounts for problems such as double tipping U Underestimated gauge reports. The gauge is under-estimating the amount of precipitation observed on radar and surrounding reports. This is common for clogged gauges Z Zero reports. The gauge reports zero when precipitation is observed on radar and by surrounding reports. This is common for clogged gauges as well. E Enormous/extreme reports. The gauge reports a value larger than 0.10” and typically is due to equipment malfunction or decoding issues. DATABASE REMOVAL FLAGS I Ignored gauges. Upon further quality control these gauges were found to have 24-hour totals representative of the radar estimates and surrounding gauge network. The data is not removed from the database and is used in operations for hydrologic modeling. R Removed gauges. Upon further quality control, these gauges were found to be unrepresentative of the radar estimated and/or reported significant and notable issues with quality. The data is removed from the database and not used in operations.

14. Distributions of LOUZIE Flags (LMRFC Practice in 2007) O, U, E Z L (Kim et al., 23rd Conference on Hydrology, AMS Phoenix, AZ 2009)

15. Example of Flags for GREL1 (Kim et al., 23rd Conference on Hydrology, AMS Phoenix, AZ 2009)

16. Ten-levels of Quality Guidance Code • Buddy samples are a collection of data within ± 1hr and ± 0.5 degree of the target data. • Six variables from buddy samples are used in the production of quality guidance codes.

17. Comparison against HPD (DSI-3240) • The gauge at HPD station 224778 (NWSLI; KSOM6) punches measurement value on roll paper as well as transmits SHEF data in real-time via DCP. • When punched roll paper damaged, then whole month records are gone. Here, Feb- Mar 2006; May – Oct 2006. Otherwise, they agree reasonably well.

18. Comparison on 6-Month gmosaic Comparison of six month total (Apr. - Sep. 2003) rainfalls from two hourly gmosaics: (top) gmosaic from the input of ReproHADS + ASOS hourly gauge data. (bottom) gmosaic from the input of Level-2 HADS + ASOS hourly gauge data. Level-2 HADS is an output of spatial QC with quality guidance code (QG1) Overall, HADS level-2 input gmosaic field is smaller in magnitude by QC action. Spatial QC removed small but repetitive errors whose cumulative effect is pronounced in longer-term rainfall.

19. Warm Season Diurnal Pattern of Precipitation from 7 years An example: The diurnal pattern of rainfall occurrence at HADS station OCOF1 (Ochopee, Florida) derived from HADS level-2 data. The frequency max is at 15 LT, but max frequency of heavy rains (> 0.8 in/hr, red bar) is at 14 LT. Line connected with solid circles show distribution of missing values.

20. Summary - 1 • Reprocessing work has • recovered missing values (see slide 6) • increased top-of-the-hour observations (see slide 6) • identified time reference error of real-time environment • facilitated versioning of algorithm and product • Reprocessing work also highlights • need for refinement of metadata • need for quality history of each station • need for closer collaboration with operational community

21. Summary - 2 • Reprocessed HADS Ver1.0 is available now (for 2002 – 2008). Ver1.1 contains rescued HADS. • Reprocessed HADS Ver2.0 will be available (for 2002 – 2008) by the end of CY09. • Quality of metadata impacts quality of data • Non-unique station identifiers (i.e., recycled ID) • No information on instrument type • Delayed update from owner to distributor • Others • Use of Preprocessed HADS by more users will improve quality. • Repro HADS Ver1.0 is input to Multisensor Precipitation Reanalysis (MPR, stay tuned for next presentation).