Datzilla

1. Datzilla A web-based error reporting and tracking tool Your friendly Datzilla Gatekeeper NWS data stewardship workshop – Peachtree City, GA, 20-21 June 2007 http://www.ncdc.noaa.gov

2. What is Datzilla? Datzilla is a Web-based tool to report and track errors in NOAA-held data, metadata, or data delivery systems. It is maintained by the Southern Regional Climate Center and NCDC is a participant to the extent of addressing data it holds and systems it maintains.

3. Errors vs. Corrections • Datzilla intended to correct errors • Corrections to observations are not errors. • For changes to recent observations (common) • William Angel (COOP / ASOS) – William.E.Angel@noaa.gov • Blake Lasher (LCD ASOS) – Blake.L.Lasher@noaa.gov • Send amended forms to NCDC via normal channels. • Corrections to historical observations (uncommon) • Amended forms • Datzilla (in some cases) • Must have a compelling reason to change a historical obs. • Corrections due to equipment problems may also require a metadata rendition.

4. Some possible reasons for a Datzilla ticket to NCDC • Data issue (usually) • That TMAX of 74 should be a 47! • Your #$&!*@ QC clobbered my data! • System issue • CDO inventory doesn’t match the data I got! • Metadata issue • This station’s COOP number is wrong!

5. Who should be submitting error reports to Datzilla? • NCDC personnel (yes) • RCC climatologists (yes) • State Climatologists (yes) • WFO DAPM/OPL (yes) • Non-NOAA researchers (no) • COOP Observer (NO) Outside parties should make reports to local WFO, SC or RCC.

6. Creating a DatzillaError Report Step 1: Log in and select “New” from Actions list. If you do not have an account, you may request one here CLICK THERE http://datzilla.srcc.lsu.edu/datzilla

7. Good tickets and Bad • Pre-report investigation (critical) • Is the error really an error? • Is it the right station? • Is it reproducible (browsers, systems)? • Do you have the right system? • ACIS, ThreadEx, NCDC data may be different • Have you proposed a solution? • “I think it’s too high.” is not a solution.

8. A good ticket • Clear title • Good description of problem • ID of station • Attached form • Proposed solution

9. Please don’t drive us crazy with wild goose chases! A bad ticket • Vague title • No supporting material • No suggested fix • Hunches • Data “corrections” • Vague description of problem • No ID of specific station, element, date

10. Investigating the error • xmACIS • An RCC-maintained display tool for NOAA data • Interprets data flags from NCDC data sets • Includes recent data that may not be QC’d • Prioritizes simultaneous values from different data sets • Displays “M” for invalid as well as missing data • May thread data from several stations • Primary, Secondary, Tertiary forms: • 1001, B-91, 10-A, … • 1014, LCD, CD, CRB • Media articles, eye-witness accounts, proxy data • Paper, WSSRD, IPS … Wheeling, WV flood of 1936

11. Who will investigate? • Karsten Shein (gatekeeper / historical) • William Angel (operational data) • Henry Ray (metadata) • Bryant Korzeniewski (historical) • Mark Lackey/Deb Franklin (CDO) • Keith Eggleston (xmACIS) • … and others as needed

12. My ticket’s been open forever! • Some are easy to fix • Small-batch keying errors • A few missing values • Some are not • May be part of a larger issue • May exceed NCDC resources • May not have a straightforward fix • May involve parties/systems outside NCDC • Old pubs will not be reissued • But the data errors they may contain will be fixed

13. Your #$&!*@ QC clobbered my data! • You are right to alert us. • However, before you do … • Think about why our QC would flag that data. • Most often, the problem is not our QC • Observer date shifting • Observing at time different than expected • Instrumentation problems • Known climate anomalies? • Unreported temporary anomalies?

14. Any questions? Datzilla is maintained by: Kevin Robins (SRCC) krobbins@srcc.lsu.edu Datzilla POC at NCDC: Karsten Shein Karsten.Shein@noaa.gov 828.271.4223 Once again, your friendly (no, not that friendly) NCDC Datzilla Gatekeeper! NCDC Halloween, 2006

15. Web Search, Store, Retrieve, Display(WSSRD) NWS Data Stewardship Meeting – Peachtree City, GA, 20-21 June 2007

16. What is WSSRD? • Electronic image archive (database) • Web-based interface • http://noaa.imcwv.com/ • 50 + million images • COOP network forms • Nearly all post-1984 • 14 of 50 states back to 1893 • Access to: • U.S. Government and contractors • Educational institutions doing env. Research • Researchers associated with NOAA projects

17. The WSSRD interface CRB Old 1st order daily forms (SWO) CD and LCD COOP Observation forms Some F-6 forms Older 1st order monthly forms

18. WSSRD COOP Search This is the (E)xpression builder to widen your search. Keep your searches general, and use wildcards (*) where possible This is the (L)ist builder to widen your search. You’ll probably never use these fields

19. WSSRD COOP Search Results Click here to view record Note: A search by COOP number would not have returned this record!

20. COOP form on WSSRD

21. Alternatives and Plans • Most COOP forms available in NCDC’s IPS • Access via normal NCDC website • WSSRD in process of being replaced by COTS software (in review) • Transition process may take a while.

22. Want An Account or More Info? Please contact Cindy Karl at NCDC Cynthia.B.Karl@noaa.gov Or visit http://www.ncdc.noaa.gov/oa/climate/cdmp/wssrd.html

23. WHEN MAKING A WSSRD ACCOUNT REQUEST… • Please Indicate via E-mail to Cindy Karl… • Your Job Title (e.g. Lead Meteorologist) • Your Organization (e.g. NWS-GSP) • The Data That You’ll Need To Access (e.g. COOP) • Will Accept Account Requests For… • Researchers and Those From Universities (*.edu Addresses) • Please Include Information on Your Research as it Relates To Your Need for a WSSRD Account • Indicate the Time Frame Needed For Use of WSSRD (e.g. 90 days) • Those Sponsored by NOAA (e.g. Contractors, Interns, etc.) • NOAA Sponsor Must Send E-mail Request on their Behalf