RA QARTOD Implementation

1. RA QARTOD Implementation

2. Opening Remarks Discussion will follow, objectives for session: Understanding of the state of QARTOD at the RAs Identify gaps in implementations across RAs (variables, type of stations, etc.) Gather feedback on new manuals or updates to manuals Generate enough conversation so we can all start collaborating together Food for thought: How many implementations of QARTOD are too many? Are end-users interested in individual flags? Roll-up flags?

3. MARACOOS Implementation for HFR: Progress Challenges & Next Steps Radial tests (syntax, max velocity, location, radial count, spatial median) Python scripts to parse & reformat data + run tests File format to integrate flags + metadata Determining thresholds Re-evaluate 2018 archive to ID adjustments Automating tests that require multiple timesteps (temporal gradient)

4. MARACOOS DUCK station radial data for November 9, 2016 23:00 UTC (a) Original map. (b) Map displaying radials that pass the spatial median test.

5. CeNCOOS CeNCOOS Challenges: Diversity in Data Sampling rate Instrument quality Diversity of Oceanography Diversity of Domain Expertise Burk-o-Lator: High Sampling Rate, Lots of moving parts, in a lab (protected from environment) • Jupyter notebooks for 22 different stations (9 providers) → QARTOD tests. • Notebooks executable on Google Colab • Export test parameters to JSON • Provide Parameters to Axiom. Mouth of the Golden Gate: High Biofouling, Low Salinity pulses, Upwelling pulses, High Tidal energy

6. AOOS Goal: All AOOS-funded assets have QC flags by summer 2019 Functionality: Ingest flags from providers, calculate for everyone else Include in graphs, ERDDAP, all, downloads Link to provider QC page if available; list test config if we run it Roll up across time for data inventories, stats, monitoring

7. AOOS Running tests using fork of IOOS QARTOD github repo Current IOOS version: https://github.com/ioos/qartod (docs) Axiom version: https://github.com/axiom-data-science/ioos_qc (docs) Compatibility: Tests are essentially the same, but code structure, method signatures and build process have all changed Major differences: • Encapsulate all test config with QcConfig object • Convert "count" arguments into "time" (in seconds) arguments, for readability and transferability fromioos_qc.configimport QcConfig config = { 'qartod': { 'gross_range_test': { 'suspect_span': [1, 11], 'fail_span': [0, 12], } } } q = QcConfig(config) results = q.run(inp=list(range(13)))

8. SECOORA

9. SECOORA

10. SCCOOS Current status • SCCOOS real-time instruments include fixed-pier CTD’s, bottom-anchored moorings, pH sensors • Currently QA’d variables: temperature, salinity, conductivity, pressure, chlorophyll, PCO2, TCO2 • QARTOD checks are done during the sensor-to-NetCDF conversion; range, flatline, spike Future Opportunities • pH and O2 data coming online • QARTOD for pH not yet available • QARTOD for O2? • HABs data coming online • currently no guidance for HABs data? • WoRMS, OBIS, Darwin-Core

11. PacIOOS PacIOOS realtime instruments include fixed-pier CTD’s and bottom-anchored moorings Variables include temperature, salinity (conductivity), turbidity and chlorophyll QARTOD checks are done during the sensor-to-NetCDF conversion At present we apply the following: • timing/gap test • syntax • location • gross range However… • we simply report data (timing), read what we can (syntax) and instruments are fixed (location) • so the only meaningful check is gross range

12. NANOOS NANOOS integrates real-time fixed-pier sensors, depth profilers, and bottom-anchored moorings • Often multiple sensor packages, variables, and depths • Both NANOOS supported and independent local providers Several providers have implemented (or in dev.) QARTOD testing on their end. Will carry their QARTOD flags forward when feasible Implementation (in development) • Implementing centralized QARTOD checks and flags, during transformation from integration relational database to NetCDF files to be disseminated via ERDDAP • Developing on top of ioos quartod Python package • Building on GLOS’ Jupyter notebook for per-sensor assessments In a few months: syntax, timing/gap and gross range checks This year: spike and flat-line checks, some local ranges Visualization • Intend to use SCCOOS’ depiction of QC flags via ERDDAP • Later: Explore roll-up flag options within NVS. Have annotation and styling capabilities, but visual presentation already complex

13. GCOOS All data are coming from LDNs (1,324 sensors); GCOOS do not own assets QARTOD QC’d all historical and incoming data (timing, syntax,location,gross range,climatology) but not on spike, rate of change and flatline tests All files are distributed with QC flags Parameters evaluated:

14. CARICOOS

15. NERACOOS NERACOOS supports 21 real-time observing assets Systems are operated by academic, private, and federal partners, who are responsible for QARTOD QARTOD or QARTOD equivalent tests implemented on majority of the systems Working towards implementation of QARTOD flags Challenges • NERACOOS funding priority is on keeping systems operating. Minimal funding available for DMAC upgrades. • Legacy systems (pre-IOOS) need to be replaced but are supporting operational products Opportunity - can we leverage OOI buoy data management system to upgrade some of our systems?

16. GLOS GLOS QARTOD process • 75 operational assets; 10 non operational assets(historical) • automated process; runs hourly • Only reporting; no action • QARTOD flags available in netcdf files via TDS QARTOD tests • Gross Range, Spike, FlatLine, Rate of Change • Applied on total of 20 variables(sensor data) Next Steps • Adding more tests and refining test threshold values • Visualizing QARTOD flags; trends • QARTOD feedback loop • Leveraging event driven programming; alerts/notifications • Setting up a 2-way communication between GLOS and data providers