Q2O – QARTOD to OGC q2o.whoi

1. Q2O – QARTOD to OGC http://q2o.whoi.edu Integrating QA/QC tests, flags and standards into Sensor Web Enablement Janet Fredericks Woods Hole Oceanographic Institution Martha’s Vineyard Coastal Observatory August 26-28, 2008 NSSTC/UAH - Huntsville AL

2. This morning … • Introduction to Q2O project: who, what, why, when • Present example implementation from MVCO ADCP waves and CDIP data buoy waves • Discussion of QA/QC implementation • And perhaps we can add the where!?!

4. OOSTETHYS (MVCO in Fall 2006)

5. Why OGC?

6. Sensor Observation Service • “Provides an API for managing deployed sensors and retrieving sensor data” • Key here is access through a service • Three mandatory “core” operations GetObservation Describe Sensor GetCapabilities

7. SensorML What is it? SensorML provides standard models and an XML encoding for describing any process, including the process of measurement by sensors and instructions for deriving higher-level information from observations. Processes described in SensorML are discoverable and executable. All processes define their inputs, outputs, parameters, and method, as well as provide relevant metadata. SensorML models detectors and sensors as processes that convert real phenomena to data.   MBotts

14. QARTOD is a multi-organizational effort to address the Quality Assurance and Quality Control issues of the Integrated Ocean Observing System (IOOS). • Motivation • Expected increase in the number of observations, observing systems, and users • Need for simple, accurate and consistent quality content and descriptions for the real-time data to convey expected level of data quality to users Methodology: Engaging the community to develop standards. CourtesyJulie Bosch

15. Activity initiated by NOAA NDBC and CO-OPS in 2003 • Series of Workshops • Workshop format with plenary and breakout sessions • Approach to tackling the QA, QC, and metadata issues evolved • From separate breakout sessions for QA, QC, and metadata for all observations combined • To breakout sessions by observation focus group (covering QA, QC, and metadata) • Each observation focus group addresses the same guidance questions • Participation • Approximately 80 participants per workshop • Primarily observing system operators and data managers • Representation from federal agencies, oceanographic institutions, universities, and instrument manufacturers • Julie Bosch

16. QARTOD I: December 3-5, 2003 NDBC, Stennis Space Center, MS • Task to develop minimum standards for QA/QC methods and metadata • Considering existing observing systems and existing practices (presentations) • QARTOD II: February 28-March 2, 2005 CO-OPS, Norfolk, VA • Focus on wave, in situ currentand remote current (HF Radar) measurements • Identify unique calibration, metadata and QA/QC needs • QARTOD III: November 2-4, 2005 SIO, La Jolla, CA • Continued waves, in situ currentsand remote currents work • Added CTD focus group • Primarily addressed QC with some metadata emphasis • QARTOD IV: June 21-23, 2006 WHOI, Woods Hole, MA • Continued wave, in situ current and CTD work • Added DO focus group • Primarily addressed QA; increased emphasis on capturing metadata requirements • International considerations • Courtesy Julie Bosch The Workshops

17. Outcomes • Waves • Results being incorporated into National Waves Program data management plans • Results compiled and submitted to the IOOS DMAC Standards Process • In situ currents • Results specific to ADCPs compiled and submitted to the IOOS DMAC Standards Process • Remote currents • HF Radar community continued independent of QARTOD • Developed Data Management Standards for HF Radar • Submitted to the IOOS DMAC Standards Process • New initiative • Implementing QA/QC from QARTOD for in situ ocean sensors using OGC Standards/Sensor Web Enablement • Courtesy Julie Bosch

18. more information…http://www.qartod.org

19. NOAA-CSC RCOOS awarded Jan 2008 Integrate QARTOD QA/QC recommendations into OGC Sensor Web Enablement: • develop data dictionaries; set them up in SensorML profiles; • demonstrate products by integration into oostethys_sos and update or complement cookbooks with QA/QC implementation • Document and integrate results by providing results to community building organizations, such as MMI, ACT and QARTOD

20. Principal Investigators Janet Fredericks, Woods Hole Oceanographic Institution (WHOI); Julie Bosch, NOAA National Coastal Data Development Center/DMAC; Michael Botts, author of SensorML, along with his OGC development team from the University of Alabama at Huntsville (UAH); Philip Bogden, the Gulf of Maine Ocean Observing System (GoMOOS)/OOSTethys/OGC Oceans IE; and Sara Haines, University of North Carolina/ Chapel Hill and SEACOOS

21. http://q2o.whoi.edu Funded 3 yrs

22. Semantic Interoperability • Domain experts define best practices and required vocabularies – working with IT experts to keep interoperability goal in mind and guide in developing relationships for RDF definitions (http://qartod.org) • Build buy-in within and across communities to promote interdisciplinary potential (E.g., http://marinemetadata.org)

23. For each domain (e.g.., waves/currents) • Meet with domain experts to develop vocabularies, definitions and relationships (develop data model/SensorML profiles) • Present data model and profiles to broader community (develop tools and guides) • Hold workshop to demonstrate implementation and introduce/test products

24. Q2O Schedule • 2008 – complete waves implementation and demonstration • 2009 – complete in situ currents implementation begin CTD/DO work • 2010 – complete CTD/DO work and (if funded) demonstrate in situ currents and CTD/DO

25. Past Q2O Workshops • February 2008 – Introduction of QARTOD leads for each domain to the Q2O project and the SWE development team to QARTOD • June 2008 – Met with waves and in situ currents domain experts to discuss SWE and the concepts; discussed data dictionaries, use cases and reviewed QARTOD recommendations

26. Tasks for This Workshop • Review the data model for QARTOD recommended tests and the preliminary demonstration of SensorML profiles for waves (MVCO TRDI-ADCP and CDIP Buoy examples)

28. SWE-SOS? SWE-SOS QC-flags QC-flags Data Provider info & processing Set up and Deployment descriptions Processing history Test parameters specific to deployment Or Space time toolkit, or NWS or science user (Matlab) or NCDDC …. SWE-SOS NDBC MVCO QC-flags NDBC processing Manufacturer’s info & processing Serial NumberModel Number Sensor capabilities Processing history Test parameters specific to sensor Processing to meet the requirements of IOOS/NDBC etc Testing not available to a local node (nearest neighbor, regional range checks)

32. How and with What do we convey that information? • Get Capabilities • lists available data (properties) • uses SOS, Observation Offering • Describe Sensor • Provides sensore characteristics, deployment characteristics and processing methods • Uses SensorML • Get Observation • Provides the data • Provides test results • Points to file with processing/test info • Uses O&M • What information can we provide to data users or systems (OOSTethys)? • What sensors we have available as a service • Description of the sensor • Description of where / how / when it is deployed • List of the processing methods used on the data • List of the QC tests applied • The criteria used in the QC tests • The results of the QC tests • The data • … What do we have (know) to start with? A sensor (wave buoy or ADCP) with certain characteristics A sensor history QA info associated with a sensor Deployment characteristics Methods to process the data QC Tests to apply to the data … Show the data model (tests, criteria), dictionary, and where all the pieces and parts are located Shown as snapshots of information on multiple web Pages, docs, etc Show info conveyed in the SWE xml files Julie Bosch