EDSDWG Joint Meeting November 2, 2011 Ross Bagwell Hampton, Virginia NASA/CTS - GSFC

1. Measuring the Quality of Satellite Remote Sensing Data: Update on Efforts with QA4EO EDSDWG Joint Meeting November 2, 2011 Ross Bagwell Hampton, Virginia NASA/CTS - GSFC

2. Background GEO Task for Data Management: DA-09-01a: GEOSS quality assurance strategy • The Global Earth Observation System of Systems (GEOSS) must deliver “timely, quality, long-term, global information ” to meet the needs of its nine “societal benefit areas”. • This will be achieved through the synergistic use of data derived from a variety of sources (satellite, airborne and surface-based) and the coordination of resources and efforts of the members. • Accomplishing this vision, starting from a system of disparate systems that were built for a multitude of applications, requires the establishment of an internationally coordinated framework to facilitate interoperability and harmonization. • The success of this framework is dependent upon the successful implementation of a single key principle: • all EO data and derived products shall have associated with it a documented and fully traceable quality indicator (QI) • Success necessitates the means to efficiently communicate these attributes to all stakeholders.

3. Strategy Development • Led by a small CEOS team • Two community workshops • “Guiding principles” – Geneva, October 2007 • “Establishing an Operational Framework” – DC, May 2008 • CEOS Subgroups • Ad-hoc meetings • Endorsed by CEOS in November 2008 • Direct response to GEO Task DA-06-02 (now DA-09-01a) • Facilitate GEO vision for GEOSS • Inclusion of WMO (October 2009) • Evolving to meet all EO needs of GEO (including in-situ) • One of the key topics discussed in Oxford 18-20 October 2011

4. Essential Principle of QA4EO • Quality Indicators (QI) should be ascribed to data and products at each stage of the data processing chain (from collection and processing to delivery) • QI should provide sufficient information to allow users to readily evaluate a product’s suitability for their particular use or application (“fitness for purpose”) • QI needs to be based on a documented and quantifiable assessment of evidence demonstrating the level of traceability to internationally agreed reference standards • Ensure international harmonization and consistency • Quality Indicator • A means of providing sufficient information for a user of data or a derived product to assess the suitability for their particular application • Information should be based on a quantitative assessment of its traceability to an agreed reference measurement standard, but can be presented as a numeric or text descriptor providing the quantitative linkage • Traceability • Measurement result that can be related to a reference through a documented, unbroken chain of calibrations, each contributing to the measurement uncertainty

5. QA4EO Guidelines • Seven key guidelines of QA4EO: • Establishes QI on a satellite sensor derived data product • Provides guidance for documentary procedure • Provides guidance for reference standards • Provides guidance to comparisons to establish measurement equivalence • Provides guidance to establish validated models, algorithms and software • Provides guidance to the expression of uncertainty of measurements • Provides guidance to establish quantitative evidence of traceability • all are in support of QA requirements of GEO • intended to be generic in scope to cover all EO data-related activities • provide guidance (and indicative templates) on how to establish a QI and the means to obtain and document associated evidence • encourage/anticipate community specific interpretations • need to be supplemented by community/organization-specific best practices or procedures (operational in nature) • based on formal quality management systems • There are also 8 draft procedures (available online)

6. QA4EO Defined • What QA4EO is • A general framework • Based on 1 essential principle • Composed of 7 key guidelines • “living” documents that offer a flexible approach to allow tailoring to be commensurate with final objectives • A user (consumer) driven process • What QA4EO isn’t • A certification body • A set of standards for QA/QC activities and processes that would limit competitiveness or innovation and evolution of technology and methodologies • A framework developed with a top-down approach • A process and implementation that is judgmental and/or bureaucratic

7. QA4EO Strategic Implementation Board for GEO • Members • GEO Secretariat • GEO DA-09-01a task team co-chairs • FAO • GSICS • Metrology • NIST • CEOS • SBAs • Disasters • Health • Energy • Climate* • Water • Weather • Ecosystems • Agriculture • Biodiversity • Observers • GEO ADC • GEO Data Sharing Task Force • INSPIRE • OGC • GMES • GCOS • ISPRS

8. Feedback from QA4EO Workshop • QA4EO is important • Clear need to provide uncertainty estimates and traceability with data • Quality information is not always available or not easily found • Lack of consistency (concepts, terms, and definitions used) • Descriptions need to be understood by all users, not just experts in metrology • Benefits of QA4EO need to be announced with examples • Need to educate scientists on the need to provide uncertainties with measurements • Need to consider public dataset usable by all • Need to work across all users • Need to work to have a common quality process • Need to create a small number of example projects • There is a lot of work to be done • GEO Data Core – ADC (additional document that lays out databases that are QA4EO compliant) • International follow-on to GEO after 2015

9. Data Quality Guidelines • Not compulsory • Designed to assist EO communities • Allow DQI to be propagated through complete process chain • Facilitate transparency and consistency (internationally) • Need translation & require usage training for communities • Require some infrastructure/coordination • Require detailed procedures • Serve as an educational framework • Have the capability to evolve

10. Data Quality Indicators (DQI) • Not a new concept • ISO standards • OGC standards • NIST guidelines • EPA documentation • NOAA NCDC Index • Quality Information vs Quality Indicators • e.g. modelers have different requirement from data providers/producers • Need uncertainty of bias, description of how measurement was performed • Many times the information is there, but is buried • Comes down to the “fitness for purpose” • Users don’t have the time/energy/money/knowledge/experience to dig into the QI (or don’t see the “headline QI” they are looking for) • Need to make it easier for the user to find the information they want • One side is the provider, other side the user – possible for a user index assigned to the dataset to indicate usage for number of times for a certain purpose (not a quality index)

11. NASA Focus on QA4EO • Participation with GEO and CEOS • Various Subgroups and SBA efforts • Major Issue:  Providing appropriate and useful documentation of errors and uncertainties associated with our data, data products, analyses, and model results • Diane Wickland, CC&E Meeting • Primary effort within ESIP • Information Quality (IQ) Cluster (Greg Leptoukh) • Focus from ESDIS to the ESDSWG • Organization of existing DQI • Other potential opportunities for collaboration through • Sharing of information • Lessons learned • Future plans in Data Quality and Data Provenance • M. Maiden, ESA-NASA bilateral meeting

12. NASA Activities • Identifying Quality Indicators for remote sensing data • Addressing (and applying) ISO standards for data quality • White paper of remote sensing data quality (starting with inventory of best practices, developing structure and establishing standards, and eventually developing a framework for consistently addressing, computing/estimating, capturing and presentation quality of remote sensing across multiple current and future missions • During the ESDSWG, we will make a call for potential opportunities for collaboration through sharing of information, lessons learned, and future plans in data quality and data provenance • Planning a Metadata Workshop focusing on ISO 19115 (and related standards). Among the topics will be will be the incorporation of errors and uncertainties associated with our data. • Supporting the ESIP Federation Information Quality cluster activity

13. International Activities • Several discussions with Pascal Lecomte • Current Head of the European Space Agency (ESA) Climate Office • Former Head of Data Quality and Algorithm Management Office at ESA and CEOS WGCV Chair • Bojan Bojkov, the current Head of Data Quality and Algorithm Management Office at ESA and the WGCV Atmospheric Composition WG Chair expressed an interest in sharing ideas and tools within CEOS/WGCV umbrella and working together with NASA • Several discussions with GeoViQua, the EU-funded project, whose goal is to augment GEOSS Common Infrastructure (GCI) with innovative quality-aware visualization tools and geo-search capabilities • GeoViQua Project is very complementary to several NASA-funded data-quality-oriented project • Strong interest for collaboration