Assessing the Maturity of Climate Data Records

1. Assessing the Maturity of Climate Data Records John J Bates National Climatic Data Center, NOAA/NESDIS

2. Overview • Climate variability and change are having profound effects on society • Society deserves full and open access to the data and methods used to produce climate products • Scientists often use nomenclature and methods that are difficult for non-specialists, and sometimes even specialists, to understand • Scientists must do a better job in standardizing their nomenclature and methods, and more clearly communicate with the public

3. Motivation • What is at stake? • History shows that weather observations did not become useful for society until a lexicon was agreed to • The Beaufort scale did this for wind climatology and maritime commerce in the 19th century • For climate services to benefit society, they must adopt a lexicon that sets expectations for openness, process and transparency that are accessible to the public • How might we define a climate record lexicon useful to the public in the 21st century?

4. Motivation • Climate Change is a highly applied scientific field with major aspects related to regulation and societal wellbeing • Society is, understandably, demanding scientific process for generating CDRs to be open and transparent • Increasingly complex observing systems require more process control to ensure quality, access, and preservation • For example the NASA/NOAA Suomi NPP satellite is generating over 5 TB/day of data • Software Engineering is also increasingly complex and process management is required to optimize cost, schedule, productivity and quality • These benefits are greatest for larger organizations that perform tasks over many years

5. Climate Observation Best Practices 60S-60N ERSST Annual Anomaly (1880-2009) • Steps to long-term monitoring • Over the last 20-30 years many investigators have developed methods for seaming together observations with evolving coverage and accuracies • From these experiences, common elements are emerging on best practices for Climate Data Records • How do we capture and make available these best practices and pass them to new generations? 60S-60N ERSST Annual Anomaly (1880-2009) 60S-60N ERSST Annual Anomaly (1880-2009)

6. Climate Observation Best Practices –Needs for Long-term Preservation What common preservation information do we need and how to capture communicate this?

7. A Maturity Matrix for Assessing the Completeness of CDRs • The Maturity Matrix assess Software Readiness, Metadata, Documentation, Product Validation, Public Access, and Utility on a Maturity on a scale of 1-6 in each area • Details can be found in Bates and Privette 2012 • DOI :10.1029/2012EO440006 • These attributes monitor process, documentation, and access to CDRs. • These are essential in ensuring openness and transparency • The Maturity Matrix attempts to ensure these attributes are available for a scientific assessment of a CDR • This is not itself a scientific assessment • The climate community was invited to apply the Matrix and comment

8. Community Input on the Maturity Matrix • The Maturity Matrix has been evaluated by a GCOS Workshop, EUMETSAT and the EU FP7 project CORE-CLIMAX , the ESA CCI, and the CEOS SST Virtual Constellation – • EUMETSAT CORE-CLIMAX Workshop in January 2014 performed a self assessment of 30 CDRs (23 satellite, 6 in situ, one re-analysis) • They updated the maturity matrix based on feedback and use and called it a system maturity matrix (in order to separate this from an assessment of the CDR science)

9. Community Input on the Maturity Matrix • CORE-CLIMAX workshop results: • Evaluation and Quality Control needs to consider both scientific and process quality expressed as maturity • Because the Maturity Matrix always needs interpretation it shall not be used for a beauty contest by adding up or averaging scores and ranking • Metrics for scientific and process oriented assessments need further development • Process maturity should periodically be assessed for ECV Climate Data Records entering climate (change) services

10. CDR Maturity Matrix – Next Steps • The CEOS-CGMS Joint Working Group on Climate is asking members to provide further evaluation of the CORE-CLIMAX System Maturity Matrix prior to recommending adoption (by later this year; NASA is a member of both organizations) • The WCRP Data Advisory Council is considering guidelines for scientific assessments and such assessments should be considered separate from system or process maturity • Software assessment tools (called static analysis tools) should be used to evaluate code to minimize complexity and remove excess code

11. CDR Maturity Matrix – Final Thoughts The stakes in climate change are too high to assume a standard research approach to the creation of CDRs. Society is demanding more documentation, openness, and transparency. It is imperative that we, the climate community, responds with quantifiable metrics that inform society of both the scientific quality and process maturity of CDRs.