NPOESS EDRs vs. Climate Data Records (CDRs): Steps Toward Operational Production

1. NPOESS EDRs vs. Climate Data Records (CDRs) John J. Bates, Chief Remote Sensing Applications Division NOAA’s National Climatic Data Center

2. Outline • What is a CRD and why should NOAA produce CDRs? • Steps Toward Operational Production of Climate Data Records – Scientific Data Stewardship • Prioritization • Production • Productivity • Organization • Conclusion

3. Scientific Data Stewardship • NOAA’s Scientific Data Stewardship rooted in NRC dialogue and reports • NOAA/NRC SDS leads • Bates • Goldberg

4. NRC – NOAA Response • With the transition of the U.S. Global Change Research Program into the Climate Change Science Program (CCSP), NOAA was identified as the lead U.S. Agency on climate. • “NOAA’s new climate mandate is fundamentally different from its traditional weather forecasting mandate and raises a new set of challenges owing to the varied uses of climate data, the complexities of data generation, and the difficulties in sustaining the program indefinitely.” (NRC) • In response to this change and the planned transition of NASA research climate observing missions to the NPOESS mission, the Scientific Data Stewardship Project was proposed to begin the operational production of climate data records.

5. Key Elements of a Successful CDR Program A Climate Data Record (CDR) is a time series of measurements of sufficient length, consistency, and continuity to determine climate variability and change • CDR Organizational Elements • High-level leadership council • Advisory council to represent climate research community and other stakeholders • Fundamental Climate Data Record (FCDR) Teams • Thematic Climate Data Record (TCDR) Teams • CDR Generation Elements • High accuracy and stability of FCDRs • Pre-launch characterization of sensors and lifetime monitoring • Thorough calibration of sensors • Well-defined criteria for TCDR selection • Stakeholder involvement and feedback for TCDRs • Well-defined criteria for TCDR validation • Use of in-situ data for validation • Sustaining CDR Elements • Available resources for reprocessing CDRs as new information becomes available • Provisions for feedback from scientific community • Long-term commitment of resources for generation and archiving of CDRs and associated data Fundamental Climate Data Record (FCDR): Time series of calibrated signals for a family of sensors together with the ancillary data used to calibrate them. Thematic Climate Data Record (TCDR): Geophysical variables derived from FCDRs, often generated by blending satellite observations, in-situ data, and model output.

6. What is a CDR vs. EDR? Climate Data Records Data (Direct & Remotely Sensed) Time-tagged Geo-Referenced & Calibrated Sensor DataRecords (SDRs) Converted to Bio-Geophysical Variables Homogenization and Inter-Calibration EnvironmentalData Records(EDRs) Fundamental Climate Data Records (FCDRs) Climate Data Records or Homogenized Time Series Converted to Bio-Geophysical Variables Thematic Climate Data Records(TCDRs)

7. Only CDRs Provide the Accurate Rate of Global Warming We require Intercalibrated CDRs in order to merge multiple, legacy observing systems (Red is EDR, Blue is CDR)

8. Is There an El Niño?Expect the Unexpected Volcanic eruptions contaminate infrared-only SSTs. Lesson is that we must correct and reprocess EDRs to CDRs for critical climate variables

9. CDRs Seamlessly Transition Over Major Technology Upgrades: AVHRR to MODIS

10. Operational Climate Data Records –Prioritization, Production, & Productivity

11. Brief History of Prioritization • Mid-June 2006 CCSP Prioritization Workshop • Included WG Representatives • Had difficulty in balancing satellite measurement priorities with in-situ priorities – and priorities of different communities • Initial Development of a Hypertext Alternative • Workshop constrained by 2-D spreadsheets • Hypertext (Web-based) navigation approach provides better balance between complexity and understandability • Led to a representation showing how Essential Climate Variables interact with • Science Issues • Societal Benefits

12. Prioritization Approach • CCSP and Other Groups have identified ~40 “Essential Climate Variables” (ECVs), e.g. • Total Solar Irradiance • Snow Cover • Sea Level • Agencies involved in CCSP work have identified • Thematic Areas where data may provide societal benefit • Scientific Questions where data is needed for answering research questions • Prioritization aided by weighting contribution of ECV data in both Thematic Areas and Scientific Questions • Started with Spreadsheet – then have been extending weighting approach to database with web interface • Creates a priority according to connection and breadth of impact • Trying to avoid disciplinary myopia • Trying to ensure considering both scientific interest and societal benefits • Adding additional information based on availability of current and past data sets

13. Prioritizing Observations and Climate Data Records – Societal Benefits and Key Science Issues • Societal benefits of the Global Earth Observations System of Systems (GEOSS) emphasize the tradition of applied climatology but using 21st century systems • Key Science Intergovernmental Panel on Climate Change (IPCC) Issues address the policy-relevant scientific research topics of highest priority for Climate and Global Change studies

14. Prioritization - Future Work • Extend Input Opportunities • Creating web site with extensive information on satellites, instruments, in-situ data sources, and data products • Need Deeper and More Rigorous Approach to Prioritization • Can we quantify the economic value of climate data? • Can we quantify the required “Signal to Noise” for climate data on a rigorous basis? • Rigorous error budgets – and identification of physical basis for error estimates • Objective approaches to weighting influence of different variables on climate change (e.g. OSSEs)

15. NOAA CDR Production - Climate Central Requirements • For reprocessing, SDS requires an IDPS-like system, (Climate Central) to process: • SDS interdependent with CLASS, e.g., large data set I/O • CLASS and SDS mutually dependent • Requires full production engineering NOAA/CLASS Users NOAA SDS CLIMATE CENTRAL RDR SDR Multi-satellite FCDRs & TCDRs ACQUIRE RDRs IMPROVED SDRs MULTISATELLITE FCDRs PROCESS TCDRs PROVIDE ASSESSMENTS IMPROVEMENTS Products and Services Provide FCDRs and TCDRs

16. Production Example - Sea Surface Temperature Linkages & processes needed to create daily SST product

17. Production - Sea Surface TemperatureObservations Satellite Climate In-Situ Model NOAA POES NASA AMSR GTS Buoy & Ships NCEP Sea Ice

18. Production - Sea Surface TemperatureData Management / Processing All data management and processing performed at NCDC

19. Production - Sea Surface TemperatureAnalysis / Examples of Expected Uses

20. Productivity - Hurricane Intensity Reanalysis • Hurricane intensity historically estimated from “best track” data, in spite of its inherent temporal heterogeneities • An objective reanalysis of homogeneous 23-year satellite data set produced by NCDC • U Wisconsin developed an objective analysis algorithm to work with NCDC data. • “UW/NCDC” intensities have little temporal bias ~169,000 images ~2,000 tropical cyclones

21. NOAA’s Scientific Data Stewardship Program FY06 EDSM $2.5M C2D2 $1 to $1.5M Governance and Management Structure Scientific Data Stewardship Program Management NOAA Climate Program & Climate Board NOAA (5-10%) C2D2 NCDC ORA NOAA Observing System Council Scientific Data Stewardship CLASS Working Group NOAA Science Advisory Board Climate Working Group External (1-2%) Operational CDR Generation & Data Mgmt. NOAA, Other Agencies External Scientific (~90%) Research to Operations Research Climate Data Science Teams CDR Generation NOAA Other Agencies Universities Private Sector CLASS NOAA IT Infrastructure operators FCDR Teams Observations Scoping Requirement Systems TCDR Teams R&D Products and Services Theme Areas 21 Currently exists FY05 FY06

22. Conclusions • NPOESS EDRs  CDRs, but are a helpful first step • The foundation of operational production of satellite CDRs has begun within NOAA • Prioritization, production, productivity, and organization have been scoped and analogs begun using current data sets • Resource requirements for SDS are now being captured in the NPOESS climate remanifesting exercises