CF Metadata Conventions: Governance, Support, and Future

1. CF Metadata Conventions: Governance, Support, and Future Karl E. Taylor Program for Climate Model Diagnosis and Intercomparison Lawrence Livermore National Laboratory Presented to the GO-ESSP Community Workshop Paris, France 11 June 2007

2. Intro to CF metadata conventions • Climate and Weather Forecast Metadata that makes data files self-describing • Examples of CF metadata • Coordinate information needed to locate data in space and time • Further grid information (e.g., grid cell bounds, area) • Standard names – in conjunction with other attributes helps users determine whether data from different sources are the same physical quantity and helps distinguish variables within archives • Some processing info. (e.g., zonal mean, climatological processing)

3. Brief history of CF • Started with “COARDS” standard, and developed through efforts of a handful of volunteers • 2003: CF 1.0 released • 2005: CF white paper discussing future governance circulated • 2006: White paper revised and presented to WCRP WGCM • 2006: Current governance structure established • 2003-present: increasing acceptance of CF and adoption by several community coordinated projects: • IPCC AR4 archive conforms • PMIP, HTAP, regoinal groups, seasonal groups, ENSEMBLES …

4. Governance structure • Original CF “authors” turned over control to two working committees: • Conventions Committee • Standard Name Committee • Committee work done via email and web discussion (“Trac”) • Support provided by: • BADC: Alison Pamment (50%) – standard names • PCMDI: Kyle Halliday (20%) – web site support • WCRP/WGCM has been approached to appoint a CF Governance Panel charged with responsibility for stewardship of CF.

5. What happened at last GO-ESSP meeting? • Discussed CF governance & procedure for making changes • Some pressing CF issues were identified and in some cases individuals volunteered to try to make progress • Augmented grid information, especially for less structured grids. • Relationship between CF and GIS grids & metadata • Standard name issues (including dialects, profiles, hierarchy structures) • Relationship between CF and netCDF4 • CF handling of in situ observations • How to handle discovery information • Sample datasets and reference implementations

6. Agenda for today • Governance issues – what needs fixing? • Underlying principles – any adjustments needed? • Technical issues – what’s been done? Can we make progress on any of the discussion topics? • Supportive software – status reports Any additions? Other concerns?

7. Conventions Committee: Support and Governance Agenda Item GO-ESSP Community Workshop Paris, France 11 June 2007

8. Current committee members • Karl Taylor (chair) • Kyle Halliday (secretary) • Balaji • John Caron • Jonathan Gregory • Tom Gross • Steve Hankin • Jamie Kettleborough • Russ Rew • Rich Signell

9. Terms of reference • Further develop (revise) CF conventions • Consider implementation of CF metadata conventions in other file formats (besides netCDF) • Determine what is meant by CF conformance

10. Current procedure for modifying/extending CF • Anyone can propose a change • Discussion opens & a member of the committee volunteers to moderate • A provisional resolution is reached by consensus and summarized by moderator • Reference files are produced illustrating the new feature • If not a trivial change, trial implementations are carried out. • If problems emerge iterate • When consensus reached, the proposed change is provisionally accepted. (advanced implementers may begin adoption) • Collection of changes evaluated as a package leading to new CF versions released by consensus • Test reference files become part of the “test data”

11. Problems encountered. • Progress on some proposals, but few have been carried through to implementation. • Contrast with standard names committee • Few actions have taken place • With a couple of exceptions, the volunteer moderation of discussion topics has floundered • Procedure for introducing changes outlined in white paper: • Email • Web-based Trac system • Jonathan’s thoughts

12. Issues Concerning Underlying Principles of CF Agenda Item GO-ESSP Community Workshop Paris, France 11 June 2007

13. Underlying principles • Special needs of observations • Should the scope of metadata accommodated by CF include: • Description of experiment • Description of model • Description of higher-order grid-characteristics (e.g., relationship between u & v grids, which grid cells are connected to which) • Competing needs of data producers & application developers • CF extensions can be implemented faster by producers who sometimes are impatient for changes to be formally adopted • Should requirement be relaxed that all CF metadata reside in the same file as the data itself? • Bundling of common definitions of a quantity into I.D.’s (defined by a combination of attributes including the standard name + others) • Aggregation layers • CF profiles

14. Miscellaneous Technical Issues Agenda Item GO-ESSP Community Workshop Paris, France 11 June 2007

15. Miscellaneous technical issues • Ensembles (including realization_weight and other issues): Alison • Time axis “issues” for forecasts: Jonathan • Definition of the “geoid”: Alison • Use of “anomaly”: Alison or Jonathan • Subgrid variation description issues (use of “where”): Jonathan • Time and calendar issues for paleoclimate simulations: Jean-Yves? Balaji?

16. The Future of CMOR Karl E. Taylor Program for Climate Model Diagnosis and Intercomparison Lawrence Livermore National Laboratory Presented to the GO-ESSP Community Workshop Paris, France 11 June 2007

17. The Climate Model Output Rewriter (CMOR) • Provides a uniform interface for contributing CF-compliant data (adhering to “good practices”) to the growing number of MIP’s (e.g., CMIP, AMIP, CFMIP, PMIP, APE, HTAP) • For IPCC AR4, nearly all model output in the archive was written through CMOR. • Assures CF-compliance and all MIP-established output requirements are met. • Traps many metadata errors when files are produced. See http://www-pcmdi.llnl.gov/software/about_software.php

18. Project-specific CMOR tables facilitate and ensure consistency of model output from contributing groups • Proper specification of several coordinate attributes, including: • Correct standard name • “axis”, “positive”, and “formula_terms” attributes, as appropriate • Proper specification of several variable attributes, including: • Correct standard name • Required dimensions • “cell_methods” attribute • A capability to • Reorder axis order • Reverse axis direction (or translate longitude dimension) • Convert units (through udunits)

19. CMOR flagged common errors, including • Pointing out when required metadata are omitted. • Rejecting incorrect metadata (wrong units, inadmissible attribute values, etc.) • Rejecting inconsistent coordinate dimensions passed by user to CMOR. • Rejecting non-monotonic coordinate values or inconsistent boundary values, as passed by user. • Rejecting values that are clearly unrealistic (likely indicating improper units conversion or incorrect sign).

20. What changes will be implemented in the next 6 months? • Station data (needed, for example, by the HTAP project), including • Station names • Station locations • Output from regional models • With logically rectangular grids • But with non-cartesian longitudes and latitudes • i.e., data(i,j), lon(i.j), and lat(i,j)

21. Longer term prospects for CMOR • Difficult to see extending CMOR to write the grid-description files mandated by the GridSpec proposal. • Probably possible, however, to write the actual data files through CMOR (and leave it to other software to produce the special GridSpec files).