netCDF

1. netCDF An Effective Way to Store and Retrieve Scientific Datasets Jianwei Li 02/11/2002

2. Motivation • Related scientific datasets in a single file. • Define a file format • Present the multi-dimensional arrays in a self-describing way • Common data access through a simple interface • Typical ways of access to multi-dimensional arrays • Platform-independent / Network-transparent Here comes netCDF!

3. Outline Today • Overview of netCDF • netCDF File Format • netCDF library and Programming Model • Advantages & Disadvantages • Future Plans for NetCDF

4. What Is netCDF? NetCDF (network Common Data Form) is an interface for array-oriented data access. It defines a machine-independent file format for representing multi-dimensional arrays with ancillary data, and provide support for creation, access, and sharing of array-oriented data. The netCDF software is implemented as I/O libraries for C, FORTRAN, C++, and Perl and other language for which a netCDF library is available. • Self-describing. • Portable. • Direct-access. • Appendable. • Sharable.

5. Why use netCDF? • Facilitate the use of common datasets by distinct applications. • Permit datasets to be transported between or shared by dissimilar computers transparently, i.e., without translation. • Reduce the programming effort usually spent interpreting formats. • Reduce errors arising from misinterpreting data and ancillary data. • Facilitate using output from one application as input to another. • Establish an interface standard which simplifies the inclusion of new software into existing system.

6. Platforms that netCDF runs on? • OpenVMS • OS/2 • SUNOS • ULTRIX • UNICOS • Windows NT • AIX • HPUX-9.05 • IRIX, IRIX64 • MSDOS (gcc, f2c, GNU make) • Linux • OSF Platforms that range from CRAYs to personal computers and include most UNIX-based workstations.

7. netCDF dataset Components • Dimensions • name, length • Fixed dimension • UNLIMITED dimension • Variables: named arrays • name, type, shape, attributes • Fixed sized variables: array of fixed dimensions • Record variables: array with its most-significant dimension UNLIMITED • Coordinate variables: 1-D array with the same name as its dimension • Attributes • name, type, values, length • Variable attributes • Global attributes

8. An Example netCDF File

9. netCDF File Format • Header • Number of record variables • Dimension list • Global attribute list • Variable list • Data (row-major) • Fixed-sized data data for each variable is stored contiguously • Record data a variable number of fixed-size records, each of which contains one record for each of the record variables in order. • Both use extended XDR

10. Data access in netCDF dataset Direct Access! Efficient for small subsets out of large datasets Forms of access: • access to all elements (whole array); • access to individual elements, specified with an index vector; • access to array sections, specified with an index vector, and count vector; • access to subsampled array sections, specified with an index vector, count vector, and stride vector; • access to mapped array sections, specified with an index vector, count vector, stride vector, and an index mapping vector (map the indices of an element in an external subarray to the offset in internal buffer).

11. Dataset Access Example Start[] = {2, 1}, Count[] = {3, 2}, Stride[] = {3, 2}, Imap[] = {1, 3}

12. netCDF Library & Programming Model Modes: definition mode, data mode IDs: dataset ID, dimension ID, variable ID, attribute number Create & WriteRead by name Read sequencially Add dim,var,att

13. Example library functions

14. netCDF data types and type conversion • Different library functions for each data type • Automatic conversion to or from external type • Conversion from one numeric type to another

15. Other Data Types? • Data Structure • No direct support • Can build some data structure by grouping arrays or by using data in one array as pointers into another array. Inefficient, and may not be self-describing! • Character String • not a primitive netCDF external data type • Charater-string attributes: well represented by NC_CHAR values and its length. • Charater-string variables: use a character-position dimension as the most quickly varying dimension for the variable (dim-length = the max string length)

16. Share Data Between Read and Write • Support for multiple read and single write • Use NC_SHARE mode to create or open dataset to use unbuffered I/O • Or call nc_sync() to force writing changes back to disk • Header changes in define mode are synchronized to disk only when nc_enddef() is called, reader must call nc_sync() to bring header up-to-date in its memory

17. Advantages • Simple and useful libraries for array-oriented data access • The dataset is self-describing • Portable and easy to share • Flexible and efficient data access • Mechanism to support extendable arrays

18. Disadvantages • No support for multiple concurrent write • Limited number of external numeric data types: 8-, 16-, 32-bit integers, or 32- or 64-bit floating-point numbers. • No support for nested data structures such as trees, nested arrays, or other recursive structures • Only one unlimited dimension. Multiple variables sharing an unlimited dimension, must all grow together. • Redefine(structure changes) that requires more file space is accomplished by copying the dataset.

19. Future plans for netCDF • Parallel write support • Support for bit data type and transparent data packing for bit variables • Support for efficient structure changes • Support for nested arrays and other data structure