Advanced Utilities Extending ncgen to support the netCDF-4 Data Model

1. Advanced UtilitiesExtending ncgen to support the netCDF-4 Data Model Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009

2. Overview • The NCGEN4 Utility • NCGEN4 Command Synopsis • netCDF4 in CDL: Types • netCDF4 in CDL: Typed Attributes • netCDF4 in CDL: Groups • Scope Rules • Specifying Data Constants • Rules for Using Braces • Special Attributes • Debugging Note: The Cycle • Building and Installing NCGEN4 • Extended Example

3. The NCGEN4 Utility • NCGEN4 extends the CDL language • Includes all of the netCDF-4 data model • True inverse of ncdump • Includes the special attributes (chunking, etc) • Supports binary, C, NOT FORTRAN (yet) • Experimental: Java and NcML • Can produce 4 kinds of binary netCDF files: • netcdf-3 (classic) 32 bit • netcdf-3 (classic) 64 bit • netcdf-3 (classic, but stored in netcdf-4 file format) • netcdf-4 (supports full netcdf-4 model)

4. NCGEN4 Command Synopsis Primary Command Line Options [-b] Create a (binary) netCDF file (default) [-o <file>] Name for the binary netCDF file created [-k <file_format>] Format of the file to be created 1 => classic 32 bit 2 => classic 64 bit 3 => netcdf-4/CDM 4 => classic, but stored in an enhanced file format [-x] Don't initialize data with fill values [-l <language>] Specify output language to use when generating source code to create or define a netCDF file matching the CDL specification. <filename> Input CDL file

5. netCDF4 in CDL: Types • New section called “types:” • Consistent with the output of ncdump • Supports the new primitive data types: ubyte, ushort, uint, string, int64 (LL), uint64 (ULL) • Supports the new user-defined types: • int enum enum_t {off=0,on=1,unknown=2}; • opaque(11) opaque_t; • compound cmpd_t { vlen_t f1; enum_t f2;}; • int(*) vlen_t;

6. netCDF4 in CDL: Typed Attributes • vlen_t v:attr = {17, 18, 19}; • Attribute typing is optional (=> type inferred) • Warning! x:attr = “abc”; is inferred to be type char, not string • Instead say string x:attr = “abc”; • Why? for backward compatibility with ncgen • Good practice to add “_t” to the end of all type names • Why? Because X :attr = … might be interpreted incorrectly; is X a type or variable?

7. netCDF4 in CDL: Groups • group: g {…} • A group can itself contain dimensions, types, variables, and groups • Name prefixing allows references to types and dimensions that reside in other groups • Example: /g/cmpd_t => Do not use ‘/’ in your names • Pretty much like the Unix file system • Or Windows, but using forward slashes

8. Scope Rules • Scope rules determine how references to a dimension or type without a prefix are interpreted • General rule: • Look in immediately enclosing group • Look in the parent of the immediately enclosing group and so on up the enclosing groups • For dimensions, if not found => error • For types, continue to search the whole group tree to find a unique match, then error if not found

9. Specifying Data Constants • Constants for user defined types require the use of braces {…} in certain places. dimensions: d=2; types: int(*) vlen_t; compound cmpd_t { int64 f1; string f2;}; variables: vlen_t v1(d); cmpd_t v2(d); data: v1 = {7, 8, 9}, {17,18,19}; v2 = {107LL, “abc”}, {1234567LL, “xyz”};

10. Rules for Using Braces • The top level is automatically assumed to be a list of items, so it should not be inside {...} • Different than C constants lists • Instances of UNLIMITED dimensions (other than as the first dimension) must be surrounded by {...} in order to specify the size. • Instances of VLENs must be surrounded by {...} in order to specify the size. • Compound instances must be embedded in {...} • Compound fields may optionally be embedded in {...}. • No other use of braces is allowed.

11. Special Attributes • Special attributes specified in an ncgen4 CDL file will be properly handled • Consistent with ncdump -s • Global special attributes • “_Format” – specify the netCDF file format • “classic” • “64-bit offset” • “netCDF-4” • “netCDF-4 classic model” • Overridden by the -k flag

12. Special Attributes (cont.) • Per-variable special attributes • “_ChunkSizes” – list of chunk sizes 1 per dimension • “_DeflateLevel” – compression level: integer (0-9) • “_Endianness” – “big” or “little” • “_Fletcher32” – “true” or “false” to set check summing • “_NoFill” – “true” or “false” to set persistent NoFill property • “_Shuffle” – “true” or “false” to set shuffle filter • “_Storage” – “contiguous” or “chunked” to set storage mode

13. Debugging Note • Use the “Cycle”, Luke • Use ncgen/ncgen4 to convert your <file>.cdl to <file>.nc • Then use ncdump to convert your <file>.nc to <file2>.cdl • Compare <file>.cdl to <file2>.cdl • Watch out for UNLIMITED! dimensions: u = unlimited; variables: v1(u); v2(u); data: v1 = {1,2,3,4}; v2 = {7,8}; Ncdump produces v2 = {7,8,_,_};

14. Building and Installing NCGEN4 • Easy: add --enable-ncgen4 to your list of ./configure flags • Ncgen4 will be installed along with ncdump and the original ncgen

15. Extended Example netcdf foo { types: ubyte enum enum_t {Clear = 0, Cumulonimbus = 1, Stratus = 2}; opaque(11) opaque_t; int(*) vlen_t; dimensions: lat = 10; lon = 5; time = unlimited ; variables: long lat(lat), lon(lon), time(time); float Z(time,lat,lon), t(time,lat,lon); double p(time,lat,lon); long rh(time,lat,lon); string country(time,lat,lon); ubyte tag;

16. Extended Example (cont.) // variable attributes lat:long_name = "latitude"; lat:units = "degrees_north"; lon:long_name = "longitude"; lon:units = "degrees_east"; time:units = "seconds since 1992-1-1 00:00:00"; // typed variable attributes string Z:units = "geopotential meters"; float Z:valid_range = 0., 5000.; double p:_FillValue = -9999.; long rh:_FillValue = -1; vlen_t :globalatt = {17, 18, 19};

17. Extended Example (cont.) data: lat = 0, 10, 20, 30, 40, 50, 60, 70, 80, 90; lon = -140, -118, -96, -84, -52; group g { types: compound cmpd_t { vlen_t f1; enum_t f2;}; } // group g group h { variables: /g/cmpd_t compoundvar; data: compoundvar = { {3,4,5}, Stratus } ; } // group h }

18. Questions?