270 likes | 459 Views
THREDDS Data Server. John Caron Unidata/UCAR Mar 14, 2005. Outline. THREDDS Catalog Services THREDDS Data Server NetCDF-4 NetCDF-Java 2.2. THREDDS. Thematic Realtime Environmental Distributed Data Services Middleware between data providers and data users
E N D
THREDDS Data Server John Caron Unidata/UCAR Mar 14, 2005
Outline • THREDDS Catalog Services • THREDDS Data Server • NetCDF-4 • NetCDF-Java 2.2
THREDDS • Thematic Realtime Environmental Distributed Data Services • Middleware between data providers and data users • Enable discovery and use of scientific data • Not a data portal • XML over HTTP • Java Servlets
Catalog.xml Catalog Generator Catalog.xml THREDDS Catalogs HTTP Server Catalog.xml Data Server(s) DODS, ADDE, FTP, HTTP Client Application Datasets hostname.edu
Catalog Service Catalog Generator Catalog.xml Query Resolver Service DQC.xml Resolver Service URI URL Dynamic Catalogs, data services HTTP Tomcat Server Catalog.xml Client Application Data Server(s) DODS, ADDE, FTP, HTTP Datasets hostname.edu
Dataset Query Capability (DQC) • Dynamic and very large datasets • NEXRAD (level 3) from IDD • 150 stations • 20 products • Irregular times • The DQC describes the queries that the data server is capable of responding to.
Resolver Services • Logical Dataset, eg “latest ETA model run” • Dataset with Service type “Resolver” • On the client, the URI of the logical dataset is sent to the data server • The server finds what is available and returns an actual dataset URL.
Limitations • Catalogs are very general – “data protocol agnostic”. • Automatic metadata extraction needs to “crack” the datasets. • Data Servers must be separately installed, maintained, etc
THREDDS Data Server • Add the data access! • 100% Java – single thredds.war • requirement, not goal • Data input is NetCDF Java 2.2 library • Data output: • OpenDAP • WCS (in progress) • FTP, others (future)
THREDDS Data Server HTTP Tomcat Server Catalog.xml • THREDDS Server • OPeNDAP • WCS Application NJ22 library hostname.edu Datasets IDD Data
THREDDS OpenDAP Server • Catalogs instead of dods_dir • Next version of C++ library will also have THREDDS catalogs • Catalogs automatically generated • Based on DODS-Java library and the Aggregation Server. • Aggregation probably done at nj22 level
THREDDS WCS Server • WCS 1.0 specification • Proposing to add NetCDF as “blessed” binary file return type • OGC interoperability experiment • Difficult to find clients to work with • Working with CADCorp client • Liping Di (George Mason) • NASA GSFC ‘Earth Science Gateway’ • Looking for others
THREDDS WCS Server • Return formats • Geotiff floating point • NetCDF binary • No reprojections for now • Must have Grid coordinate system • Issues • Vertical coordinates • Time coordinates • Large number of coverages (granularity)
NetCDF-4 • Project funded by NASA to create new version of netCDF using the HDF5 file format. • “Extend and merge” netCDF and HDF5: • Widespread use and simplicity of netCDF-3 • Generality and performance of HDF5 • Specifically, we are funded to create netCDF-4 C library API, using HDF5 library underneath. • Russ Rew (PI), Ed Hartnett (Unidata), Mike Folk (co-PI), Quincey Koziol (NCSA)
netCDF-3 Interface netCDF-4 Library HDF5 Library NetCDF-4 Architecture NetCDF-4 C Library
NetCDF-4 and Java • 100% Java library for netCDF-4 files possible? • NetCDF-Java 2.1 already a 100% Java library for netCDF-3 files (and OPeNDAP) • NetCDF-Java 2.2: read HDF5 to determine what netCDF-4 data model should be • “Common Data Model” emerged for netCDF, HDF5 and OpenDAP. • OpenDAP 4.0 protocol being worked on • The 3 groups are actively discussing
NetCDF-Java 2.2 (nj22) • 100% Java library for netCDF, HDF5, OpenDAP, and more • THREDDS integrated • Under development • Alpha release: Nov 2004 • Beta release: May 2005 • Release: summer 2005
THREDDS Catalog.xml Application Scientific Datatypes NetCDF-Java version 2.2 architecture Image Trajectory Grid Station Radial NetcdfDataset NetcdfFile OpenDAP ADDE HDF5 I/O service provider NetCDF-3 NetCDF-4 GRIB NIDS GINI Nexrad … DSMP
I/O Service Provider Implement this interface: public interface IOServiceProvider { boolean isValidFile( RandomAccessFile raf); void open( RandomAccessFile raf, NetcdfFile ncfile); Array readData( Variable v2, List section); // only if you use Structures Array readNestedData( Variable v2, List section); }
CDM Goal: N + M instead of N * M things on your TODO List File Format #1 Visualization &Analysis NetCDF file File Format #2 OpenDAP Server File Format #N WCS Service
Coordinate Systems • Same underlying mathematics as VisAD, ASCII
Scientific Data Types • Point / Station (metars) • Unconnected • DQCs used with IDD station/radar data • Trajectory (floating buoys) • 1D connected • Radial (NEXRAD) • elevation ,azimuth, distance • Grid (Model output) • cartesian x, y, z • WCS
THREDDS Data Server HTTP Tomcat Server Catalog.xml • THREDDS Server • OPeNDAP • WCS Application NJ22 UI library NJ22 library NJ22 library hostname.edu Datasets IDD Data
Summary • THREDDS data server combines Catalog Services and data servers • As nj22 matures, data server will have more functionality • OpenDAP, WCS • NetCDF-4: simplicity of netCDF and flexibility of HDF5