In Situ Data, ncBrowse and OPeNDAP

1. In Situ Data, ncBrowse and OPeNDAP Presented byDonald Denbo ncBrowse: Donald Denbo, John Osborne Climate Data Portal: Donald Denbo, Jason Fabritz, Bernard Kilonsky, John Osborne, Charles Sun, Willa Zhu, Nancy Soreide http://www.epic.noaa.gov/cdp & http://www.epic.noaa.gov/java/ncBrowse

2. Outline • Accessing distributed in situ data • How Climate Data Portal does it • Basic client - server messages • ncBrowse - OPeNDAP • netCDF conventions • Attribute - variable mapping. • Recommendations • Develop OPeNDAP conventions framework • Develop in situ Java-DODS API that implements in situ convention. OPeNDAP Technical Working Conference

3. Climate Data Portalaccessing distributed in situ data • Links data from several NOAA ocean observing systems • Data access is accomplished in 3 steps. • LDAP Directory Service is used to find relevant servers. • Coarse Latitude/Longitude/Depth/Time selection • Refine selection of individual stations using ndEdit • Selected data can be downloaded and saved locally or displayed. • Client-Server communications are specified using CORBA IDL, defining both server and client APIs. OPeNDAP Technical Working Conference

4. LDAP Directory Service • Search on geographic region, observed quantities or keyword • Automatically populated from data servers. • Metadata instantly known to LDAP directories. • Access control for experimental data products. OPeNDAP Technical Working Conference

5. DatasetSelection OPeNDAP Technical Working Conference

6. Coarse Selection - lat, long, time box • Coarse selection in lat/long/time box • Important to reduce the number of stations returned from very large datasets. OPeNDAP Technical Working Conference

7. Fine Selection with ndEdit– view location in time band • Interactive, linked windows • View any 2D slice • xy, xt, xz, yt, yz, yt • Investigate space-time distribution • Select data • in any View Select time rangeSee datain range X-Y map view Y-Ttime range X-Y map view Y-Ttime range OPeNDAP Technical Working Conference

8. Fine Selection with ndEdit – select along an isobath Section Tool - select stations in a time band- along a depth contour OPeNDAP Technical Working Conference

9. Fine Selection with ndEdit – select along an isobath Selected stations (in a narrow time band) OPeNDAP Technical Working Conference

10. API Requirements to Support Multi-Tiered Data Selection • Determine what datasets are available from a server. This can be accomplished by: • Directory Service. LDAP or THREDDS • Query the server for a dataset list • Select a dataset and given a coarse selection get a list of stations and appropriate metadata. • Retrieve stations from a list of selected stations. OPeNDAP Technical Working Conference

11. Idealized messages between components

12. ncBrowse • A general purpose, interactive tool for graphical browsing of data and attributes in netCDF files. • OPeNDAP Data Access • OPeNDAP datasets compatible with the Java-netCDF API. • OPeNDAP datasets with more complex data models than netCDF supports, while challenging, is currently in development • Significantly enhancement with 3-D visulaizations using the VisAD library. OPeNDAP Technical Working Conference

13. ncBrowseOPeNDAP Issues • Client software most frequently use the netCDF API interface to OPeNDAP, thus minimizing the software modifications necessary. • The netCDF API cannot directly and/or transparently access more sophisticated OPeNDAP data types such as Structure and Sequence. OPeNDAP Technical Working Conference

14. ncBrowseOPeNDAP Goals • Provide ncBrowse users access to the growing collection of datasets available via OPeNDAP. • Access the dataset descriptor (DDS) and data attribute (DAS) structures. • Combine the DAS and DDS to provide easy access to variables and their metadata. • Provide an easy-to-use interface to the OPeNDAP Sequence and Structure objects. OPeNDAP Technical Working Conference

15. Combined DDS DAS

16. OPeNDAP - ExampleConnection to “CHRIS” - wind stress OPeNDAP Technical Working Conference

17. OPeNDAP - ExampleWind Stress from NOAAs Buoys OPeNDAP Technical Working Conference

18. OPeNDAP - ExampleNSCAT - Mean Wind Field OPeNDAP Technical Working Conference

19. ncBrowseissues • Combining DAS and DDS difficult because: • DAS and DDS structures don’t match • DAS and DDS namespaces are not always compatible • Not always possible to match attributes to variables with an automated system. • netCDF conventions • No guaranteed way to determine that netCDF conventions apply. • In Java, data structures for DAS and DDS are substantially different causing a little confusion and awkward programming. OPeNDAP Technical Working Conference

20. Simplified Class Structure

21. Recommendations • Develop OPeNDAP conventions framework. • Formalize existing DODS convention for netCDF files. • Develop in situ data access convention. • Develop a Java-DODS in situ API. OPeNDAP Technical Working Conference

22. Develop Conventions Framework • Create a repository of OPeNDAP conventions • Define a standard attribute to indicate convention(s) being used. • Convention framework should allow: • Multiple conventions in effect at the same time. • Hierarchy of conventions. For example, XXX/time_series and XXX/station. • The netCDF convention facility is a good model for formal conventions. OPeNDAP Technical Working Conference

23. netCDF Convention • netCDF files have a “standard” method of being expressed in the DAS and DDS. • Unfortunately • There is no “standard” way of determining that you should interpret Grid and Array objects and map DAS attributes to variables using this convention. • Proposal • That current practice be formally documented and a “OPeNDAP/netCDF” convention be created. OPeNDAP Technical Working Conference

24. In Situ Convention and API • Develop convention that supports accessing in situ data. The convention should provide support for: • Retrieving dataset information. For example, geographic and temporal extent, available variables, etc. • Efficiently accessing station information. (Latitude, longitude, depth, time, etc…) This information can be used for the refinement of a coarse selection. • Standard method of determining a unique station identifier. • Develop Java API that supports and enforces the in situ conventions. DODS-URL would be constructed based on convention, and returned objects would be independent on transport specifics (whether DDX or DDS is used). OPeNDAP Technical Working Conference