The Open source Project for a Network

1. The Open source Project for a Network Data Access Protocol (OPeNDAP): A Discipline Neutral Approach to Interoperability on the Internet Presented at theFall AGU Meeting in San Francisco, CA9 December 2003Peter Cornillon and Dan HollowayGraduate School of OceanographyUniversity of Rhode and James Gallagher The Open source Project for a Network Data Access Protocol (OPeNDAP)

2. Outline • Trends in data system development • Data system elements • OPeNDAP • A data access element • In NVODS – an example • Status • Conclusions

3. Trend in Data System Development Is away from centrally designed, implemented and maintained systems toward The integration of independently designed, implemented and maintained system elements.

4. Data systems generally involve some combination of the following elements Discovery Analysis/ Visualization Archive Access/ Delivery

5. Historically these elements or subsets thereof have been developed and managed by the group assembling the data system.

6. Discovery Analysis/ Visualization Data System A Archive Access/ Delivery

7. Discovery Analysis/ Visualization Analysis/ Visualization Analysis/ Visualization Analysis/ Visualization Data System B Archive Access/ Delivery

8. With the advent of the Internet data system elements are increasingly being designed, implemented and managed independently by different groups.

9. Discovery Discovery Access/Delivery Application/ Visualization With a Plethora of System Elements … Archive Archive … Access/Delivery .. Application/ Visualization

10. And “complete” data systems being created from different combinations of the elements

11. GFDL netCDF URI HDF GSFC Binary Matlab IDL GrADS Ferret IDV Excel VisAD Access ncBrowse An example: NVODS – The National Virtual Ocean Data System … OPeNDAP …

12. More NVODS – Data Discovery GFDL netCDF URI HDF GSFC Binary IDL IDV Access ncBrowse Matlab GrADS Ferret VisAD Excel GCMD NVODS OPeNDAP

13. NVODS – Defining the Data System GFDL netCDF URI HDF GSFC Binary Matlab Access IDL ncBrowse IDV VisAD Ferret GrADS Excel ODC GCMD NVODS OPeNDAP

14. Responsibility Note that in distributed systems responsibility is distributed. For NVODS responsibility for • The data lies with the data providers. • The data access protocol lies with OPeNDAP. • Application packages (Matlab, Ferret, Excel…) with the developers of these packages. • Data location with the GCMD and NVODS.

15. Interoperability And Metadata

16. The Ultimate Objective of a Data System • To provide requested data to the user’s analysis/visualization package in a consistent, readily useable form. For example: A user might want all ocean temperature values (with associated times and locations) that lie between 90 and 110 m and have uncertainties less than 1ºC.

17. Interoperability • Know the format of these data objects. To achieve this objective, system elements must interoperate; i.e., the system must: Be capable of finding all data of interest. • Be capable of transforming from any of these formats to that required by the application software. • Understand the semantics of the data.

18. Metadata The degree of system interoperability is determined by the associated metadata. These interoperability requirements require in turn descriptions of the data; i.e., metadata.

19. Syntactic and Semantic Metadata The required metadata falls in two classes: • Syntactic metadata – Informationabout the data types and structures at the computer level - the syntax of the data; e.g., variable T is a 20x40 element floating point array • Semantic metadata – Information about the contents of the data set. e.g., variable T is sea surface temperature with units of ºC [RBH Note: This syntax is actually data structure. Syntax is the format of request.]

20. T 10 20 30 40 Syntactic and Semantic Metadata • Syntactic metadata provides the information needed to read and plot the data, but in general not to label the axes.

21. Syntactic and Semantic Metadata • Semantic metadata provides the information needed to label the axes in a plot. Temperature (ºC) 0 4 8 12 16 18 • 20 30 40 • Time (Hours)

22. Interoperability and OPeNDAP The two types of metadata suggest two levels of interoperability: • Syntacticinteroperability – Consistent format representation across data sets. • Semantic interoperability – Consistent semantic interpretations of the data. OPeNDAP mandates syntactic interoperability via a strict syntactic description of all data available via the system.

23. Interoperability and OPeNDAP OPeNDAP does not mandate semantic interoperability although it does allow for it. • The OPeNDAP data access protocol supports containers for semantic metadata, but places no requirements on the contents of these containers. • Some data sets are well described, others are not.

24. OPeNDAP in NVODS GFDL netCDF URI HDF GSFC Binary Matlab Access IDL ncBrowse IDV VisAD Ferret GrADS Excel ODC GCMD NVODS OPeNDAP

25. Data Selection • The OPeNDAP data access protocol allows for data subsetting at the server. • All data sets may be subsetted by variable, by the value of a variable for sequences and by array index for arrays. • If the data sets semantics are known, the data may be subsetted based on these semantics.

26. Discipline Neutrality • The OPeNDAP data access protocol is discipline neutral. It provides for • Consistent syntactic representation of the data regardless of the semantics. • Discipline specific semantics may be layered on the data.

27. Discipline Neutrality • Because of its discipline neutrality it is being used in a variety of data systems: • Oceanography • Meteorology • Solar-terrestrial physics • Hydrology

28. OPeNDAPStatus

29. OPeNDAP Server Sites in the US

30. OPeNDAP Client and Server Status Flat Binary netCDF HDF4 Matlab DSP Tables SQL FITS CDF CEDAR Data Data Data Data Data Data Data Data Data Data netCDF Matlab JGOFS FITS FreeFrom HDF4 DSP JDBC CDF CEDAR OPeNDAP Data Connector IDL Client Matlab Client netCDF Java netCDF C Web Browser Ferret GrADS IDV VisAD ncBrowse Matlab IDL Excel

31. Special Servers

32. Center for Ocean-Land-Atmosphere Studies Statistics

33. COLA/GDS Statistics

34. COLA/GDS Statistics

35. COLA/GDS Statistics

36. Lamont Statistics

37. Interesting OPeNDAP Access Statistics LDEO data accesses for 1st quarter of 2002 LDEO data accesses for 4th quarter of 2002

38. Conclusions • Responsibility in distributed data systems of the future will be distributed among the developers of the elements comprising the system. • To maximize the flexibility in assembling systems, system elements should remain as general as possible. • The OPeNDAP data access protocol has been designed to be discipline neutral in anticipation of the shift in data system development.

39. Conclusions Data systems based on the integration of independently developed system elements offer many more opportunities than more traditional centrally developed ones.

40. Conclusions “When systems evolve separately it is like sexual reproduction as opposed to centrally developed systems which evolve asexually” J. Caron 2003