1 / 16

Enhancing Data Access with Dapper Server

DAPPER is a web server enabling distributed access to in-situ data via OPeNDAP protocol. With support for multiple clients like GrADS and Matlab, it offers data from PMEL EPIC database and NODC World Ocean Database 2001.

lindabhill
Download Presentation

Enhancing Data Access with Dapper Server

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DAPPER: An OPENDAP Server for In-Situ Data Joe Sirott Donald W. Denbo, Willa H Zhu University of Washington PMEL/NOAA

  2. What is Dapper? • Web server that provides distributed access to in-situ data via OPeNDAP protocol • Clients include GrADS, Matlab • Available data: • PMEL EPIC database • NODC World Ocean Database 2001

  3. Problems with in-situ data • Irregular (x,y,z,t) • Not all variables available in all profiles • Data from a given profile may be revised as QC is applied • Each profile usually stored as individual file • No standard APIs

  4. Problems with in-situ data • Irregular (x,y,z,t) • Not all variables available in all profiles • Data from a given profile may be revised as QC is applied • Each profile usually stored as individual file Use Climate Data Portal for aggregation

  5. Problems with in-situ data • No standard APIs Use OPeNDAP Sequence datatype

  6. CDP service netCDF service netCDF profile netCDF profile Dapper architecture OPeNDAP Protocol (HTTP) OPeNDAP client Dapper CORBA (IIOP) Climate Data Portal Java netCDF library MySQL

  7. Climate Data Portal (CDP) • CORBA based • Metadata in database (MySQL) • Variables (measurements) in individual netCDF files Developed by Don Denbo and Willa Zhu

  8. OPeNDAP Sequence example (header): Dataset { Sequence { Float32 longitude; UInt32 time[tsize = 2]; Float32 latitude; Int32 _id; Sequence { Float32 depth; Float32 temperature; } profile; } location = { { -42.01, {77, 3850926208}, 53.3402, 3645, { { 3, 16.698 }, { 5, 16.71 }, …

  9. OPeNDAP (cont.) • Sequences accessible to clients via Java or C++ libraries • _id field in “outer” sequence allows unique identification of an individual profile

  10. Dapper design • Consists of configurable services • Climate Data Portal service • netCDF service • Services route OPeNDAP request • Implemented as Java servlet

  11. CDP service netCDF service netCDF profile netCDF profile Dapper architecture revisited OPeNDAP Protocol (HTTP) OPeNDAP client Dapper CORBA (IIOP) Climate Data Portal Java netCDF library MySQL

  12. CDP service example • HTTP request: http://foo.org/.../dset.cdp.dods?lon,lat, depth,_id,temp&lat>0&lat<20 • Dapper contacts CDP via CORBA IIOP • CDP accesses MySQL for metadata, appropriate profile files for data • CDP returns data to Dapper via CDP API • Dapper returns to client as OPeNDAP sequence

  13. CDP service performance • < 500 us/profile for coordinate info • 5 ms/profile to read one variable with z axis containing 1000 pts • netCDF library enhancements: • Parsing of netCDF header/attributes is optional • Use String(char[]) constructor instead of String(byte[]) constructor 2X speed up

  14. netCDF service • Individual profile files available as OPeNDAP Grid datatype • netCDF library enhancements: • True streaming server

  15. Future work • Stream data from CDP • Distribute netCDF file processing using scatter/gather patterns

  16. Availability http://www.epic.noaa.gov/dapper Joe.Sirott@noaa.gov

More Related