190 likes | 209 Views
Dapper and Argo. Joe Sirott PMEL/NOAA. What is Dapper?. Web server that provides distributed access to in-situ data via OPeNDAP protocol Clients include GrADS, Matlab, NcBrowse, Java Ocean Atlas Available data: PMEL EPIC database ARGO profiles NODC World Ocean Database 2001
E N D
Dapper and Argo Joe Sirott PMEL/NOAA
What is Dapper? • Web server that provides distributed access to in-situ data via OPeNDAP protocol • Clients include GrADS, Matlab, NcBrowse, Java Ocean Atlas • Available data: • PMEL EPIC database • ARGO profiles • NODC World Ocean Database 2001 • All components are open-source
What does Dapper do? • Efficiently aggregates thousands of in-situ data files (>2000000 for WODB2001) • Handles several in-situ data formats: • Argo GDAC netCDF • PMEL EPIC netCDF • COARDS compliant netCDF • Supports time series or profiles • Supports space/time queries • Based on HPCC funded Climate Data Portal (Soreide, Zhu, Denbo)
What is OPeNDAP? • OPeNDAP (Open source Project for a Network Data Access Protocol) is a protocol for remotely accessing scientific data • Uses HTTP as transport
Dapper OPeNDAP Example (space/time query) • Client HTTP request:http://foo.org/.../dset.cdp.dods?lat,temp,pres&lat>0&lat<20 • Dapper HTTP response:Dataset { Sequence { Float32 lat; Float64 _id; Sequence { Float32 pres,temp; } profile; } } location = { { -4.01,1, { {4.3,26.808}, {100,24.982}, …}, { -52, 2, {{8.3,3.77},{100,3.743},…}
Dapper OPeNDAP Example (single profile) • Client HTTP request:http://foo.org/.../dset.cdp.dods?lat,lon,time,profile&_id=20 • Dapper HTTP response:Dataset { Sequence { Float64 time; Float32 lat,lon; Sequence { Float32 pres,temp; } profile; } } location = { {1034140577999,-52.9,310.144}, {{8.3,3.77},{9.1,3.77},…}};
Dapper architectural overview • Uses relational database to store metadata and coordinate boundaries for each profile/time series file • Data maintained in original format • All data delivered as OPeNDAP sequences
CDP service netCDF service netCDF profile netCDF profile Database loader Dapper architecture OPeNDAP Protocol (HTTP) OPeNDAP clientDChart Dapper HTTP Server CORBA (IIOP) Climate Data Portal Java netCDF library MySQL
DChart • Browser based in-situ OPeNDAP client • Based on FY2004 HPCC funded project for shipboard display of cruise data • AJAX based user interface
DChart Design Goals • Highly responsive browser based user interface • Low learning curve • Rapid previewing and downloading of Dapper datasets • Highly interactive map for region selection • No Java applets or proprietary plug-ins
CDP service netCDF service netCDF profile netCDF profile Database loader Dapper architecture diagram OPeNDAP Protocol (HTTP) OPeNDAP clientDChart Dapper HTTP Server CORBA (IIOP) Climate Data Portal Java netCDF library MySQL
Web Browser XML/JavaScript (HTTP) OPeNDAP Protocol (HTTP) JavaScript DChart Servlet Dapper JFreeChartSGT OPeNDAPClient XMLEngine DChart architecture
CDFsync • Rapid synchronization of netCDF in-situ files over a network • Only compressed file differences transmitted across network • Based on rsync, but optimized for netCDF and file lists consisting of thousands or millions of files • Funded by HPCC 2004 (Charles Sun et al) • Available at http://www.epic.noaa.gov/epic/software/cdfsync/
Future plans • Constraints (QC flags, attributes) • Metadata display • Trajectory and section plots • Multiple plot windows • Save/restore Web app state
More info PMEL Dapper Server http://dapper.pmel.noaa.gov/dapper PMEL Dapper UI http://dapper.pmel.noaa.gov/dchart Downloads, propaganda http://www.epic.noaa.gov/epic/software/dapper/ Joe.Sirott@noaa.gov