1 / 7

GrADS-DODS Server An open-source tool for distributed data access and analysis

<?xml version="1.0" encoding="ISO-8859-1"?> <gds name="my data server" home="/"> <catalog temp_size_limit="1000"> <data> <dataset name = "test” file = "testdata/little_endian.ctl“> </data> </catalog>.

candide-workman
Download Presentation

GrADS-DODS Server An open-source tool for distributed data access and analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. <?xml version="1.0" encoding="ISO-8859-1"?> <gds name="my data server" home="/"> <catalog temp_size_limit="1000"> <data> <dataset name = "test” file = "testdata/little_endian.ctl“> </data> </catalog> ... GrADS-DODS ServerAn open-source tool for distributed data access and analysis Joe Wielgosz, Brian Doty, Jennifer AdamsCOLA/IGES - Calverton, MD http://www.iges.org/grads/gds OPeNDAP Working Conference - March 19, 2003

  2. GDS binary LAS netCDF web browser GrADS HDF4 GrADS GRIB Java servlet BUFR Ferret OPeNDAP Matlab OPeNDAP server IDL Unidata IDV comparisondata GDS ncBrowse and more... OPeNDAP server What the GDS can do servers clients Make GrADS-readable datasets - both gridded and in-situ - accessible across the network, to a diverse range of clients Perform server-sideanalysis and comparisons against other distributed datasets

  3. Core code refactoredAnagram - generic data server frameworkSwappable, reusable modulesDesigned for efficiency - streaming I/O XML-based configuration,with more flexibility in:Dataset loadingLoggingSecurityResource management Scales better to 1000's of datasetsOrganizes data catalog into directoriesFaster startup and smarter caching New data type support Station data - GrADS format and BUFR Remote OPeNDAP data Subsampling ("striding") for gridded data Improved web interfaceCustom links to help, home, dataset infoURL-based administration interface What's new since 1.1

  4. Administrator-friendly <gds> <catalog temp_size_limit="1000"> <data> <dataset name="test" file="testdata/big_endian.ctl" format="ctl" /><datadir name="/mnt/data1" suffix=".ctl"/> </data> </catalog> <log mode="rotate" frequency="monthly" file="log/gds.log" level="info" /> <grads> <invoker grads_bin="/users/joew/bin/grads"/> </grads> <servlet> <filter-abuse enabled="true" hits="1000" timeout="24" /> <filter-overload enabled="true" limit="20" /> <filter-analysis enabled="true" /> </servlet> <mapper> <service-admin enabled="true" auth="open-sesame" /> </mapper> <privilege_mgr default="public"> <ip_range mask="127.0.0.1" privilege="private" /> <privilege name="public"> <deny path="private_data" /> </privilege> <privilege name="private" /> </privilege_mgr> </gds> Install in minutes (really!)No root privileges neededCross-platform Java and ANSI C Stable and fast COLA public GDS currently handling >600k hits/month Easy to configureEdit one (simple) XML file, and make updates on-the-fly SecureRestrict dataset access & resource usage by IP address And more...Automatic scans for new datasets Detailed loggingGraceful handling of heavy loadsEasily integrated with Apache... Complete online documentation: http://www.iges.org/grads/gds/doc

  5. Some public GDS servers: (google on "grads dods server") • COLA Public Data Server: cola8.iges.org:9191 (soon to be wxdata.org) • COLA Monsoon Data Server: monsoondata.org • NOAA/CIRES CDC: www.cdc.noaa.gov/dods • FNMOC / GODAE: usgodae.org • NCEP: nomad2.ncep.noaa.gov • NASA / GSWP: voda.gsfc.nasa.gov:9090 • NASA / NSIPP: beta.gsfc.nasa.gov:9090 plus activity at centers in France, Britain (BADC), and Japan... In use, well-received • Positive response from: • COLA scientists • GrADS user community - research, corporate, hobbyists • NOAA/CIRES CDC (earliest adopters outside COLA)

  6. Continuing refinements to existing code • uncovering bugs (some suspected to still exist) • improving documentation where needed • addressing usability issues • Station data support • Finishing up BUFR I/O interface in GrADS • Adding daily station reports to COLA public server • Releasing station-data-enabled GrADS client • Working on interoperability with other archives and clients • Under consideration: • built-in metric collection • catalog server • login-based authentication • near-line data support • listening for ideas What to expect

  7. COLA thoughts on OPeNDAP • "DODS" vs "OPeNDAP" • With apologies, waiting for 4.0 - don't want to confuse user community • New protocol • I see DAP 4.0 as a chance to reduce complexity and increase robustness • We all have limited development resources - keep size of protocol proportional • Interoperability means common ground, not sum of all features • Anagram / Server-side Java • Java is an ideal server platform - stable, cross-platform, easy to code • Consider Anagram framework + Java-DODS as servlet counterpart to DAP C++ CGI framework? Slot in custom functionality to a standard core - good for gateways, access control / resource limits, web services, etc. • Analysis protocol • Increasing usage at COLA - perceived by scientists as extremely valuable • Time to think about interoperability? Complex issues involved • Modeler's needs • Grid / ESG II • Authentication important for collaborative projects • Can OPeNDAP provide a way for small shops to get Grid-enabled?

More Related