1 / 21

GDS – The GrADS/DODS Server

GDS – The GrADS/DODS Server. Jim Kinter Center for Ocean-Land-Atmosphere Studies (COLA) NVODS Workshop 10 September 2003. GrADS-DODS Server (GDS)*. GrADS. Joe Wielgosz, Brian Doty, Jennifer Adams James Gallagher, Daniel Halloway *(server-side integration of GrADS and DODS - now OpeNDAP).

ross-barton
Download Presentation

GDS – The GrADS/DODS Server

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GDS – The GrADS/DODS Server Jim Kinter Center for Ocean-Land-Atmosphere Studies (COLA) NVODS Workshop 10 September 2003

  2. GrADS-DODS Server (GDS)* GrADS Joe Wielgosz, Brian Doty, Jennifer Adams James Gallagher, Daniel Halloway *(server-side integration of GrADS and DODS - now OpeNDAP) Jennifer Adams, Reinhard Budich, Luigi Calori, Brian Doty, Wesley Ebisuzaki, Mike Fiorino, Tom Holt, Don Hooper, Jim Kinter, Steve Lord, Gary Love, Karin Meier, Matt Munnich, Uwe Schulzweida, Arlindo da Silva, Michael Timlin, Pedro Tsai, Brian Wilkinson, Katja Winger (and others)

  3. integrated analytical quantitative intuitive (for domain scientists) format-independent data type-independent e.g. grids, stations etc. supporting rapid, domain-relevant subsetting Ultimate Goal • Data Interoperability • Data Distribution • Distributed Analysis • that is

  4. Grid Analysis and Display System Visualizing Maps, Charts, Animations Interactive Quantitative INTEGRATED USER INTERFACE Analyzing Accessing, Subsetting 4D Grids In Situ Obs Images { Expressions, Functions of Original Variables General slices of User Definable, Extensible Arbitrary Domains Optimized for Typical Geophysical Queries

  5. GrADS – A Tool for Geophysics • “Natural” user interfacefor scientific computations, and graphical production • Used at O(102) laboratories worldwide • Used by over O(103) scientists worldwide • E.g., 2002 J. Climate - Over ½ of all figures (and computations?) produced using GrADS • Handles many geophysical data formatsin “native” mode • Widely used for analysis and display of data from the National Weather Service, other WMO sources

  6. GrADS Analysis Model • ENABLES VERY SOPHISTICATED ANALYSIS TASKS • IN A HIGHLY ENCAPSULATED WAY • Scientists only need to specify: • dimension constraint • list of data sets • analysis expression • This approach to geophysical data analysis, despite its apparent simplicity, is extremely powerful.

  7. GDS binary LAS netCDF web browser GrADS HDF4 GrADS GRIB Java servlet BUFR Ferret OPeNDAP Matlab OPeNDAP server IDL Unidata IDV comparisondata GDS ncBrowse and more... OPeNDAP server What the GDS can do servers clients Make GrADS-readable datasets - both gridded and in-situ - accessible across the network, to a diverse range of clients Perform server-sideanalysis and comparisons against other distributed datasets

  8. Data Interoperability Example: Data from two Servers sdfopen http://cola8... set gxout shaded set time jul1980 d p sdfopen http://cdc... set gxout contour d prate.2*86400*31

  9. Example: Analysis at the Server sdfopen http://cola8.iges.org:9090/dods/_expr_{ssta,z5a} {tmave(maskout(aave(... }{-180:0,0:90,500:500, jan1950,dec1990} set gxout shaded display result

  10. Some public GDS servers: (google on "grads dods server") • COLA Public Data Server: cola8.iges.org:9191 COLA Monsoon Data Server: monsoondata.org • NOAA/CIRES CDC: www.cdc.noaa.gov/dods • FNMOC / GODAE: usgodae.org • NCEP (NOMADS): nomad2.ncep.noaa.gov • GFDL (NOMADS): data1.gfdl.noaa.gov • NASA / GSWP: voda.gsfc.nasa.gov:9090 • NASA / LIS: lis1.sci.gsfc.nasa.gov:9090 • NASA / NSIPP: beta.gsfc.nasa.gov:9090 • IPRC: aprdc.soest.hawaii.edu:9090 • CSAG (South Africa) www.csag.uct.ac.za:9090 plus activity at centers in France, Britain (BADC), Italy (CINECA) and Japan... GDS in production, well-received • Positive response from: • COLA scientists • GrADS user community - research, corporate, hobbyists • NOAA/CIRES CDC (earliest adopters outside COLA)

  11. incentive? how? need more robot-friendly content on top page? • COLA GDS User Categories • Report from Jennifer Adams (2 years of statistics): • Users who have automated their access • Users who come regularly, but not automatically • Project-oriented users who access intensely • but only in the short-term • 4. Users who download data and treat GDS • as a subsetting FTP server • (this may be a subset of category 3) • 5. Casual/curious users who are just looking, not • downloading • 6. Robots (e.g., google; these don’t really count, • even though they are "unique IPs")

  12. 100 GB Bytes on All COLA GDS 10 GB 1 GB Jan2002 Jul2002 Jan2003 Jul2003 106 105 Hits on All COLA GDS 104 Jan2002 Jul2002 Jan2003 Jul2003

  13. NCEP Global Weather Forecasts Desktop Weather Forecasting NCEP COLA Global Weather Forecasts GrADS-DODS Server Region-Specific Lateral BCs PILOT PROJECT WITH US NWS IN SOUTH AFRICA, VIETNAM WWW PC-Based Regional NWP

  14. Core code refactoredAnagram - generic data server frameworkSwappable, reusable modulesDesigned for efficiency - streaming I/O XML-based configuration,with more flexibility in:Dataset loadingLoggingSecurityResource management Scales better to 1000's of datasetsOrganizes data catalog into directoriesFaster startup and smarter caching New data type support Station data - GrADS format and BUFR Remote OPeNDAP data Subsampling ("striding") for gridded data Improved web interfaceCustom links to help, home, dataset infoURL-based administration interface What's new in 1.2

  15. GDS – What’s new and interesting for NVODS • GrADS 1.9 • beta release by 10/31/2003; production by 12/31/2003 • much more robust support for in situ data • DTYPE station via OpeNDAP • DTYPE BUFR (WMO station data format) • Handling GRIB-2 (WMO gridded data format) • new interface for netCDF/HDF for non-COARDS-compliant data sets (via GrADS descriptor file) • GDS 1.2 – Anagram • framework for building servers • set of reusable classes • documented on www.iges.org/grads/gds • white paper in preparation (Joe Wielgosz)

  16. <gds> <catalog temp_size_limit="1000"> <data> <dataset name="test" file="testdata/big_endian.ctl" format="ctl" /><datadir name="/mnt/data1" suffix=".ctl"/> </data> </catalog> <log mode="rotate" frequency="monthly" file="log/gds.log" level="info" /> <grads> <invoker grads_bin="/users/joew/bin/grads"/> </grads> <servlet> <filter-abuse enabled="true" hits="1000" timeout="24" /> <filter-overload enabled="true" limit="20" /> <filter-analysis enabled="true" /> </servlet> <mapper> <service-admin enabled="true" auth="open-sesame" /> </mapper> <privilege_mgr default="public"> <ip_range mask="127.0.0.1" privilege="private" /> <privilege name="public"> <deny path="private_data" /> </privilege> <privilege name="private" /> </privilege_mgr> </gds> Stable and fast COLA public GDS currently handling > 1.5 million hits/month Easy to configureEdit one (simple) XML file, and make updates on-the-fly SecureRestrict dataset access & resource usage by IP address And more...Automatic scans for new datasets Detailed loggingGraceful handling of heavy loadsEasily integrated with Apache... • Administrator-friendly Install in minutes (really!)No root privileges neededCross-platform Java and ANSI C Complete online documentation: http://www.iges.org/grads/gds/doc

  17. GDS Enables … • Sharing data: Enterprise-wide; Internet-wide --- data-format independent • Data interoperability: Consistent metadata for many data types • Distributed analysis:Saves scientists’ time*, reduces network load; improves interactivity • Automation of analysis techniques: Analysis techniques can be captured in the form of scripts and provided on server and/or client “Data-portal changed my life.” – Ben Kirtman (COLA)

More Related