240 likes | 566 Views
HYCOM Data Service www.hycom.org/dataserver. Ashwanth Srinivasan, (FSU) Steve Hankin (NOAA/PMEL) Contributors: Jon Callahan (Mazama Consulting) Roland Schweitzer (Weathertop Consulting), Ansley Manke (NOAA/PMEL) Jeremy Malczyk (UW/JISAO) Peter Cornillon (URI). An overview .
E N D
HYCOMDataServicewww.hycom.org/dataserver Ashwanth Srinivasan, (FSU) Steve Hankin (NOAA/PMEL) Contributors: Jon Callahan (Mazama Consulting) Roland Schweitzer (Weathertop Consulting), Ansley Manke (NOAA/PMEL) Jeremy Malczyk (UW/JISAO) Peter Cornillon (URI) An overview
HYCOM Data Service: History and Current Status • Data Service was launched in 2002 in Miami and served HYCOM Outputs via FTP, LAS and OPeNDAP servers • Started Serving Near Real Time Atlantic data in 2003 • Hosted about 3 TB of Data until Dec, 2006 in Miami. • Jan -2007 – service moved to FSU and hosted on a 100 TB SAN • Currently serving approximately 60 TB of data
New Hardware • 100 TB Fibre Channel SAN • Three 8 CPU machines • 32 GB RAM/machine • Software • Red Hat Linux • Red Hat cluster suite and Global File System • Apache Web Server • THREDDS • LAS Server • Vsftpd server
Operational Enhancements • All components (LAS,OPENDAP,FTP) co-located in the same physical machine. • Subset of files are uncompressed for faster access • In memory, decompression by using RAM scratch disks • High-availability architecture – minimal downtime during updates
Currently Available Datasets • Daily outputs from the real time Global HYCOM+NCODA analysis (Apr 2007 –current) • Global HYCOM+NCODA analysis (2004-2005 available now; 2005-2007 will be available shortly) • Four years of Global Non-Assimilative Simulation (2003-2007) • Near real-time 1/12 Atlantic Ocean prediction system output (June 2003 – Present) • Monthly mean 1/12 Pacific Ocean simulation output (1978-2003)
Global HYCOM Data Serving Challenges • Very large data/file sizes – each day of output is 12-19 G – consists of 26 (15+9) 2d and 3d variables (4500 x 3298 x 32 pts) • Curvilinear, staggered and hybrid vertical coordinates makes analysis and visualization on large files resource intensive and time consuming • Need to handle model products in non-standard native HYCOM format and in standard format like NetCDF (B.Cs are needed on native grids and in both NetCDF and HYCOM IO formats)
HYCOM Ocean Prediction SystemData Flow 50 G pulled to FSU ~ 2-3 hrs Conversion to NetCDF and re-gridding to Z levels Global run at NAVO daily – 5 day hindcast and 5 day forecast in curvilinear and hybrid coordinates THREDDS and LAS Update *Approximately 4-5 hrs of data processing daily
Means of Data Access • THREDDS Provides a dataset catalog and OPeNDAP access to all datasets. • Live Access Server Visualization and downloads in different formats, batch access etc. • OPeNDAP enabled clients Choice of common applications or user written programs to access data • FTP NetCDF files for each variable
HYCOM LAS: http://hycom.coaps.fsu.edu/las/ • Provides NetCDF data in both native HYCOM (staggered) and regular (non-staggered) grids • On-the-fly conversion to fixed Z levels (but limited to pre-configured Z levels) • tested HYCOM format IO as a product –but needs better integration • Full resolution global outputs not available currently via LAS
Hybrid-Z regridding fixed-Z native-Z
HYCOM specific Matlab tools from the OPeNDAP Developers • The Matlab GUI developed for the1/12o North Atlantic Model Runs has been significantly modified. • Access the GUI from: http://opendap.org/download/ml-toolbox.html • The OPeNDAP-IDL client is now robust and can be used to access HYCOM data (no GUI yet). • Many modifications to the netCDF-OPeNDAP libraries. • Server4 - supporting GridFTP is now available.
Access the OPeNDAP Matlab GUIs from the OPeNDAP download webpage http://opendap.org/download Or go directly to the GUI web page http://opendap.org/download/ml-toolbox.html
Community Collaborations • Working with John Caron at Unidata on testing the Forecast Aggregation server with large operational datasets • Building a HYCOM .[ab] format IO service provider (ISOP) to be incorporated into the JAVA NetCDF 2.2 library • Partnership with IOOS data management Caucus
Other Addition/Improvements • Forcing data • Reference observational data sets • A completely redesigned website is being planned along the lines of GODAE server • Searchable mailing list, FAQ’s • Links to other data repositories