1 / 20

Jordan C. Alpert/NCEP/EMC jordan.alpert@noaa Glenn Rutledge/NCDC Jun Wang/SAIC

EMC. NOAA Operational Model Archive Distribution System NOMADS Toward an Operational Service Orientated Architecture. NCEP/Environmental Modeling Center. Jordan C. Alpert/NCEP/EMC jordan.alpert@noaa.gov Glenn Rutledge/NCDC Jun Wang/SAIC .

edythe
Download Presentation

Jordan C. Alpert/NCEP/EMC jordan.alpert@noaa Glenn Rutledge/NCDC Jun Wang/SAIC

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EMC NOAA Operational Model Archive Distribution SystemNOMADS Toward an Operational Service Orientated Architecture NCEP/Environmental Modeling Center Jordan C. Alpert/NCEP/EMC jordan.alpert@noaa.gov Glenn Rutledge/NCDC Jun Wang/SAIC 5th GO-ESSP Earth Science Portal Workshop, June 20, 2006 at Lawrence Livermore National Laboratory where the nation’s climate and weather services begin

  2. NOMADS Web Page • Completely • re-designed. • Uses “php” • Easier navigation • Better • Documentation • NOMADS is • renamed: • NOAA National…

  3. What is service oriented architecture? Vision “NOMADS is a distributed service oriented architecture, a system-of-systems integration based on using loosely coupled connections among independent systems to create a scalable, extensible, interoperable, reliable, and secure framework.” … reliable?

  4. What is NOMADS? (con’t) • A digital archive of NOAA’s operational weather models, and an “innovative data access philosophy to promote interoperable access across the geosciences” (BAMS, Rutledge et. al., 2006). • A Pilot project at NCEP that advances integration of real time model data. • NOMADS: An integrator of common web services infrastructure to support the discovery, access and transport of data (NOAA GEO-IDE Document).

  5. The NOAA Operational Model Archive and Distribution System NOMADS Goals • provide distributed access to models and associated data, • promote model evaluation and product development, • foster research within the geo-science • communities (ocean, weather, and climate) • to study multiple earth systems using • collections of distributed data, • develop institutional partnerships via distributed open technologies.

  6. Service oriented Goals (DMIT) Goals • To take advantage of internet technology opportunities. • Improve efficiency and reduce costs by bridging the barriers between existing, independent “stove pipe” systems. • Integrate the data management activities of NOAA projects. • Individual components retain responsibility and authority within the context of a systematic set of principles. • To Develop and adopt standards for metadata, data discovery and data transports, formats and protocols.

  7. The operational (reliability) problem To provide Operational Services where security, timeliness (time critical) and reliability are paramount. Operational public access services for public access to data, products and information services. Scientific services where efficient and flexible discovery and access to data sets is required. Commercial value-added services and user client applications.

  8. NOAA/NCEP Operational ftp Services For Model Data • Two centers (ftpprd… and operational official tgftp…) • Standard ftp servers • Load balance, Fail over, State of the art hardware and 24/7 support. • A little difficult to navigate* (e.g., file naming convention) • Entire files (0.25GB) must be downloaded when only small portions may be necessary. • Holdings not complete (0.5 degree GFS, NAM hourly, Ensembles…) Complete and “fast ftp” (Partial ftp transfers), a NOMADS user client application, for official ftp data set holdings is under discussion. *The understatement of the year

  9. What the user sees at the operational ftp site: Example of tgftp location for GFS model (first entry) files from June 15: ftp://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/MT.gfs_CY.00/RD.20060615/PT.grid_DF.gr1 An Excerpt from the list of 500 files, note name and size: File: fh.0030_tl.press_gr.onedeg 26562 KB 06/15/2006 File: fh.0030_tl.sflux 42981 KB 06/15/2006 Excerpt from the ftpprd inventory description: Inventory of File gfs.t00z.pgrbanl Model: GFS Cycle: 00 UTC Forecast: 0 HRS Number of Records: 267 Grid Identification : 3 Number Level/Layer Parameter Description 0001 1000 ISBL HGT 1000 hPa isobaric level Geopotential height gpm] 0002 975 ISBL HGT 975 hPa isobaric level Geopotential height [gpm] 0003 950 ISBL HGT 950 hPa isobaric level Geopotential height [gpm]

  10. Toward Operational services • We know how to do the hardware part. $ = load balance, fail over, 24/7, redundancy • What about users usage of the applications, e.g., DAP and GDS and others • Operationalise data flow to NCDC from NCEP. • GRIB2 with jpeg packing increases CPU resources needed. • Software to handle sub-seting and GRIB2 • GDS, OPeNDAP, and other applications need GRIB2 software stratagy. • Implement with partial http transfers and index files.

  11. Highest month: 4.3Tb w/3.8 million downloads – NCEP has ~1-5Tb /month User Statistics NCDC Only BAMS Paper

  12. Real Time (NCEP) nomad3 server • GDS/OPeN(DAP) DODS: typically 250,000 queries/day from the GDS log (already /2). • pdisp (“Great Displays”) 2694/day. • http access_log ~200,000 hits/day • ftp2u (fast ftp index download) over 5/11 – 6/15 8,623 repackaged GRIB downloads/day. Need better ways of evaluating use for good health and for comparison of servers.

  13. Real Time (NCEP) NOMADS Server • Users query files that are not present in tight loops. • During the 12 hours [00Z-12Z] on June 1, of the 371 different users who queried the httpd server error_log, the top 7 users with the most queries are listed: #queriesUser 1579 hawaii.edu 1616 abo.wanadoo.fr 1870 natpool.mwn.de 3293 accuweather.com 4039 saildocs.com 7864 hmg.inpg.fr 100465 labsolar.ufsc.br A solution: Check log file (error_log) user access rate. If the rate criteria (6/minute)is exceeded then place user’s IP address in IP tables to block access for that IP address for a period of time (10 minutes). Placing users in a “penalty box” when they are repeatedly accessing the server in tight loops is the same as users placing a “sleep 600” command or wait 10 minutes in their unix script for loop. This represents a throttle that can be applied to other situations such as many GDS queries when one is needed.

  14. Multiple paths to format independent data access The NOMADS System Design

  15. Search, Discovery, Access, and Analysis A metadata server, DIMES (Distributed Metadata Server) integrated with GDS to form MIDAS (Kafatos, Yang, Zhao) : Metadata Integrated Data Analysis Server • Content consistency between data server and metadata server. • Interactive metadata search and data access and analysis. • Potential for more effective and efficient searching. • Potential for better interfaces. • More seamless transition from data search to data access. • Uses XML to represent, store, retrieve and interoperate metadata with minimum semantic enforcement. • Contains a metadata model, XML query engine, web-based prototype interface. • Enables the application power of Grads and GDS server side data analysis.

  16. One can search by Time resolution (forecast time), Spatcial resolution, Search_Space (region), Search_Time (Cycle), and Text (variable and other Title text in the Metadata file.

  17. After selection of the Time and Space Box, the user can generate an “sdfopen” URL command including functionality, e.g, time or space averaging, for further processing or generate the DODS URL and run GrADS client script and interactive commands.

  18. Future Plans • Incorporate NOMADS functionality into CLASS (Comprehensive Large Array Stewardship system) • Enhance the NCEP real time servers at NCDC and place in NCDC operations. • Operationalise the data flow from NCEP to NCDC. • Existing Operational NOAA servers can be enhanced with NOMADS service orientated architecture and administrators. • Support working prototypes for proof of concept and continued development toward improved services.

More Related