1 / 12

Improved Access to RDA from the MSS

Improved Access to RDA from the MSS. OSD Executive Meeting April 28, 2009. Current situation. RDA, modes of access MSS to NCAR connected machines SAN to the networked world (35 TB only) Personalized requests to DSS staff, to the world

lilli
Download Presentation

Improved Access to RDA from the MSS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Improved Access to RDA from the MSS OSD Executive Meeting April 28, 2009

  2. Current situation • RDA, modes of access • MSS to NCAR connected machines • SAN to the networked world (35 TB only) • Personalized requests to DSS staff, to the world • Ad hoc, one-off efforts, no automation, less than optimal metrics, requires strong initiative for users • Quick drive through • Illustrate foundations for the new service

  3. Registration • To get RDA data a user MUST register • Metadata and searching are freely open • DB controlled • Levels of registration control access • Non-restricted data (first level) • Restricted access requires additional registration, e.g. agreement to special usage condition for JRA-25

  4. Access offerings

  5. MSS File Selection/Information Example: ECMWF, May 2008

  6. Script to Read MSS files Enable this process for users that don’t have NCAR MSS access – outside users

  7. RDA Dataset Request System (DSRQST) • Written project description and requirements • Highlight of requirements in three categories • Users • DSS Control • Systems

  8. User Requirements • Prioritized Service Levels • UCAR, U.S. Univ, U.S Gov. etc • Only qualified users see DSRQST interface • All data restrictions strictly enforced • Download options matched RDA online • Point and click • ‘wget’ script • Automatic notification when data is ready • Password protected download directories • Unreasonable requests – pop-up window including fault description and options • Most likely too much data

  9. DSS Control Requirements • System workflow settings • Staff inspection of request, approval and start • Fully automatic – staff still receives all workflow information • Implementable at the dataset level • Controlled initiation • Service initiated by responsible staff • Maximum download amount controlled at dataset level (~ default 20-100 GB) • ‘msrcp’ MSS files out to network attached systems • Set priority process levels, 1-10

  10. System Requirements • Daemon controls number of processes • Throttle impact on MSS • Monitor temporary storage area • Automatic scrubbing of staged data • Default 5 days, more by request • File transfers stops if temporary storage is near full • All metrics logged as “one-off” requests • Cross-check file availability from SAN • Don’t transfer from MSS is already available online • Corollary: share files between requests • Restart capability – mid-stream recovery • End-to-end control from DB settings

  11. Next steps • Coding is underway, datasets for prototype are identified • When – 1-2 months • Implement data processing on files – data product preparation • Possible – split request and use multiple processes • Utilize CISL high-end computing • Implement product write service onto the MSS for NCAR users • Short retention period

  12. End • Questions • Complex system • DB – MySQL • MSS read and write interactions • html • php • java script • perl

More Related