1 / 15

Archived Satellite Data Access Through CLASS

Archived Satellite Data Access Through CLASS. John Bates, Principal Scientist NOAA/NESDIS National Climatic Data Center. CLASS Goals. Goals: As an enterprise solution, CLASS will reduce anticipated cost growth associated with storing environmental datasets by:

osborn
Download Presentation

Archived Satellite Data Access Through CLASS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Archived Satellite Data AccessThrough CLASS John Bates, Principal Scientist NOAA/NESDIS National Climatic Data Center

  2. CLASS Goals Goals: As an enterprise solution, CLASS will reduce anticipated cost growth associated with storing environmental datasets by: • Providing common services for acquisition, security, and project management for the IT system supporting NOAA Archives • Consolidating stove-pipe, legacy archival storage* systems • Relieving data owners of archival storage-related system development and operations issues 2

  3. Direct Connectivity to: • ESPC- NOAA Environmental Satellite Processing Center • National Ice Center • NOAA Coast Watch • JPSS Interface Data Processing Segment (IDPS) Current CLASS Assets Functions Test and Integration Environment • Operations • Ingest • Storage (Disk & Tape) • Public Access Development Team Fairmont, WV CLASS NCDC, Asheville NC CLASS NGDC, Boulder CO CLASS NSOF, Suitland MD Satellite Landing Zone Development Environment Replication via NOAA Science Network(N-WAVE) Users Users • Key Capabilities: • Tape Library Capacity – 2- 10,000 tape robotic libraries with a total storage capacity of 15.5 Pb (LT04 Tapes/ Native) • Spinning Disk Capacity- 2.1 Pb (NSOF, NCDC, NGDC) • 10 Gb/sec Internal Network Backbone • Redhat Linux OS Server Count- 48 (576 processors) • 10 Gb/Sec WAN (N-Wave)

  4. Data Currently Archived in CLASS Represent 95% of the Archival Holdings by Volume (Single Node) 4

  5. CLASS Projected Ingest and Delivery

  6. Customer Groupings that Access CLASS CLASS Users by Domain TB 2012

  7. Data Access from CLASS Present CLASS Preservation Planning Data Management CONSUMER PRODUCER requests Ingest Access results Archival Storage Administration Search and find info Submit ad-hoc order Submit order Submit standing order Consumer Submit service request WWW.CLASS.NOAA.GOV 7

  8. Current CLASS Capabilities Some general notes on the design and capabilities of CLASS: • The NOAA CLASS archival storage system is not designed for real-time operations. • At least six hours elapse before data are ingested into CLASS from data producers, such as the NPP IDPS. • Users must create an account to be able to access the archived data. • Different levels of access services: ad hoc via the web, bulk orders and subscription • NEW - Direct download is offered for the most recent NPP data 8

  9. Levels of Access • Small Orders (under 100 files) • Use Search to obtain inventory listing • Select files to order by checking box on each row • Detailed metadata and browse images (POES and GOES) available • Large Orders (up to 1000 files) • Use Quick Search and Order • No inventory listing for file selection • Skip browsing and file level metadata • Delivery likely within 24-48 hours • Block Orders • Same as a Large Order but must be pre-approved for greater access • Generally 2 to 3 times more data per order • Need good bandwidth • Please contact CLASS Help Desk with subject line ‘Request Block Order Access’ • Block orders are split into batches (sub-orders) and processed one at a time 9

  10. Levels of Access (continuing) • Bulk Orders • Manual servicing by CLASS and NCDC for volumes in excess of several TB’s. • Usually covers many years (many TB of data) • Usually delivered on media (tape or drive) • Given lowest priority of service • Contact NCDC with your needs • Subscriptions • Provides automatic distribution of near real-time products • Products can be ftp ‘pushed’ or ‘pulled’ • Contact the CLASS Help Desk to obtain access (subscription link on home page) • Suomi-NPP Online Data Access • 100 TB of S-NPP data are online • operational S-NPP data including SDRs and EDRs (no RDRs) • Most recent 85 days available • Global (no subsetting) • Files are compressed and TAR’d into daily folders by instrument and datatype • ftp-npp.class.ngdc.noaa.gov/<Date>/<Family>/<Datatype>/ 10

  11. CLASS Search Types • Search types • Plain vanilla ‘Search’ • Search data by region, date/time, datatype, satellite id, and node (polar-orbiting) • Result page lists files (datasets) and links to file level metadata including browse images (GOES and POES data) • Quick Search and Order (for large and block orders) • Search data by region, date/time, datatype, satellite id and node • Skips inventory review and jumps to shopping cart page. • easier access to greater volumes of data per order (usually up to 1000 files) • Users can request “block order” for greater access • Advance searches possible • File name pattern searches • Orbit number – specific or range 11

  12. Ordering data from CLASS • Step 1: Register for a user id account at www.class.noaa.gov • – just your name, e-mail address and password • Step 2:Select from the drop down product menu and highlight dataset • Step 3: From the the search interface select geographic region, enter start/end dates and times, and select one or more data types • Step 4: Click on “search” for further selection and browsing or select “quick search and order” to order all files (large order type) See tutorial for ordering data from CLASS - link is in the news section 12

  13. Delivery options • File Transfer Protocol (FTP) • Small orders ready within 24 hours • Large orders 24 -72 hours (Keep in mind most of the data are pulled from tape) • Disseminated files remain on ftp server for 120 hours (5 days) • Physical media delivery • Increasing trend for media delivery due to higher volumes • NCDC continues to support this service • Media options are LTO tapes, External Disk Drives or DVDs • Charges applied to cover handling, media and shipping • Contact NCDC directly (ncdc.satorders@noaa.gov) for cost estimates and delivery schedules 13

  14. CLASS Evolution to an Enterprise Archival System Evolve the existing CLASS hardware and software infrastructure into a distributed, modular, service-oriented architecture Allow greater flexibility in supporting, to the maximum extent possible, not only large data arrays from satellite programs but also provide additional archival storage services for all of NOAA’s environmental data that has been approved for archive . Working with the NOAA National Data Centers, the new Enterprise Archival Storage architecture will consist of: • Generic ingest services for flexible data acquisition, • Flexible access services using existing community standard, open-source and emerging technologies such as cloud services, • Standardized metadata repository to support a variety of search and discovery services and • Long-term, secure archival storage and data management capabilities. .....continued…

  15. Thanks!Questions??? 15

More Related