1 / 20

Managing processing and storage for MODIS and OMI

Managing processing and storage for MODIS and OMI. Edward Masuoka Terrestrial Information Systems Branch. What our data systems support. Integration of science team software Development of Level 1 and Level 3 products Processing science products Product Quality Assessment

melvinbrown
Download Presentation

Managing processing and storage for MODIS and OMI

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Managing processing and storage for MODIS and OMI Edward Masuoka Terrestrial Information Systems Branch

  2. What our data systems support • Integration of science team software • Development of Level 1 and Level 3 products • Processing science products • Product Quality Assessment • Archiving/distributing data products and imagery

  3. In ‘98, 47TB was a lot of storage

  4. 4 Petabytes in 4 racks in 2010

  5. 3 SGI systems in 1995> 700 Linux servers in 2010

  6. MODIS and OMI Hardwareinformation in one place

  7. Tables updated daily

  8. System Administration Tools • Depot http://www.cs.cmu.edu/~help/unix_linux/software_collections/local_depot.html Manages software under /usr/local • SATE (System Administrator Tool Environment) Handles all user accounts, Integrated with NAMS Property database (location, value description) • Problem Queue Email-based problem tracking Automated notifications from h/w are sent here • System Administrator Wiki (MediaWiki) • Procedures for managing systems and storage

  9. Development Team Tools/Process • Subversion – Configuration management • Bugzilla – Software Bug Tracking • PCR (PGE Change Request) Process Science S/W PGE (Product Generation Executive) delivered unit test (specified by developer)  science test (defined by science discipline lead)  science test results reviewed by Quality Assessment Team  Science disciplines and Science Team leader review  If approved put into production

  10. MODAPS H/W Architecture Distribution Database Server Production Database Server Black Diamond Ethernet Switch Web, ftp and proxy servers Head Nodes (Ingest and Staging Server) Outside Network Archive Servers Fast Storage High Volume Storage Compute Servers

  11. Monitoring performance with Ganglia

  12. Activity on one Cluster

  13. Moving processing resources between Clusters • Operations monitors level of use on clusters with Ganglia • Extreme Networks Black Diamond switches enable compute servers to be shifted between clusters supporting activities as needed to meet demand in different areas • Software on compute servers can be quickly reconfigured via Depot and sync from Subversion repository

  14. Databases are the foundations for processing and distribution • Processing databases track files on production systems and processing jobs • LAADS database supports search/order and custom product generation • Processing and distribution are decoupled • Searches don’t impact production rates or delivery to end-users via ftp push • Processing at 100x doesn’t impact product searches or post-processing on LAADS

  15. Current Evaluations • Running iRODS + FUSE as replacement for NFS for presenting the all_data tree (/Collection /Mission/Product/…/granule) to ftp users • Prevents ftp from hanging if storage node is down • Running PGEs in the Nebula Cloud • Need to understand cost with respect to computing, storage and network bandwidth • May be used to handle peak demand

  16. Finish in 2011 • Migrate all servers to CentOS Linux • Migrate LAADS database to PostgreSQL • Databases migrated for most production systems • Single set of security plans for MODIS and OMI

  17. Quality Assessment Teamhttp://landweb.nascom.nasa.gov/cgi-bin/QA_WWW/newPage.cgi

  18. Quality Assessment • Global browse • Images of Daily and Multi-day products • Golden Tiles (9 tiles over key land cover types) • Browse images • Time Series Plots allow comparisons between different years and reprocessing campaigns • Tools written in C interfaced to ENVI facilitate manipulation and assessment of MODIS standard products

  19. Acronyms ENVI GIS and image processing COTS s/w FUSE File System in User Space iRODS integrated Rule-Oriented Data System NFS Network File System LAADS Level 1 and Atmosphere Archive and Distribution System MODIS Moderate-resolution Imaging Spectroradiometer OMI Ozone Monitoring Instrument

More Related