1 / 59

Ocean PEATE

Ocean PEATE. Fred Patt. Introduction Design Overview Design Description Systems Engineering Closing Remarks. Agenda. SDS Data Depository and Distribution Element (SD3E) Ocean Product Evaluation and Analysis Tool Element (PEATE) Land Product Evaluation and Analysis Tool Element (PEATE)

levi
Download Presentation

Ocean PEATE

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ocean PEATE Fred Patt

  2. Introduction Design Overview Design Description Systems Engineering Closing Remarks Agenda • SDS Data Depository and Distribution Element (SD3E) • Ocean Product Evaluation and Analysis Tool Element (PEATE) • Land Product Evaluation and Analysis Tool Element (PEATE) • Project Science Office Element (PSOE) • NPP Instrument Calibration Support Element (NICSE) • Atmosphere Product Evaluation and Analysis Tool Element (PEATE) • Ozone Product Evaluation and Analysis Tool Element (PEATE) • Sounder Product Evaluation and Analysis Tool Element (PEATE) • Integration and Test System Element (I&TSE)

  3. NICSE I&TSE Ozone SD3E Sounder Atmosphere Land Ocean PSOE Ocean PEATE Agenda • Overview • Interfaces and Design • Gap Analysis • Schedule

  4. Element Overview • Acquire VIIRS RDRs, SDRs, and Ocean EDRs from the SD3E and ADS/CLASS • Assess the quality of the NPP Ocean EDRs for accomplishing NASA’s climate research requirements • Provide suggested algorithm improvements to the IDPS via the Project Science Working Group (PSWG) • Process selected data subsets in support of Evaluation and Validation activities

  5. Ocean PEATE Requirements (1 of 2) • Acquire and ingest xDRs from the SD3E (Req 3.4.1.1 to 3.4.1.5) • VIIRS RDRs • VIIRS SDRs • OCC EDRs • SST EDRs • VIIRS IPs • Acquire and ingest xDRs from ADS/CLASS (Req 3.4.1.6 to 3.4.1.12) • Acquire and ingest NPP ancillary data sets (3.4.1.13) • Acquire and ingest prelaunch data sets (Req 3.4.1.14) • Catalog and manage NPP data sets (Req 3.4.2.1) • Support distribution of xDRs to VIIRS Ocean Science Team (VOST) (Req 3.4.5.1 to 3.4.5.5)

  6. Ocean PEATE Requirements (2 of 2) • Support evaluation processing of VIIRS data using IDPS operational code: SDRs, OCC EDRs, SST EDRs (Req 3.4.4.1, 3.4.4.5, 3.4.4.8, 3.4.4.9, 3.4.4.14 to 3.4.4.18) • Support VIIRS calibration analysis (Req 3.4.4.4, 3.4.4.9, 3.4.4.23 and 3.4.4.24) • Support evaluation of VIIRS Ocean EDRs by VOST (Req 3.4.4.10 to 3.4.4.13, 3.4.8.1 & 3.4.8.2) • Perform matchups of VIIRS data with SeaBASS data • Support cross-comparison of VIIRS ocean data with concurrent sensor data sets • Support cross-comparison of VIIRS ocean data with climatological data sets • Support cross-comparion of VIIRS ocean data with alternative algorithms and products • Support internal consistency evaluation of VIIRS ocean data

  7. Matchup Analysis • Ocean data granules in ODPS catalog are automatically matched with in situ data • SeaWiFS Bio-optical Archive and Storage System (SeaBASS) stores and manages in situ holdings from field programs and supported investigators. • Ocean staff acquire, QC and analyze new data samples • Over 300,000 in situ samples stored

  8. Sensor Cross-Comparisons • Level-3 parameters (e.g., nLw) compared for common spectral bands • Common bins extracted and compared over the period of overlap between the sensors • Comparisons are performed globally (deep water, clear water, coastal), zonally and for specified regions.

  9. Internal Consistency Analysis • Global averages from successive years are overplotted to determine interannual repeatability.

  10. Assumptions – Role of I&TSE • The Ocean PEATE will rely upon the SDS-provided I&TSE (mini-IDPS) to serve as the testbed for algorithm evaluation and potential improvements and will not attempt to duplicate efforts by running the operational code within the PEATE environment. • The staff of the I&TSE will have the ability to modify any operational code as per Ocean PEATE specification and run the code within the operational environment against a set of PEATE-specified test data sets. • The I&TSE will receive and install the most current version of the code and executable programs for all operational IDPS software required to produce NASA-evaluated EDRs, starting from RDRs. • The I&TSE will maintain all processing code under configuration control. The PEATEs will have access to the configuration controlled code.

  11. Ocean PEATE Software Support • The Ocean PEATE will maintain a working knowledge of the VIIRS SDR code and the EDR code for the Ocean Color and SST EDRs. • The Ocean PEATE will maintain an understanding of the relevant IP characteristics, but will (initially) assume that the IPs are properly generated. • The Ocean PEATE will design changes to the code in the I&TSE for the purpose of algorithm improvement or problem resolution. • The Ocean PEATE will develop appropriate test cases and request runs to verify and evaluate the changes. • The Ocean PEATE will provide recommended changes to the PSG, including a description of the proposed change, effect on the EDR performance, and the evaluation of the test runs.

  12. Ocean PEATE Product Improvement • The Ocean PEATE will evaluate the means (algorithms, ancillary data sources, etc.) to produce the best-quality Ocean Color and SST products from VIIRS data. • If approved by the PSG and NASA HQ, the Ocean PEATE will produce and evaluate these products alongside the official EDRs.

  13. IDPS VIIRS Ocean EDR Data Flow Processing Module VIIRS Product VIIRS RDR Dynamic Ancillary Data Previous VIIRS Gridded Products Static Ancillary Data Previous VIIRS Gridded Products NAAPS TOD MODIS Land/Water Mask NCEP Geopotential Height Ancillary Files Ancillary Files DEM NDT

  14. IDPS VIIRS Ocean EDR Data Flow VIIRS Geolocation VIIRS SDR Processing Module VIIRS_SDR_01 RDR Decompression VIIRS_GEO_01 Geolocation VIIRS_SDR.MOD 750m SDR VIIRS_SDR.IM 375m SDR VIIRS SDR VIIRS Product VIIRS RDR Dynamic Ancillary Data Previous VIIRS Gridded Products Static Ancillary Data Previous VIIRS Gridded Products NAAPS TOD MODIS Land/Water Mask NCEP Geopotential Height Ancillary Files Ancillary Files DEM NDT

  15. IDPS VIIRS Ocean EDR Data Flow VIIRS Geolocation VIIRS SDR Processing Module VIIRS_SDR_01 RDR Decompression VIIRS_GEO_01 Geolocation VIIRS_SDR.MOD 750m SDR VIIRS_SDR.IM 375m SDR VIIRS SDR VIIRS Product VIIRS RDR Dynamic Ancillary Data Previous VIIRS Gridded Products Static Ancillary Data Previous VIIRS Gridded Products VIIRS_GD_08 750m Granulation NAAPS TOD MODIS Land/Water Mask VIIRS_GD_25 NAAPS Granulation NCEP Geopotential Height VIIRS_GD_27 L/W Mask Granulation Ancillary Files Ancillary Files VIIRS_GD_11 Ancillary Profile VIIRS_GD_09 GFS Granulation ALL_GD_01 Time Interpolation VIIRS_GD_28 Surface Pressure Adjustment DEM VIIRS_GD_12 Bathymetry Granulation NDT VIIRS_GD_13 Temperature Granulation

  16. IDPS VIIRS Ocean EDR Data Flow VIIRS Geolocation VIIRS SDR Processing Module VIIRS_SDR_01 RDR Decompression VIIRS_GEO_01 Geolocation VIIRS_SDR.MOD 750m SDR VIIRS_SDR.IM 375m SDR VIIRS SDR VIIRS Product VIIRS RDR Dynamic Ancillary Data Previous VIIRS Gridded Products Static Ancillary Data Previous VIIRS Gridded Products VIIRS_GD_08 750m Granulation NAAPS TOD MODIS Land/Water Mask VIIRS_GD_25 NAAPS Granulation VIIRS_LN_06 Active Fires VIIRS_CM_01 Cloud Mask NCEP Geopotential Height VIIRS_GD_27 L/W Mask Granulation Ancillary Files Ancillary Files VIIRS_GD_11 Ancillary Profile VIIRS_GD_09 GFS Granulation ALL_GD_01 Time Interpolation VIIRS_GD_28 Surface Pressure Adjustment DEM VIIRS_GD_12 Bathymetry Granulation NDT VIIRS_GD_13 Temperature Granulation

  17. IDPS VIIRS Ocean EDR Data Flow VIIRS Geolocation VIIRS SDR Processing Module VIIRS_SDR_01 RDR Decompression VIIRS_GEO_01 Geolocation VIIRS_SDR.MOD 750m SDR VIIRS_SDR.IM 375m SDR VIIRS SDR VIIRS Product VIIRS_SN_02 Ice Quality VIIRS RDR Dynamic Ancillary Data Previous VIIRS Gridded Products Static Ancillary Data VIIRS_CL_01 Cloud Optical Properties Previous VIIRS Gridded Products VIIRS_GD_08 750m Granulation VIIRS_ST_02 Surface Temp NAAPS TOD VIIRS_SN_03 Ice Concentration MODIS Land/Water Mask VIIRS_GD_25 NAAPS Granulation VIIRS_LN_06 Active Fires VIIRS_CM_01 Cloud Mask NCEP Geopotential Height VIIRS_GD_27 L/W Mask Granulation Ancillary Files Ancillary Files VIIRS_ST_01 Sea Surface Temperature SST EDR VIIRS_GD_11 Ancillary Profile VIIRS_GD_09 GFS Granulation VIIRS_AR_01 Aerosol Type ALL_GD_01 Time Interpolation VIIRS_GD_28 Surface Pressure Adjustment VIIRS_OC_01 Ocean Color / Chlorophyll DEM VIIRS_GD_12 Bathymetry Granulation OCC EDR NDT VIIRS_GD_13 Temperature Granulation

  18. ODPS MODIS Ocean Product Data Flow Processing Module MOD_PR01 Level-0 to 1A MODIS Product Dynamic Ancillary Data MODIS Level-0 MODIS Level-1A Static Ancillary Data MODIS 1 km Level-1B MOD_PR03 Geolocation MOD_PR02 Level-1A to 1B MODIS SST Platform ATTEPH Data MODIS Geolocation MSl12 Level-1B to 2 MODIS Ocean Color MET Ancillary Data Ozone Ancillary Files Land/Water Mask

  19. VOST CARS SD3E CLASS(ADS) Casa-NOSA AncillaryDataProviders I&TSE Ocean ScienceCommunity NICSE PSOE Ocean PEATE Interface Diagram Analysis Results, Proposed Algorithm Updates xDRs, IPs, Ancillary Data Management Direction xDRs, IPs, Ancillary Data (if unavailable from SD3E) OceanPEATE Pre-flight Algorithms, Data, Info Software, Data Ancillary Data xDR Eval. Results, Algorithm Updates Calibration Updates and Evaluations Interaction In Situ Data Alternative Algorithm Results Algorithm Updates, Test Requests & Results

  20. Ocean PEATE External Interfaces (1 of 2) • SDS Science Data Distribution and Depository Element (SD3E) • Provides NRT access to raw data • Primary source of RDRs • Provides selected SDRs and EDRs • SDS Integration and Test System Element (I&TSE) • Build and test updates to operational code in mini-IDPS • Run tests on selected data per request of PEATE • Archive Distribution Segment (ADS) • Primary source for archived data • xDRs, IPs, Ancillary Data, Operational Algorithm/Source Code and Calibration Products • Ancillary Data Providers (ADP) • Provides alternate ancillary data sets (e.g., ozone, meteorological data sets) • CasaNOSA • Serves as the NPP pre-flight repository of Government held data for distribution to Government user teams • Place to acquire pre-launch NPP algorithms and supported data files

  21. Ocean PEATE External Interfaces • NASA VIIRS Ocean Science Team (VOST) • Coordinate activities with PEATEs and PSOE on xDR and recommended algorithm improvements. Supports Independent Calibration Validation Activities • NPP Instrument Calibration Support Element (NICSE) • Provides alternative calibration LUTs and recommended improvements to calibration algorithms • PEATE provides results of LUT and algorithm tests • Project Science Office Element (PSOE) • Provides management direction • Accepts algorithm update recommendations • NASA Ocean CARS • Provides in situ data and results of alternative Ocean algorithms using VIIRS data • Ocean Science Community • Relies on Ocean PEATE to provide evaluation products and results

  22. Ocean PEATE Design • The NPP Ocean PEATE will be implemented within the framework and facilities of the current NASA Ocean Data Processing System (ODPS) • This system has been successfully supporting operational, satellite-based remote-sensing missions since 1996, and its capabilities continue to evolve and expand to meet the demands and challenges of future missions.

  23. Ocean PEATE Design Overview • Fully automated, distributed data system for acquiring, processing, archiving, and distributing scientific data • Highly scalable • Easily adaptable to support multiple concurrent missions • Graphical user interfaces for controlling and monitoring system functions and activity • Non-platform specific

  24. Ocean PEATE Design Philosophy • Building-Block approach • Programs are usually small and do one thing well • Programs are less complex and subsequently easy to maintain • Promotes reuse • Programs loosely coupled so testing and production can be done in the same environment • Adopt basic standards • ANSI, POSIX, C9x • Use existing technology when possible • Exit statuses indicate successful or failure conditions

  25. Components and Subsystems

  26. Components and Subsystems (1 of 2) • RDBMS is the primary element that manages all system activity • Generic core databases support system infrastructure and non-mission-specific functions • Mission databases catalogue products and house mission-specific data and procedures • High level of reuse possible for similar missions; e.g., MODIS Aqua/Terra, SeaWiFS, and OCTS are all ocean-color missions and have similar product suites and requirements

  27. Components and Subsystems (2 of 2) • Relational Database Management System (RDBMS) supports all of the system components (subsystems) • Scheduler is the primary controlling module within the system, supporting both time- and event-based tasks • Other subsystems are independent modules, yet rely on the Scheduler for some their functions • Scheduler • Visual Database Cookbook (VDC) • Archive Device Manager (ADM) • Data acquisition and ingest • Level-3 Scheduler • File migration and management • Data distribution

  28. Architecture: Hardware • Processing Servers • Intel-based dual Xeon / AMD-based dual Opteron • 8 GB RAM • Five 72 GB SCSI drives • Storage Servers • Intel-based P4 / AMD-based single Opteron • 1 GB / 2 GB RAM • 1.5 TB IDE RAID 5 (3ware) / 5.5 TB SATA RAID 6 (Areca) • 2 hot spare drives • Database Server • Sun V880 • 8-16 GB RAM • 6-12 70 GB SCSI HDD

  29. Distribution Servers (ftp) 6 storage nodes 7.2 TB Distribution Servers (web)1 large server – 3 TB 68 storage nodes 120 TB Ingest Servers 2 SeaSpace ground stations 5 storage nodes 6 TB Cal/Val & QC Systems Processing Cluster 34 processing nodes 1.5 TB Extreme Networks Black Diamond 6816 Gigabit Ethernet switch Mission Operations Systems Database Server 1 large server 876 GB Development Servers 1 processing node 2 storage nodes 2.4 TB Testing Cluster 13 test nodes 2 TB Backup Servers 1 large server 876 GB+2 TB tape 5 storage nodes 6 TB User Desktops Network Support Systems Ocean PEATE Data Processing System Current Components

  30. ODPS COTS and Freeware • Linux OS (CentOS 4.x) • Solaris OS • Sybase RDBMS • Subversion (source code management) • Pro-active DBA • Interactive Data Language (IDL) • Generic Mapping Tool (GMT) • Netpbm (graphic image toolkit) • HDF5 Library • Languages: C, PERL, SQL

  31. Ocean PEATE Gap Analysis • Acquire, ingest and catalog NPP VIIRS data products: RDRs, SDRs and Ocean EDRs (Data Acquisition & Ingest and Data Cataloging). • Process Ocean EDRs (SST and OCC) to Level-3 to support data product and algorithm evaluations (Level-3 Scheduler, VDC and Level-3 binner). • Perform VIIRS EDR matchups with Ocean CARS in situ data (extract code) • Process SDRs using best current algorithms to produce Ocean Color and SST products (MSl12). • Produce VIIRS proxy data using VOST-developed software (Schedule and VDC). • Support browse and distribution of data products for team members (Data Distribution).

  32. Basis of Estimate (1 of 2) • Acquire, ingest and catalog NPP VIIRS data products: • 200 LOC (combined UNIX shell, SQL, C) per new product, for each of the four VIIRS products • Process Ocean EDRs (SST and OCC) to Level-3: • 300 C LOC to add VIIRS Ocean EDR input to existing binning software (L2bin) • 300 SQL LOC per temporal range (10 ranges total) • 200 shell LOC for all ranges • Perform VIIRS EDR matchups: • 100 C LOC + 10 PERL LOC to add VIIRS Ocean EDR input to existing extraction software

  33. Basis of Estimate (2 of 2) • Process SDRs using best current algorithms: • 1000 C LOC to add VIIRS SDR input to existing Level-2 processing software (MSl12) • Produce VIIRS proxy data: • 100 shell LOC for each processing stage • Support browse and distribution of data products : • 3000 C LOC to implement browse image generation • 100 PERL LOC to add VIIRS products to existing browse and order web site

  34. Data Storage Estimate Assumption: (*) Long-term storage is sized for 100% of RDRs and 10% of SDRs and EDRs; packaged without geolocation.

  35. Ocean PEATERequirement Allocation (1 of 3)

  36. Ocean PEATERequirement Allocation (2 of 3)

  37. Ocean PEATERequirement Allocation (3 of 3)

  38. Initial Capability (L-18 months) All interfaces fully implemented and tested Verify initial versions of operational code ported and running in I&TSE L-3 product code developed and tested Data storage capacity for TBD months Initial test products generated for review by VIIRS Ocean Science Team Full Mission Capability (L-12 months) Routine exercise of interfaces to acquire proxy, surrogate (Aqua?) and/or simulated data Verify pre-launch version of operational code running in I&TSE Browse and distribution capability developed and tested Test products routinely acquired based on simulated data and posted for access by VIIRS Ocean Science Team Data storage for TBD years Schedule

  39. Ocean PEATERequirement Mapping • 100% of requirements (57) allocated to design

  40. Ocean PEATERequirement Implementation • 98% of requirements (56) implemented by full mission capability; add additional storage capacity before launch

  41. Conclusion • Ocean PEATE will meet all requirements Next Up: Land PEATE – Ed Masuoka NPP_SDS_Land.ppt

  42. Backup Slides

  43. RDBMS Components and Subsystems: RDBMS Goal: Isolate RDBMS from system software To use a different RDBMS vendor, swap in a new Database Services Layer RDBMS Vendor Library Module Vendor Client Library Database Services Layer Perl DBI Module C Interface Functions Perl Scripts C Programs

  44. Scheduler Subsystems: Scheduler • C program with supporting database procedures • Runs in a daemon-like state • Primary system element responsible for coordinating most of the system activity • Monitors task records in a to-do list database table • Runs tasks according to defined task attributes • Standard job-shell interface allows new programs to be quickly adapted for Scheduler control

  45. VDC Subsystems: VDC • Highly scalable, distributed infrastructure for concurrent processing of serial streams (e.g., L0  L1A  L1B  L2) • Suite of C programs with supporting database procedures • Uses recipes to encapsulate data-specific processing schemes, parameters, and pre-processing rules • Virtual Processing Units (VPUs) serve as distinct distributed processing resources • VPUs dynamically allocated based on available time and current OS load • Comprehensive, user-defined processing priorities

  46. VDC: Ancillary Data Stager VDC: Ancillary Data Stager • Runs in a daemon-like state • Monitors entries in the processing queue and runs the ancillary-select procedure for each entry’s recipe • Updates queue-entry status when ancillary data are available • Governed by currently configured processing priorities

  47. VDC: MakeVDC VDC: MakeVDC • Selects processing-queue entries that have met pre-processing requirements • Generate VDC job files according to configured priorities • Runs as a Scheduler task, so it can easily be configured to run as often as needed to keep the VDC queue full

  48. VDC: Engine VDC: Engine • Runs in a daemon-like state on each processing server • Each instance of the VDC Engine actively competes for jobs that it is allowed to run, based on priority and length of time in the queue • Monitors and manages processing resources • Initializes processing streams • Invokes recipe steps and monitors step-execution time • Handles operator-requested stream actions

  49. Archive Device Manager Subsystems: Archive Device Manager (ADM) • User defines logical pools of storage devices • Processes request a device in a specific pool • ADM returns information for a storage device in the requested pool • If auto-cycling is enabled, the ADM time-stamps the record for the selected device, so a different device within the pool will be selected for the next request • Disk-monitor process polls all devices periodically to record usage statistics and invoke threshold handlers

  50. Data Acquisition and Ingest (1 of 2) Subsystems: Data Acquisition and Ingest • Data types and sources are described in the database • Active, passive, and periodic notification methods • Active method scans remote systems for new files and populates the ingest queue • Passive method waits for arrival of e-mail messages describing type and location of new file and populates the ingest queue • Periodic method schedules transfers of files at user-specified intervals

More Related