630 likes | 913 Views
Ocean PEATE. Fred Patt NPP SDS Peer MOR Building 8 Auditorium August 27, 2008. NICSE. I&TSE. Ozone. SD3E. Sounder. Atmosphere. Land. Ocean. PSOE. Ocean PEATE Agenda. Ocean PEATE Overview Implementation Status Documentation Support Staff EDR Assessment Issues/Concerns.
E N D
Ocean PEATE Fred Patt NPP SDS Peer MOR Building 8 Auditorium August 27, 2008
NICSE I&TSE Ozone SD3E Sounder Atmosphere Land Ocean PSOE Ocean PEATE Agenda • Ocean PEATE Overview • Implementation Status • Documentation • Support Staff • EDR Assessment • Issues/Concerns
Ocean PEATE Overview • The NPP Ocean PEATE will be implemented within the framework and facilities of the current NASA Ocean Data Processing System (ODPS) • This system has been successfully supporting operational, satellite-based remote-sensing missions since 1996, and its capabilities continue to evolve and expand to meet the demands and challenges of future missions.
Ocean PEATE Functions • Acquire VIIRS RDRs, SDRs, and Ocean EDRs from the SD3E and ADS/CLASS • Assess the quality of the NPP Ocean EDRs for accomplishing NASA’s climate research requirements • Provide suggested algorithm improvements to the IDPS via the Project Science Office Element (PSOE) • Process selected data subsets in support of Evaluation and Validation activities
RSMAS SeaBASS VOST SD3E CLASS(ADS) Casa-NOSA AncillaryDataProviders I&TSE Ocean ScienceCommunity NICSE PSOE Ocean PEATE Interface Diagram Analysis Results, Proposed Algorithm Updates xDRs, IPs, Ancillary Data Management Direction xDRs, IPs, Ancillary Data (if unavailable from SD3E) OceanPEATE Pre-flight Algorithms, Data, Info Software, Data Alternate Ancillary Data xDR Eval. Results, Algorithm Updates Calibration Updates and Evaluations Interaction In Situ Data In Situ Data Algorithm Updates, Test Requests & Results Matchups
Ocean PEATE External Interfaces (1 of 2) • SDS Science Data Distribution and Depository Element (SD3E) • Provides NRT access to raw data • Primary source of RDRs • Provides selected SDRs and EDRs • SDS Integration and Test System Element (I&TSE) • Build and test updates to operational code in mini-IDPS • Run tests on selected data per request of PEATE • Archive Distribution Segment (ADS) • Primary source for archived data • xDRs, IPs, Ancillary Data, Operational Algorithm/Source Code and Calibration Products • Ancillary Data Providers (ADP) • Provides alternate ancillary data sets (e.g., ozone, meteorological data sets) • CasaNOSA • Serves as the NPP pre-flight repository of Government held data for distribution to Government user teams • Place to acquire pre-launch NPP algorithms and supported data files
Ocean PEATE External Interfaces (2 of 2) • NASA VIIRS Ocean Science Team (VOST) • Coordinate activities with PEATEs and PSOE on xDR and recommended algorithm improvements. Supports Independent Calibration Validation Activities • NPP Instrument Calibration Support Element (NICSE) • Provides alternative calibration LUTs and recommended improvements to calibration algorithms • PEATE provides results of LUT and algorithm tests • Project Science Office Element (PSOE) • Provides management direction • Accepts algorithm update recommendations • SeaBASS/ODPS • Provides Ocean Color in situ data • RSMAS/U. Miami • Provides SST in situ locations • PEATE provides SST EDR matchups • Ocean Science Community • Relies on Ocean PEATE to provide evaluation products and results
NICSE I&TSE Ozone SD3E Sounder Atmosphere Land Ocean PSOE Ocean PEATE Agenda • Ocean PEATE Overview • Implementation Status • Documentation • Support Staff • EDR Assessment • Issues/Concerns
ODPS Design Overview • Fully automated, distributed data system for acquiring, processing, archiving, and distributing scientific data • Highly scalable • Easily adaptable to support multiple concurrent missions • Graphical user interfaces for controlling and monitoring system functions and activity • Non-platform specific
ODPS COTS and Freeware • Linux OS (CentOS 4.x) • Solaris OS • Sybase RDBMS • Subversion (source code management) • Pro-active DBA • Interactive Data Language (IDL) • Generic Mapping Tool (GMT) • Netpbm (graphic image toolkit) • HDF5 Library • Languages: C, PERL, SQL
ODPS Architecture: Hardware • Processing Servers • Intel-based dual Xeon / AMD-based dual Opteron • 8 GB RAM • Five 72 GB SCSI drives • Storage Servers • Intel-based P4 / AMD-based single Opteron • 1 GB / 2 GB RAM • 1.5 TB IDE RAID 5 (3ware) / 9.6 TB SATA RAID 6 (Areca) • 2 hot spare drives per RAID5 • Database Server • Sun V880 • 8-16 GB RAM • 6-12 70 GB SCSI HDD
Distribution Servers (FTP) 1 processing node 6 storage nodes 48 TB Distribution Servers (web) 1 large server 8 processing nodes 68 storage nodes 599 TB Ingest Servers 2 SeaSpace ground stations 2 processing nodes 5.6 TB Cal/Val & QC Systems Processing Cluster 45 processing nodes 12.4 TB Extreme Networks Black Diamond 6816 Gigabit Ethernet switch Mission Operations Systems Database Server 2 servers 18 TB Development Servers 1 processing node 2 storage nodes 2.4 TB User Desktops Backup Servers 2 storage nodes 18 TB Critical data mirrored Network Support Systems ODPS Current Components
Reliability and Redundancy • Critical components (database server, network systems) have full maintenance contracts to ensure rapid response to problems • Multiple-server components (ingest, processing, storage, distribution) have substantial redundancy to maintain full capability; spares maintained for rapid replacement. • Testing nodes are separated from mainstream production components.
Technology Refresh • Hardware technology advances (CPU, storage) are continuously monitored to select new components for evaluation. • Typical upgrade threshold is a doubling in capacity (~18 months). • Two generations of hardware are generally in use. • Candidate components are procured, installed in testing cluster and rigorously evaluated in a production-like environment. • Following successful evaluation, multiple copies are procured, installed, tested and swapped in for older components. • ODPS design allows new components to be rapidly added to resource tables without interrupting system operations. • Performance of new components is closely evaluated following installation in operations environment. • Critical components are run in parallel with existing system to ensure reliability under production loading.
Existing Software Reuse • ODPS Components • Database • VDC/Scheduler • Data Acquisition and Ingest • Level-3 Scheduler • File migration and management • Archive Device Manager • Data distribution • Level-2 multi-mission software (vicarious calibration) • Level-3 multi-mission software (long-term trends and comparisons • Level-2 to Level-3 comparison software (residual sensor errors) • SeaBASS (in situ data management) • Matchup/extraction software
Ocean PEATE New Capabilities (1 of 2) • Acquire, ingest and catalog NPP VIIRS data products: RDRs, SDRs and Ocean EDRs (Data Acquisition & Ingest, Device Manager and File Migration and Management). • Status: Initial interface test run with SD3E. • Process selected Ocean EDRs (SST and OCC) to Level-3 to support data product and algorithm evaluations (Level-3 Scheduler, VDC and Level-3 binner). • Status: Prototype Level-3 processing has been demonstrated using sample IDPS Build 1.4 OCC and SST EDRs. • Perform VIIRS OCC EDR matchups with SeaBASS Ocean Color in situ data (extract code). • Status: Pending re-test of acquisition and ingest capabilities with latest IDPS build.
Ocean PEATE New Capabilities (2 of 2) • Incorporate VIIRS SDR processing for vicarious calibration analysis. • Status: Pending re-test of acquisition and ingest capabilities with latest IDPS build. • Produce VIIRS proxy data using VOST-developed software (VDC/Scheduler). • Status: Under development • Acquire SST in situ data from RSMAS and perform matchups with SST EDRs • Status: SST acquisition in place for MODIS; matchups pending re-test of acquisition and ingest capabilities with latest IDPS build. • Support browse and distribution of data products for team members (Data Distribution). • Status: Pending re-test of acquisition and ingest capabilities with latest IDPS build.
Ocean PEATE Data Storage Estimate • Assumptions: • Long-term storage is sized for 100% of RDRs and 10% of SDRs and EDRs • SDR volume includes geolocation
New Hardware for the Ocean PEATE • 7 Storage Servers @ 9.6 TB – first-year VIIRS data storage • Additional servers acquired post-launch to handle years 2 – 5 • No new processing or network capacity required; technology refresh cycle to be continued within the ODPS as described.
Build 1 (L-18 months) All interfaces fully implemented and tested Verify initial versions of operational code ported and running in I&TSE L-3 product code developed and tested Prelaunch VIIRS test data storage and SDS interface testing support with existing ODPS storage capacity Initial test products generated for review by VIIRS Ocean Science Team Build 2 (L-12 months) Routine exercise of interfaces to acquire proxy, surrogate (Aqua?) and/or simulated data Verify pre-launch version of operational code running in I&TSE Browse and distribution capability developed and tested Test products routinely acquired as available and posted for access by VIIRS Ocean Science Team Data storage for one year Ocean PEATE Build Schedule
Ocean PEATERequirements Implementation • 98% of requirements (55) implemented by Build 2; add additional storage capacity before launch
Sustaining Operations and Maintenance • ODPS runs 24x7 with on-site support 8x5. • Support is shared across all projects. • Staffing needs are covered by existing OBPG support personnel.
NICSE I&TSE Ozone SD3E Sounder Atmosphere Land Ocean PSOE Ocean PEATE Agenda • Ocean PEATE Overview • Implementation Status • Documentation • Support Staff • EDR Assessment • Issues/Concerns
Documentation • Ocean PEATE Level 4 Requirements and Operations Concept • ICDs • SD3D to PEATEs • MOU with RSMAS • ODPS Project Data Management Plan (review) • ODPS Software Management Plan • Network IT Security Plan
NICSE I&TSE Ozone SD3E Sounder Atmosphere Land Ocean PSOE Ocean PEATE Agenda • Ocean PEATE Overview • Implementation Status • Documentation • Support Staff • EDR Assessment • Issues/Concerns
Ocean PEATE Team • Ocean PEATE Management • Gene Feldman • Fred Patt • Validation Support • Sean Bailey • Jeremy Werdell • Software Support • Bryan Franz • Joel Gales • Data System • John Wilding • Dan Knowles • Muriel Taylor • Systems Management • Paul Smith • Joe McSorley • Chris Moellers
Ocean PEATE Staffing Levels • All staffing needs will be met with existing OBPG staff • Increases will be accommodated as other projects (e.g., MODIS Aqua) wind down.
NICSE I&TSE Ozone SD3E Sounder Atmosphere Land Ocean PSOE Ocean PEATE Agenda • Ocean PEATE Overview • Implementation Status • Documentation • Support Staff • EDR Assessment • Issues/Concerns
VIIRS Ocean Products for Evaluation • All VIIRS Ocean products to be evaluated will be acquired from the IDPS, ADS/CLASS, or the I&TSE. • Testing and evaluation of algorithm changes will require regenerating product time series in the I&TSE, to be downloaded to the Ocean PEATE. • The Ocean PEATE will design changes to the code in the I&TSE for the purpose of algorithm improvement or problem resolution, develop appropriate test cases and request runs to verify and evaluate the changes.
IDPS VIIRS Ocean EDR Data Flow Processing Module VIIRS Product VIIRS RDR Dynamic Ancillary Data Previous VIIRS Gridded Products Static Ancillary Data Previous VIIRS Gridded Products NAAPS TOD MODIS Land/Water Mask NCEP Geopotential Height Ancillary Files Ancillary Files DEM NDT
IDPS VIIRS Ocean EDR Data Flow VIIRS Geolocation VIIRS SDR Processing Module VIIRS_SDR_01 RDR Decompression VIIRS_GEO_01 Geolocation VIIRS_SDR.MOD 750m SDR VIIRS_SDR.IM 375m SDR VIIRS SDR VIIRS Product VIIRS RDR Dynamic Ancillary Data Previous VIIRS Gridded Products Static Ancillary Data Previous VIIRS Gridded Products NAAPS TOD MODIS Land/Water Mask NCEP Geopotential Height Ancillary Files Ancillary Files DEM NDT
IDPS VIIRS Ocean EDR Data Flow VIIRS Geolocation VIIRS SDR Processing Module VIIRS_SDR_01 RDR Decompression VIIRS_GEO_01 Geolocation VIIRS_SDR.MOD 750m SDR VIIRS_SDR.IM 375m SDR VIIRS SDR VIIRS Product VIIRS RDR Dynamic Ancillary Data Previous VIIRS Gridded Products Static Ancillary Data Previous VIIRS Gridded Products VIIRS_GD_08 750m Granulation NAAPS TOD MODIS Land/Water Mask VIIRS_GD_25 NAAPS Granulation NCEP Geopotential Height VIIRS_GD_27 L/W Mask Granulation Ancillary Files Ancillary Files VIIRS_GD_11 Ancillary Profile VIIRS_GD_09 GFS Granulation ALL_GD_01 Time Interpolation VIIRS_GD_28 Surface Pressure Adjustment DEM VIIRS_GD_12 Bathymetry Granulation NDT VIIRS_GD_13 Temperature Granulation
IDPS VIIRS Ocean EDR Data Flow VIIRS Geolocation VIIRS SDR Processing Module VIIRS_SDR_01 RDR Decompression VIIRS_GEO_01 Geolocation VIIRS_SDR.MOD 750m SDR VIIRS_SDR.IM 375m SDR VIIRS SDR VIIRS Product VIIRS RDR Dynamic Ancillary Data Previous VIIRS Gridded Products Static Ancillary Data Previous VIIRS Gridded Products VIIRS_GD_08 750m Granulation NAAPS TOD MODIS Land/Water Mask VIIRS_GD_25 NAAPS Granulation VIIRS_LN_06 Active Fires VIIRS_CM_01 Cloud Mask NCEP Geopotential Height VIIRS_GD_27 L/W Mask Granulation Ancillary Files Ancillary Files VIIRS_GD_11 Ancillary Profile VIIRS_GD_09 GFS Granulation ALL_GD_01 Time Interpolation VIIRS_GD_28 Surface Pressure Adjustment DEM VIIRS_GD_12 Bathymetry Granulation NDT VIIRS_GD_13 Temperature Granulation
IDPS VIIRS Ocean EDR Data Flow VIIRS Geolocation VIIRS SDR Processing Module VIIRS_SDR_01 RDR Decompression VIIRS_GEO_01 Geolocation VIIRS_SDR.MOD 750m SDR VIIRS_SDR.IM 375m SDR VIIRS SDR VIIRS Product VIIRS_SN_02 Ice Quality VIIRS RDR Dynamic Ancillary Data Previous VIIRS Gridded Products Static Ancillary Data VIIRS_CL_01 Cloud Optical Properties Previous VIIRS Gridded Products VIIRS_GD_08 750m Granulation VIIRS_ST_02 Surface Temp NAAPS TOD VIIRS_SN_03 Ice Concentration MODIS Land/Water Mask VIIRS_GD_25 NAAPS Granulation VIIRS_LN_06 Active Fires VIIRS_CM_01 Cloud Mask NCEP Geopotential Height VIIRS_GD_27 L/W Mask Granulation Ancillary Files Ancillary Files VIIRS_ST_01 Sea Surface Temperature SST EDR VIIRS_GD_11 Ancillary Profile VIIRS_GD_09 GFS Granulation VIIRS_AR_01 Aerosol Type ALL_GD_01 Time Interpolation VIIRS_GD_28 Surface Pressure Adjustment VIIRS_OC_01 Ocean Color / Chlorophyll DEM VIIRS_GD_12 Bathymetry Granulation OCC EDR NDT VIIRS_GD_13 Temperature Granulation
ODPS MODIS Ocean Product Data Flow Processing Module MOD_PR01 Level-0 to 1A MODIS Product Dynamic Ancillary Data MODIS Level-0 MODIS Level-1A Static Ancillary Data MODIS 1 km Level-1B MOD_PR03 Geolocation MOD_PR02 Level-1A to 1B MODIS SST Platform ATTEPH Data MODIS Geolocation MSl12 Level-1B to 2 MODIS Ocean Color MET Ancillary Data Ozone Ancillary Files Land/Water Mask
Ocean Team Algorithm Change Process N ST member develops evaluation strategy Products improved? PEATE staff request ITSE granule-level test ST analyzes products and test results Y PEATE staff upload test input data and request evaluation test run ST member identifies algorithm change ITSE performs test run ST provides algorithm change recommendation to PSOE ST member and PEATE staff develop code change PEATE retrieves test products ITSE performs test run PEATE and ITSE staff install and build code change ST member verifies test products PEATE retrieves test products N Y Code correct? Expected results? PEATE and ST member check code change N Y ST member evaluates test products
Evaluation vs. Product Level Granule-based Evaluations • Level-1 (SDR) Evaluations • Onboard calibration analyses • Vicarious calibration • Level-2 (EDR) Evaluations • Matchup analyses • Flag analyses Time Series Evaluations • Level-2 (EDR) Evaluations • Residual detector (striping) and scan (RVS) dependence • Level-3 Product Evaluations • Sensor and algorithm comparisons • Temporal anomaly evaluations
Vicarious Calibration • Vicarious calibration using surface measurements (e.g., MOBY) gives constant gain correction to TOA radiances over the mission • Analyses of calibration results provide insight into temporal and geometric sources of variation.
Vicarious Calibration • Strengths: • Fine-tuning of sensor absolute radiometric response; allows for correction of prelaunch radiometric calibration • Limitations: • “System-level” calibration that incorporates atmospheric correction into measurements, convolving algorithm and sensor effects. • One-time gain correction per band for the entire sensor lifetime; assumes sensor stability has already been established through other means; not a substitute for lack of instrument characterization. • Observation limitations and rigorous data quality control severely restrict the number of useful measurements (typically 1 or 2 per month per site). • Specific to ocean algorithms, not applicable to other surface types. • Availability of MOBY or equivalent is unknown for VIIRS.
Matchup Analysis • Ocean data granules in ODPS catalog are automatically matched with in situ data • SeaWiFS Bio-optical Archive and Storage System (SeaBASS) stores and manages in situ holdings from field programs and supported investigators. • Ocean staff acquire, QC and analyze new data samples. • Over 60,000 in situ profiles stored covering SeaWiFS/ MODIS time frame.
Matchup Analysis • Strengths: • The only truly independent validation of the science data products using ground truth measurements. • Limitations: • Quality of in situ data is highly variable and difficult to assess. • Coverage for OC in situ data is limited, both geographically and temporally. • Assumes that highly localized (~meters) measurements are representative of pixel (km) area. • In situ measurements require discipline expertise to analyze and compare with satellite values • Generally useful only for assessing static biases in final products. • Availability of in situ data in VIIRS timeframe is unknown.