1 / 43

July 19, 2010 Seattle, wA

July 19, 2010 Seattle, wA. Aquarius/SAC-D Science Team Meeting. Aquarius Data Processing System (ADPS) and Data Distribution Fred Patt. Overview Science Data Processing Level-2 Science Software Implementation Level-2 Data Product Format Data Archive and Distribution Aquarius Web Site

yovela
Download Presentation

July 19, 2010 Seattle, wA

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. July 19, 2010 Seattle, wA Aquarius/SAC-D Science Team Meeting Aquarius Data Processing System (ADPS) and Data Distribution Fred Patt

  2. Overview Science Data Processing Level-2 Science Software Implementation Level-2 Data Product Format Data Archive and Distribution Aquarius Web Site What We Need Agenda

  3. The Aquarius Ground System at NASA/GSFC consists of the Aquarius Data Processing System (ADPS) and the Aquarius Command and Control System. The ADPS will support: Acquisition and ingest of Aquarius/SAC-D data from CONAE. Acquisition of dynamic ancillary required to support the science processing. Processing of Aquarius data to standard product levels using algorithms provided by the Science Team. Quality control of data products using methods specified by the Science Team. Match-up of Aquarius data with in situ salinity measurements. Archive and distribution of all data products during the active life of the mission to the science team and general science community. Delivery of source data (Level-1a) and “validated” Level-2 and Level-3 derived data products to the PO.DAAC. The ADPS will be supported within the existing infrastructure of the Ocean Biology Processing Group (OBPG). Aquarius Data Processing System Overview

  4. Overview and Status Science Data Processing Level-2 Science Software Implementation Level-2 Data Product Format Data Archive and Distribution Aquarius Web Site What We Need Agenda

  5. Aquarius Data Processing Overview Standard Data Processing Web-Based Telemetry Monitoring Tools Raw Level-0 Downlink File Level-0 Report Level-0 Preprocessor Level-0 Level-1a Report SAC-D HKT Data Level-0 to 1A Level-1A temp Automated Telemetry Analysis Tool Level-1A Merge Level-1A orbit Dynamic Ancillary Data Telemetry Report Level-2 Level-1A to 2 In-Situ Match-up Process Static Ancillary Data Predicted Downlink Times AVDS Level-2 Match-up Bias Adjustment Level-2 to 3 Smoothing Level-2 to 3 Binning SAC-D HKT Data CONAE-provided Interactive Telemetry Analysis Tool Level-3 Bin Level-3 Mapping Web-Based Browse, Download and Order Tools Level-3 Binning Level-3 Map

  6. The Aquarius science data processing consists of the following steps: Level-0 Preprocessor – process each HRD downlink file to produce a single time-ordered set of Aquarius science blocks and removes duplicates– developed by Joel Gales - ADPS. Level-0-to-1A – separate Level-0 files into orbits, unpack science data, incorporate overlapping ephemeris and SAC-D HKT and convert Aquarius HKT – developed by Joel Gales - ADPS. Telemetry Analysis – HKT data from Level-1A files analyzed for web display on the Aquarius web site – developed by Susie Kennison, Norman Kuring and Liang Hong - ADPS/ACCS. Interactive Telemetry Analysis - HKT files analyze using CONAE-provided telemetry viewer. Level-1A Merge – consolidate Level-1A files from overlapping downlinks into a single, best-quality full orbit product – Joel Gales - ADPS. Level-1A-to-2 – perform calibration, atmospheric correction and salinity retrieval for Aquarius science data – radiometer and scatterometer code delivered by RSS and JPL, respectively; integration and input/output code developed by Joel Gales and Liang Hong - ADPS. Level-2-to-3 Binning – Level-2 salinity retrievals for one day geographically projected and collected into equal-area bins – existing OBPG software modified by Joel Gales- ADPS. Level-2-to-3 Smoothing – Level-2 salinity retrievals for one day optimally interpolated into equal-area bins – algorithm by Jonathan Lilly/ESR implemented by Joel Gales - ADPS. Level-3 Binning and Mapping – Level-3 binned files aggregated to longer time periods (weekly, monthly, etc.) and projected onto map grid – existing OBPG software. All processing steps are run automatically when all input data sets are available. All software exists in at least preliminary prelaunch form at this time. Science Data Processing

  7. Radiometer and Scatterometer 2009 code deliveries have been integrated and installed in the data processing system. Latest versions of Level-1A and Level-3 software have been installed. Ancillary data to support processing have been acquired and ingested. The one-year (2007) simulated data set has been processed to Level-1A, Level-2 and Level-3 products: http://oceancolor.gsfc.nasa.gov/cgi/aquarius (Level1A/2 browser) http://oceancolor.gsfc.nasa.gov/cgi/l3 (Level-3 browser) http://oceandata.sci.gsfc.nasa.gov/Aquarius (FTP-like interface) The PO.DAAC is downloading the simulated data products and making them available (see poster by Gregg Foti). AVDS processing is in progress with the 2007 data set. Science Code Status

  8. From now until launch we need to: Integrate Radiometer and Scatterometer code updates into data processing system. Incorporate additional quality flags into software and products. Develop ingest procedures for additional ancillary data required by new code. Incorporate SAC-D HKT format into Level-1A processing. Verify output products with Science Team. Begin real-time mission simulation (August 2010 --> Launch). Incorporate additional prelaunch updates into the science software (as needed). Participate in mission tests as scheduled. Science Code TBD

  9. Overview and Status Science Data Processing Level-2 Science Software Implementation Level-2 Data Product Format (separate presentation) Data Archive and Distribution Aquarius Web Site What We Need Agenda

  10. Level-2 Software Implementation • Implemented using software delivered by RSS and JPL with I/O by Joel Gales • Capabilities: • Processes a Level-1A input product. • Performs radiometer Level-1B calibration and Level-2 processing using latest code from RSS. • Performs scatterometer Level-1B calibration and Level-2 processing using latest code from JPL. • Reads SST, wind, atmospheric, sea ice and solar flux data from dynamic ancillary files. • Computes geolocation. • Generates HDF5 output product.

  11. Scatterometer ASCII Files Level 2 Processing Flow QMET Ancillary Files Solar Flux Ancillary Data SST Ancillary Files Sea Ice Ancillary Files Static Ancillary Files Look-up Tables atm_tb_aquarius Level-1A Product L2gen_scatter ATM Ancillary Files mk_aquarius_ ancillary_data Look-up Tables Radiometer Ancillary Files L2gen_aquarius Level-2 Product

  12. Dynamic Ancillary Data

  13. Overview and Status Science Data Processing Level-2 Science Software Implementation Level-2 Data Product Format Data Archive and Distribution Aquarius Web Site What We Need Agenda

  14. Aquarius Level-2 Product Content • This will actually be a combined Level 1B / Level 2, radiometer/scatterometer product. • The current product specification incorporates updated parameter lists based on the recent code deliveries from the radiometer and scatterometer teams. • Specific parameters to be output are configurable at runtime. • Parameters are output per beam and block (1.44 sec) • All output products include standard metadata, flags and geolocation fields.

  15. Radiometer Parameters (1 of 2)

  16. Radiometer Parameters (2 of 2)

  17. Scatterometer Parameters (1 of 2)

  18. Scatterometer Parameters (2 of 2)

  19. Overview and Status Science Data Processing Level-2 Science Software Implementation Level-2 Data Product Format Data Archive and Distribution Aquarius Web Site What We Need Agenda

  20. Overview of the currently operational web-based browse, order and distribution tool that is available to support the distribution of Aquarius data within the Aquarius Data Processing System that also serves as a data quality monitoring tool for both internal and external use. ADPS Data Archive and Distribution

  21. Aquarius Level-1-2 Data Browser http://oceancolor.gsfc.nasa.gov/cgi/aquarius

  22. Aquarius Level-1-2 Data Browser http://oceancolor.gsfc.nasa.gov/cgi/aquarius

  23. Aquarius Level-3 Data Browser http://oceancolor.gsfc.nasa.gov/cgi/l3 All Beams - daily

  24. Overview and Status Science Data Processing Level-2 Science Software Implementation Level-2 Data Product Format Data Archive and Distribution Aquarius Web Site What We Need Agenda

  25. Aquarius Website Consolidated Data access, Information, Services and Community Feedback

  26. Aquarius Wiki Online Documentation and Software Descriptions available online at: http://aquarius.nasa.gov/ Under Data and Documents Link labeled “Aquarius Ground System and Mission Operations Wiki Documentation”

  27. Overview and Status Science Data Processing Level-2 Science Software Implementation Level-2 Data Product Format Data Archive and Distribution Aquarius Web Site What We Need Agenda

  28. What We Need • Review and analysis of the 2007 simulation products • Agreement on the ancillary data sets and sources. • Agreement on the Level-2 parameters. • Agreement on the at-launch processing baseline (e.g., NCEP winds). • Schedule and responsibility for algorithm and software updates between now and launch.

  29. Backup Slides

  30. Ground System Interfaces S-Band S-Band ASI Malindi X-Band CONAE X-Band Commands Real-Time Telemetry & Coordination Data Mission RF Archive Operations Services Command Center Wallops Svalbard Alaska McMurdo CUSS CODS NASA Ground Network Aquarius Data, SAC-D Telemetry and Orbit Products Cordoba Aquarius Command Schedules And Reports Schedule Confirmation And Reports ASI Matera Ancillary Data Sources Calibration & Instrument Configuration Change Requests Principal Investigator and Science Planning Team ADPS ACCS Aquarius Aquarius Data Command & Processing Control System System Algorithms & Science Code Evaluation Products Instrument Engineering Team Science Team and Algorithm Providers Aquarius Instrument Testbed Flight Software Updates Insitu Data Bias Adjustments Aquarius Match-up Data Level 1, 2, 3 Aquarius Data AVDS Aquarius Validation Data System PO.DAAC Aquarius L1, 2, 3 Data Ancillary Data Feedback 1- SAC-D Ground System to Aquarius Ground System ICD 2- SAC-D / NASA PSLA, Network Requirements Document & Scheduling Interface 3- SD - Observatory Ground IF Specification Document 4- Aquarius Science Team to Aquarius Ground System ICD 5- AQ - Aquarius Ground System – Aquarius Instrument Team Anomaly ICD and Aquarius Test bed User Manual 6- Aquarius Data Product User Guide (PO.DAAC) & Format Specification (ADPS) 7- Ocean Data Processing System Data and Software Management Plan, ATBD’s and Software User’s Guides 8- A V D S to A D P S ICD 9- Aquarius Ground System to PO.DAAC ICD Data Users and Science Team Software SeaDAS

  31. Aquarius Software Status • For each software module, we will show: • High-level flow (inputs and outputs). • Current status and capabilities. • Remaining work to be done. Input File Processing Module Intermediate Data Output File

  32. Level-0 Preprocessor • Implemented by Joel Gales • Current Capabilities: • Processes a Level-0 downlink file at CONAE to remove duplicate blocks and sort by time. • To be done: • Final data quality checks. Level-0 Downlink File l0gen Level-0 File

  33. Level-1A Files Level-0 to 1A • Implemented by Joel Gales • Current Capabilities: • Processes a Level-0 multi-orbit file to multiple Level-1A orbit files with overlap. • Inputs SAC-D ephemeris and telemetry from separate files. • Unpacks Aquarius science data. • Converts Aquarius telemetry to engineering units. • Generates HDF5 L1A files. • To be done: • Implement actual SAC-D telemetry input file formats when these are known. SAC-D Ephemeris Data Level-0 File SAC-D Telemetry l1agen

  34. Level-1A Merge • Implemented by Joel Gales • Current Capabilities: • Merges multiple Level-1A files for the same orbit from successive downlinks. • To be done: • Data quality checks to keep only “best” data. Level-1A Files l1amerge Level-1A Product

  35. Level-2 to 3 Binning • Implemented by Joel Gales based on current OBPG binning software • Current Capabilities: • Processes multiple Level-2 products (e.g., for 1 week). • Average samples for each beam in 1 degree bins. • Outputs a Level-3 binned product per beam. • To be done: • Add flag exclusion rules. Level-2 Products l2bin Level-3 Binned Beam Product

  36. Level-2 to 3 Smoothing • Implemented by Joel Gales based on algorithm from Jonathan Lilly • Current Capabilities: • Processes multiple Level-2 products (e.g., for 1 week). • Generates smoothed values in 1 degree bins. • Outputs a Level-3 smoothed product. • To be done: • Additional smoothing algorithms or options. • Add flag exclusion rules. Level-2 Products l2smooth Level-3 Smoothed Product

  37. Level-3 Binning • Implemented by Joel Gales based on current OBPG binning software • Current Capabilities: • Processes multiple Level-3 binned products. • Averages samples in common bins. • Outputs a single Level-3 binned product. Level-3 Binned Products l3bin Level-3 Binned Product

  38. Level-3 Mapping • Implemented by Joel Gales based on current OBPG binning software • Current Capabilities: • Reads a Level-3 binned product. • Outputs a Level-3 mapped (lat-lon gridded) product. Level-3 Binned Product l3bin Level-3 Mapped Product

  39. OBPG Computing Facility NASA Goddard Space Flight Center Greenbelt, Maryland Server location: Building 28 Room W220 - 900 square feet dedicated to the project - Raised computer floor with under-floor A/C - Primary and backup A/C units - 80 KVA Powerware UPS unit with dual 50 KVA PDUs, additional 5 KVA rack-mounted UPS units - Connections to GSFC high speed networks: SEN

  40. Distribution Servers (ftp) 8 storage nodes 49.3 TB 1 server 250GB Distribution and Storage Servers (web) Production: 3 processing nodes 75 storage nodes 1.03 PB AQOPS Servers (web) Production: 2 processing nodes Forum Servers (web) Production: 2 processing nodes Ingest Servers 2 SeaSpace workstations 78 GB 1 storage node 5.1 TB Cal/Val & QC Systems Processing Cluster 46 processing nodes 15 TB 4 storage nodes 10.9 TB Extreme Networks Black Diamond 8810 Gigabit Ethernet switch 480 ports Aquarius Telemetry Viewer 2 Windows PC Mission Operations Systems Aquarius Command Planning Tool Servers 2 processing nodes 586 GB Development Servers 2 superstore nodes 10 TB Subversion CM Servers 2 superstore nodes 586 GB Testing Cluster User Desktops Backup Servers 2 storage nodes 19.2 TB Database Servers 2 superstore nodes 16.4 TB Network Support Servers OBPG System Hardware

  41. Aquarius Backup Strategy • Data • On-site • Downlinks, L0, L1a data files • primary copy on RAID6 storage node • mirror copy on separate RAID6 storage node, at file creation • Science Products • primary copy on RAID6 storage node • mirror copy if desired, or regenerated from L1a • Off-site • Downlinks: stored at CONAE • L1A archived at PO.DAAC • Source Code • On-site • primary copy on subversion server, RAID6 • mirror copy on backup Subversion server, RAID6, nightly mirror • Off-site • Subversion data copied to external disk, taken off-site every 1 month • Multiple copies, given to Project Manager, SE, Processing Manager • Other • On-site • AQOPS, Forum: database mirrored to backup server nightly • Telemetry Viewer, other: image backups periodically, manual • Off-site • AQOPS, Forum: databases included in source code off-line backups • Telemetry Viewer, other: image backups stored off-line like source code

  42. IT Security Plan ADPS and ACCS: covered under the same plan as other Ocean Biology Processing Group (OBPG) systems. NASA IT Security Plan Identification System name: Sciences and Exploration Directorate (SED) Multi-Program/Project (MPP) IT Science Systems System Abbreviation: SED MPP System Unique Identifier: CD-014-L-GSF-6004 Information System Type: General Support System Information System Security Categorization: Low (transitioning to medium within the year) Information System Operational Status: Operational Effective 5/31/2007. Passed Certification & Accreditation in Summer 2007 - in process of being upgraded Plan designated Sensitive But Unclassified by NASA Implementation questions: addressed off-line

  43. Ground System Staffing NASA Management - Gene Feldman: Oversight of the overall effort, communication/coordination with other NASA organizations and centers and CONAE; oversight of science support, algorithm integration, QC, validation and in situ data support. Contractor Management - Fred Patt: Oversight of contract support staff, including hiring, tasking, review of deliverables and schedules; ADPS interface definition and product format specification. Mission Operations - Susie Kennison: Technical lead for instrument operations and commanding support; interface to instrument engineers and CONAE mission operations. Data Systems Engineering - John Wilding: Technical lead for the data system integration effort; overall system design, implementation and testing. Software Engineering – Joel Gales: Technical lead for the level conversion / product generation software, including development of the Level 0-to-1 software and integration/packaging of the science (Level 2 and 3) processing software to be provided by the algorithm developers. Data Systems Support - Dan Knowles: Integrate the product generation software into the existing automated processing system, develop ingest modules, develop database tables for archive products, develop schedules and recipes for data acquisition, production and migration. Systems Administration - Paul Smith: Specify and acquire new processing and data storage hardware, integrate into existing data processing facility, verify/upgrade network capability for external data transfers. Quality Control and Validation - Liang Hong: Working with the Science Team, develop methodologies, tools, procedures and reports for operational QC; specify data and develop algorithms and tools for data and science algorithm validation. In Situ Data Support - Jeremy Werdell: Work closely with the Aquarius Validation Data System (AVDS) to develop methods and tools for data analysis and quality control, building upon the existing SeaWiFS Bio-optical Archive and Storage System (SeaBASS). Archive and Distribution - Norman Kuring: Incorporate Aquarius product browse, search, order and distribution into existing web-based Ocean Color capability and work with JPL PO.DAAC regarding “institutional” archive support requirements.

More Related