CLARREO Science Data System Architecture & Standards Chris Currey CLARREO Data Manager

1. CLARREO Science Data SystemArchitecture & StandardsChris CurreyCLARREO Data Manager

2. Rich Ullman – “plans for a science data system including interfaces to the mission command and control system, science teams, instrument support, customers, data centers, applications, etc.  The focus should be the systems architecturalelementsof the science data processing.  We also want to discuss particular recommendations for standards.  What standards does CLARREO expect to use?  What standards do you wish existed?” • Science Data System Architecture • Standards & Software Re-use • Phase A Trades/Studies

3. Science Data Products * MODIS and AIRS are on Terra, Aqua only: may not overlap with CLARREO mission; the remaining sensors are all planned for missions in the CLARREO time period #For IR, CLARREO calibrates CrIS, IASI which in turn calibrate leo and geo imager infrared channels.

4. Science Data Production Schedule • Telemetry downlink 500 Gbits/day/spacecraft (compressed) • L0 175 GBytes/day (2 spacecraft, uncompressed) • Data Availability*: • BetaEdition 1 • Level-1L+6 monthsL+12 months • Level-2 L+9 L+12 • Level-3 benchmark L+9 L+15 • Level-4 inter-calibrations L+12 L+18 • Data Latencies: • Level-0 48 hours after measurement • Level-1 12 hours from availability of Level 0 data product • Level-2 24 hours from availability of Level 0 data product • Level-3 1 week after month to process • Level-4 1 month after inter-calibration collection period • Estimated archive L0-L4 volume: 350GBytes/day; 128TBytes/year • * months from launch, Beta release not for publication, Edition 1 validated

5. Science Data Spacecraft CMD & TLM Data Sharing & Coord Space Segment Ground Segment Climate Science Community EDOS Box (Ebox) Ground Station White Sands Complex, NM Science Data System Poker Flats, AK / Svalbard, Norway / Wallops, VA Distributed Active Archive Center (DAAC) Level-0 Processing Facility, GSFC NISN EbNet Science Team Science Investigator Processing System (SIPS) NISN Cal-Val Team Science Computing Facility (SCF) Mission (Payload) Operations Center Spacecraft Operations Center Instrument Ops Team Spacecraft Ops Team Mission Planning System CMD and TLM System Analysis and Trending System Mission Planning System CMD and TLM System Analysis and Trending System NPOES NPP Launch Segment, VAFB METOP Space Weather Notification, GSFC Flight Dynamics Facility, GSFC Collision Avoidance Support, GSFC CLARREO Observatory Simulator Flight Software Vendors Reference Intercalibration Coordination I&T Facilities

6. Data Reception, Transmission, and Routing (PSLAsw/ CLARREO) • NASA Ground and Space Networks • Utilize X-band and S-band Antennas at Poker Flats, AK; Wallops, VA; and Svalbard, Norway • Leverage Tracking & Data Relay Satellites (TDRS) for launch & early orbit, maneuvers, and spacecraft emergencies • NASA Integrated Services Network (NISN) • Provides routing of command and S-band data • Delivers L0 science data to LaRC • EOS Data Operations System (EDOS) • Existing EDOS capability at NASA Ground Network X-band facilities • Autonomous, multi-mission X-band data capture system • Capable of providing real-time data statistics, data routing, and level-zero processing

7. Science Data System Interfaces DAAC: Distributed Active Archive Center EDOS: EOS Data Operations System IC: Inter-Calibration L0 – L4: Level 0 to Level 4 Science Data Products SCF: Science Computing Facility SIPS: Science Investigator-Led Processing System EPH: Ephemeris

8. SDS Context Diagram

9. CCSDS (Consultative Committee for Space Data Systems) packets • CCSDS File Delivery Protocol (CFDP) (Class 2) – Phase A study • Deliver files across space link w/ re-transmission of missing data • CRC on each file • Impact on L0 file format and processing? • Where to uncompress downlinked data? L0 software or SDS? • CLARREO coordinating with ESMO CLARREO Standards – Telemetry

10. L0 Processing part of ground system prior to SDS • Multiple versions of L0 Processing software • TRMM version; runs at USN (CALIPSO) • EOS version; runs at EDOS (CERES) • Need to define CLARREO Level 0 File formats • How to handle L0 Data Quality records? • Track L0 delivery statistics & missing/bad data impact on science processing • L0 files to be compatible with ECS Toolkit I/O routines? • CLARREO working with ESMO CLARREO Standards – L0 Processing

11. CERES and CALIPSO run DPREP in production • DPREP produces ECS Toolkit compatible ephemeris & attitude files (need for geolocation routines) • Re-use latest version of DPREP w/o RogueWave dependency? (contact D. Marinelli?) • CLARREO Project determining need for backup ground tracking and ephemeris post-processing with GPS measurements (Phase A) CLARREO Standards – DPREP

12. Identify portions of ECS Toolkit to re-use for PGEs • Core metadata routines • Process Control Files (PCF) • Coefficient and RTP access? • Geolocation routines • Time routines • Level 0 file I/O? • Other? • Part of Phase A Science PGE Framework study CLARREO Standards – Toolkit

13. CERES & CALIPSO experience with HDF • CLARREO considering HDF5, HDF-EOS5, netCDF3, netCDF4 • Goal to use same data format package for all standard products • CLARREO uses multiple data types: calibrated spectra, instantaneous retrievals (profiles and scalars), gridded data, and collections of co-located samples and calibration coefficients • Modeling community likes netCDF • Benefits of netCDF4? • Collaborative effort between Unidata and The HDF Group • Combine best features of both? • Satellite projects using? GMAO GEOS-5 planning upgrade • Phase A Study – prototype L1-L4 CLARREO products & evaluate performance CLARREO Standards – File Formats

14. Data Product Metadata (HDF products only) • Science codes generate core metadata using ECS SDP Toolkit • Generate product specific metadata • SIPS/ASDC Interface Metadata • SIPS generates ASCII metadata file to archive each data product • SIPS provides data set (collection) level metadata to ASDC • Ingest metadata (XML) triggers SIPS operations • ASDC Archive & Order Tools Metadata • Archive database is populated with metadata parsed from these files (data file and collection) • Metadata are exported to EOS Clearing HOuse (ECHO) • Need to automate data lineage record during production! CLARREO Standards – Metadata

15. It’s not just about software! • Transitioning from SGI Origins to Linux clusters • More choices, more complexity, more headaches! • Fabrics and shared storage systems are critical • Unified fabric to support inter-process, I/O, and network communications? • InfiniBand, FibreChannel, or 10GigE? • Communication protocols: FC, IPoIB, RDMA-based protocols (SDP, SRP), RDMAoE, FCoE, …? • Need interconnect that scales easily, reliably, and economically • Need to purchase SCF equipment for Phase A • Intel Nehalem processors (IDL & MatLab compatible) • Xsigo I/O virtualization – optimize blade bandwidth and provide access to FC, 10 GigE, or IB • OpenFabrics software stack? CLARREO Standards – Computer Infrastructure

16. Define Science PGE Framework Requirements & Architectural Design • C++ class library to provide data management features for PGEs • Science algorithms to be defined by Science Definition Teams • Features: • Generic dynamic data structures • Generic I/O - Binary (CCSDS packets), data product format (HDF, netCDF) • Remote access to external datasets (need for Inter-Calibration) • Coefficient and RTP access • ECS Toolkit abstraction (geolocation, time, PCF access, L0 file I/O, …) • Common error handling, logging, database access, … • Studies: • Extend CALIPSO Framework? • CERN Science Analysis Framework (ROOT) applicability? • Improve I/O performance to large shared storage systems (SAN/NAS) Phase A - Science PGE Framework Study

17. PGE Framework

18. Phase A - Science Ops. Automation Study • Automation Package Requirements & Architectural Design • Purpose to configure and execute science PGEs • Support all fully automated science operations (benchmarking, calibration, Inter-Calibration, reprocessing) • Experience: S4P package used for CALIPSO data-driven automation • Machine-to-machine communication for exchanging IC data between CLARREO SDS and external SDSs/DAACs • Phase A: • Develop use cases for CLARREO science operations • Discuss architectural approach, programming languages (Java, C++?), implementation plan

19. OpenDAP protocol + GRADS analysis software access to online data • CERES data validation studies; ASDC to serve out data • OpenADDE protocol + McIDASanalysis software access to online data • Currently use for inter-calibration studies w/ geostationary data CLARREO Standards – Data Access

20. BACKUP

21. S-band Antenna GNSS Zenith Antenna GNSS Ram Antenna GNSS Wake Antenna Reflected Solar Electronics Enclosures Reflected Solar Two-Axis Gimbal Reflected Solar Suite IR Suite S-band Antenna X-band Antenna Multiple Instrument Electronics Enclosures Observatory Features Instrument Observatory I Observatory II Orbit 609 km polar 609 km polar ± 0.52 secs ± Orbit Maintenance ± 0.1º Inclination ± 0.1º Inclination Attitude Control LVLH 3-axis LVLH 3-axis Active Pointing 0.1º to 1.0º (TBR) 0.1º to 1.0º (TBR) Observatory Concept Overview Payload Suite Instrument Observatory I Observatory II FTS FTS Infrared 5 to 50 micron 5 to 50 micron Spectrometer Nadir Nadir Grating Grating Reflected Solar 320-640 nm 320-640 nm Spectrometer 1 Gimbal-mounted Gimbal-mounted Grating Grating Reflected Solar 600-1200 nm 600-1200 nm Nadir Spectrometer 2 Gimbal -mounted Gimbal -mounted Grating Grating Reflected Solar 1150-2300 nm 1150-2300 nm Spectrometer 3 Gimbal -mounted Gimbal -mounted POD POD 0.52 secs GNSS Radio Occultation 1 Receiver 1 Receiver System 3 Antennas 3 Antennas

22. IR Scene-Select Mirror Nominal Operation

23. Reflected Solar Pointing Requirements and FOV • Instrument Requirements (600 km Orbit) • Ground instantaneous field of view (GIFOV): ~0.5 km at nadir • Swath width or ground field of view (GFOV): ~100 km • Field of view (FOV): ~10 deg • Instantaneous field of view (IFOV): 0.05 deg (0.5km @ 600km) • Overall Attitude Knowledge: 34 arcsec • Spacecraft Pointing Knowledge: 23 arcsec • Spacecraft Induced Jitter: 2 arcsec peak-to-peak over 0.05 sec • Instrument Thermal Distortion: 10 arcsec • Instrument Jitter: 2 arcsec peak-to-peak over 0.05 sec • Pointing Requirements for Intercalibration and Solar-Lunar Calibrations • Field of Regard (FOR): • Up to +/- 55 degrees off nadir for intercalibration with other spacecraft • Up to + 180 degrees off nadir for Solar/Lunar calibrations • Two-axis pointing-tracking requirements: • ≥ 1 deg/sec2 (TBR) minimum acceleration capability • ≥ 5 deg/sec (TBR) slew rate (non-data collection mode) • ≤ 3 deg/sec (TBR) tracking rate for intercalibration (data collection mode)

24. GNSS-RO Instrument Operations • RO measurements taken autonomously • Very little commanding required • Clock corrections performed by science data system (if required) • 1000 occultations/day/satellite • 500 each from GPS and Galileo • Assume not tracking GLONASS • Automatically steerable phased array antennas used to view GNSS constellation Earth-occulting satellites • Allows Setting and Rising occultations with Observatory yaw flip • POD antenna views GNSS constellation “reference” satellites

25. Inter-Calibration Science Operations