1 / 13

Science Data Processing

Science Data Processing. www.stsci.edu/software/OPUS/ www.dpt.stsci.edu/. OPUS Science Data Processing. OPUS Processes. Data Partitioning segments the telemetry stream into standard EDT dataset appends multiple dumps into a single exposure file uses an index file to avoid sorting packets

kedma
Download Presentation

Science Data Processing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Science Data Processing www.stsci.edu/software/OPUS/ www.dpt.stsci.edu/

  2. OPUS Science Data Processing

  3. OPUS Processes • Data Partitioning • segments the telemetry stream into standard EDT dataset • appends multiple dumps into a single exposure file • uses an index file to avoid sorting packets • flags packet errors caught by PACOR • Support Schedule • gathers proposal information from PMDB • test proposals required for development • test version of PMDB must be populated by TRANS

  4. OPUS Processes (cont.) • Data Validation • decodes the exposure and engineering parameters in the telemetry and compares them to the planned values • internal header specification (from Ball) • PDB (EUDL.DAT, TDFD.DAT) must be fully populated and defined in DM-06 [DONE] • Planned value comparisons (“flags and indicators”) need to be determined by the Instrument Scientists [09/2002] • World Coordinate System • implements a translation from telescope coordinates through the instrument light-path to an astronomically valid pointing • aperture positions must be defined [09/2002]

  5. OPUS Processes (cont.) • Generic Conversion • Generic Conversion outputs uncalibrated data • raw, asn, spt, trl • data will be output in standard FITS format with image or table extensions • primary header will contain keywords inherited by all extensions and a null data array • Extensions • Images grouped by imsets consisting of science array, data quality array, and null error array. • data quality array will be null if no telemetry dropouts

  6. OPUS Processes (cont.) • Generic Conversion (cont.) • Required for development • DM-06 to develop algorithms for data formatting [DONE] • keyword definitions (ICD-19) are currently being worked by the Instrument Scientists • world coordinate definitions [09/02] • calibration switches and selection criteria [09/02] • calibration file name keywords [09/02]

  7. OPUS Processes (cont.) • Data Collector • OPUS will ensure all component exposures are present before processing further • association table contains information about the product dataset • dataset self-documenting • only associations required for data processing will be constructed in the OPUS pipeline • CR splits • Repeat obs

  8. Calibration • OPUS will use STSDAS calibration software • run on ALPHA / Tru64 UNIX platform in operations • expands size of dataset • converts integer raw data to real • possible expansion of data quality array. • expansion of the error array. • Need calibration reference files for testing (at least dummies)

  9. Other Science Data Modes • data content & keywords for other science data modes must be defined by Instrument Scientists • microprocessor memory dump [09/02] • engineering diagnostic data [09/02]

  10. CODE REUSE • Core OPUS system (OPUS 13.2) • ~90,000 lines of code • 100% reuse • WFC3 specific processes • Based on FUSE study (Rose et al. 1998, “OPUS: The FUSE Data Pipeline”, www.stsci.edu/software/OPUS/kona2.html) • 5076 lines of code • 71% reuse of existing OPUS modules (ACS, NICMOS) • Major effort in functional and throughput testing

  11. Major Science Data Processing Requirements Summary • Internal header specification • DM-06 to document content and format of science internal header [DONE] • PDB (EUDL.DAT, TDFD.DAT) defined. [DONE] • Keyword definitions [DONE] • Planned values “flags and indicators” for Data Validation • Aperture definitions

  12. Test Data Requirements • Test data from detectors on ops bench expected in March 2002 and from integrated instrument in June 2002 • including all science and engineering modes • SMGT & IPT test data must include: • PMDB population and PDB definition • images that enable OPUS to determine proper orientation • list of possible error conditions to simulate • data that simulate those error conditions • enough data for throughput test

  13. Thermal Vac Support • .SDI files will trigger full dataset name: • ipppssoot_csiiyydddhhmmss • allows reuse of SMS • Optical Stimulus Snapshots • named the same as the image file: csiiyydddhhmmss.oss • allows corresponding OSC data to populate keywords • Support Schedule defaults • since exposures will not have PMDB support, a default support schedule will be used. Auxilliary keyword values can be supplied in a separate exposure-specific ASCII file.

More Related