1 / 31

Profile of NPOESS HDF5 Files

Profile of NPOESS HDF5 Files. Kim Tomashosky, Ken Stone, Pat Purcell, Ron Andrews NPOESS Program Raytheon Company Aurora, Colorado. NPOESS HDF5 General Overview. Ron Andrews. HDF5 Conceptual Diagram. HDF5 XML User’s Block.

aiko
Download Presentation

Profile of NPOESS HDF5 Files

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Profile of NPOESS HDF5 Files Kim Tomashosky, Ken Stone, Pat Purcell, Ron Andrews NPOESS Program Raytheon Company Aurora, Colorado

  2. NPOESS HDF5 General Overview Ron Andrews

  3. HDF5 Conceptual Diagram

  4. HDF5 XML User’s Block • The XML User’s Block for NPOESS Data Products provides a ‘quick-look’ into the metadata of the associated HDF5 file. • The XML User’s Blocks are defined in the following volumes: • CDFCB-X Volume V – contains the XML User’s Block formats for the RDRs, SDRs, TDRs, EDRs, ARPs, and IPs • CDFCB-X Volume VI – Ancillary, Auxiliary, Reports, and Messages – contains the XML User’s Block formats for the ancillary, auxiliary, reports, and messages that are wrapped in HDF5 • The size of the HDF5 XML User’s Block will be a factor of 512 bytes, with a maximum size which is still to be determined • Provides elements: • N_Processing_Domain • Mission, Platform, and Instrument Names • N_Dataset_Type_Tag • Number_of_Data_Products • CollectionShortName(s) • Aggregation Information • Timestamps (Creation Timestamp, Observation Timestamps) • Percent Missing Data

  5. General HDF5 File Structure

  6. HDF5 File Root Group Metadata Elements – Example CDFCB-X Table

  7. HDF5 File Product Group Metadata Elements – Example CDFCB-X Table

  8. HDF5 File Product Aggregation Dataset Metadata Elements – Example CDFCB-X Table

  9. HDF5 File Product Granule Dataset Metadata Elements – Example CDFCB-X Table

  10. HDF5 File Product Granule Dataset Metadata Elements – Example CDFCB-X Table

  11. HDF5 File Product Granule Dataset Metadata Elements – Example CDFCB-X Table

  12. HDF5 File Product Granule Dataset Metadata Elements – Example CDFCB-X Table

  13. HDF5 File Spacecraft Diary Group Metadata Elements – Example CDFCB-X Table

  14. HDF5 File Spacecraft Diary Aggregation Metadata Elements – Example CDFCB-X Table

  15. HDF5 File Spacecraft Diary Dataset Metadata Elements – Example CDFCB-X Table

  16. HDF5 Conceptual Diagram - Data

  17. NPOESS Quality Flags Overview • The concept is to provide for consistently stored, high density, quality information about the delivered data – simplifying usability while maintaining storage efficiency • Quality flags are qualifications of one or more consecutive bits in each byte. • Quality flag arrays follow the structure of the data product • The size of the arrays are equal to or less than the size of the data to which the quality information applies (dimensions retain correspondence to data product arrays) • Quality flags will appear in the HDF5 files as n number(s) of two or three dimensional, 1-byte arrays. • The number of arrays is dependant on the number and size of each data product's quality flag definitions • Quality flags will be packed such that each flag is entirely contained within a single byte, occasionally resulting in a byte with reserved or meaningless bits • Byte alignment is the same for every quality flag array • First bit is the LSB, bytes are Big-Endian • Each byte may contain multiple bit-level flags

  18. 2-Dimensional Array Example

  19. 3-Dimensional Array Example

  20. Detailed NPOESS UML Models Ken Stone

  21. RDR UML Model

  22. Common RDR Layout

  23. SDR/TDR UML Model

  24. EDR UML Model

  25. Ancillary/Auxiliary UML Models

  26. NPOESS Sample Data Patrick Purcell

  27. Product Example UML Model

  28. HDF5 View - ImageView

  29. Product Profile Example

  30. HDF5 API – C++ Reader Sample Code Note: A granule holds a reference to a dataset region ... ... while an aggregation contains a reference to an object.

  31. NPOESS HDF5 Summary • The NPOESS Program delivers the official deliverable data products (RDR, SDR/TDR, EDR/ARP/IP) and dynamic ancillary data and auxiliary data in HDF5 Files • The HDF5 Files have an XML User Block that can be accessed without HDF5 tools - provides a “quick-look” into the metadata before opening the HDF5 file • Metadata within the HDF5 files are stored as attributes • There are general UML Models for the NPOESS official delivered data that provide a common framework • Official deliverable data products are organized by reference objects (aggregations) which contain one or more reference regions (granules) • Although data may be accessed directly through the All_Data group, the Data_Products group provides integrated access: • Allows the user to access both metadata and data through a common HDF5 group • Metadata is accessed directly by reading the Attribute values • Datasets may be accessed by dereferencing the object ID stored in the Data_Products Group for the aggregation or granule

More Related