Earth Science Data Types (ESDTs)

1. Earth Science Data Types (ESDTs) A.K.Sharma

2. ESDT Process: Creation through OPS Installation To Support Public Data Release START IT ECS SCI ECS Approval CMT CCB Desired Release Date Reports DAAC Mgmt. Review Distribute Release Sched. Receive Checklist Reports Summarize Checklist Reports Reports to CMT and CCB Confirm Valids Submission Create Release Schedule Define ESDT Requirements Direct ECS and ECS SCI ECS Elements Review Reports Release Schedule Plan DAAC Mgmt. Approval ESDT Priorities SE SDS OPS Coordination Checklist Reports Monitor Delivery Status Update DAAC Schedule Process Related TTs Obtain ESDT Confirm Arrival of Correct ESDT Copy ESDT to EDG DDMT Valids to EDG Test & Analyze Checklist Report to ECS SCI Schedule ESDT Install to Support Public Release ECS Deliver ESDT 4 Weeks Prior to Release Friday 3am Eastern SS EDG Submit Valids 2 Weeks Prior to Release EDG ESDT END ESDT & DDMT Valids For TS2, TS1, OPS: Receive ESDT Install ESDT Test Checklist Report to ECS SCI Datasets Available To Public On Release Date ESDT Review ODL Preparation Test Checklist Report to ECS SCI Submit Valids to EDG External to DAAC DAAC ECS Review/Approval

3. What are Metadata? • Metadata are descriptive information about the data, employing a common set of terms and definitions • An Earth Science Data Type (ESDT) is the interface between the science data and the ECS system How are Metadata Used in ECS? • Search and access of data • Inventory searches submitted by users via the ECS Client • Collection discovery via the Advertising Service • Other uses • Subscriptions based on metadata • Staging of input files based on metadata for Data Processing (e.g., time, location, version)

4. Overview: Role of ESDTs • Representation of data products from the Earth Scientist’s perspective • Earth Science Data Types (ESDTs) embody the core science data knowledge for data products contained in the EOS Data and Information System (EOSDIS) • Mechanism used to inform the Science Data Server about • Core and product-specific metadata attribute values for each collection and data granule • EOSDIS Core System (ECS) Data Server services (e.g., Insert, Search, Acquire) which can be invoked for the collection • One ESDT per collection • Before a new collection can be added to ECS • An ESDT must be created and submitted to Science Data Server (SDSRV), a component of the Data Server Subsystem (DSS)

5. Overview: Role of ESDTs (Cont.) • An ESDT, added to the SDSRV, defines to the ECS subsystems • Collection level metadata attributes and values • Granule metadata attributes, source of attribute values and attribute valid values • Product-Specific attributes and their valid values and ranges • Subscribable events • Services

6. Overview: Role of ESDTs (Cont.) • Each ESDT consists of two components • Descriptor file • ASCII file in Object Description Language (ODL) format describing metadata and services • ODL is an object-based language with “keyword = value” (keyword/value) statements that allows the class and attributes of an entity to be specified • ODL is used to convey relations between attributes and their characteristics within the descriptor files • Dynamic Link Library (DLL) • Shared object file coded in C++ • Handles the implementation of the specified services on each collection described in the Descriptor file

7. Descriptor File • All collection-level attributes are known ahead of time so their values can be specified in the descriptor file • The ESDT descriptor has six basic sections: • Collection level metadata information • Granule/Inventory level metadata information • Unparsed/Archive metadata information • Structure information which holds the CSDT (Computer Science Data Type) mapping - (the actual physical file organization and storage implementation) • ESDT service descriptions • ESDT supported event handlers

8. Product Specific Attributes • Optional Additional Attributes or Characteristics which the data provider makes available to further describe the Collection or Granule, e.g., • NUMBADDATA = INTEGER • NUMSPECIALDATA = INTEGER • NUMPROCESSDATA = INTEGER • NUMMISSINGDATA = INTEGER • note: A listing of the PSAs defined by other Instruments can be found at the following URL: http://ecsinfo.gsfc.nasa.gov/ECS_metadata/PSA/

9. Group = SERVICES • The ACQUIREobject specifies the characteristics of the Acquire service • Acquire service allows users to retrieve data granules • The INSERT object specifies the characteristics of the Insert service • The Insert service is used to store data granules in the archive • The UPDATEMETADATA object allows changing of already inserted values with new values • The BROWSE object allows access to browse images associated with a particular granule • The GETQUERYABLEPARAMETERS object allows access to a list of collection and inventory attributes defined within the ESDT descriptor file • The INSPECT object allows users to look at values from the Inventory metadata attributes • The INSPECTCL object allows users to look at values from the Collection metadata attributes

10. Group = SERVICES Cont.. • The DELETE object allows the removal of granules from the archive as well as the metadata • The DELETEFROMARCHIVE object allows the deletion of data granules from the archive but does not delete the associated metadata • Subsetting services are included as applicable for extracting information by: • Altitude (only) • Area (only) by bounding box • Area (only) by WRS Row, Path • Parameter (only) • Time (only) • Rows (Swath products only) • multiple means, area by WRS Row, Path • multiple means, area by Bounding Box • Apply Mask • Extract Narrow

11. Collection MetadataDLL Name libDsESDTSyBASIC.001Sh.soSpatial Search Type = RectangleCollection Description ClassShort Name = AIRAASCILong Name = AIRS/Aqua AMSU-A geolocated science countsCollection Description = AIRS/Aqua AMSU-A1 and AMSU-A2 combined geolocated counts for scene footprintsVersion ID = 1ECS CollectionRevision Date = 2001-09-24Processing Center = GSFCArchive Center = GSFCVersion Description = Beta DeliverySingle Type CollectionCollection State = In WorkMaintenance and Update Frequency = DailySpatialSpatial Coverage Type = HORIZ&VERTSpatial Domain ContainerVertical Spatial DomainVertical Spatial Domain ContainerVertical Spatial Domain Type = Minimum AltitudeVertical Spatial Domain Value = SFC

12. Vertical Spatial Domain Container Vertical Spatial Domain Type = Maximum AltitudeVertical Spatial Domain Value = TOA Horizontal Spatial Domain Container Bounding Rectangle West Bounding Coordinate = -180.000000North Bounding Coordinate = 90.000000East Bounding Coordinate = 180.000000South Bounding Coordinate = -90.000000 Coordinate System Container Horizontal Coordinate System Container Geographic Coordinate System Latitude Resolution = 0.250000Longitude Resolution = 0.250000Geographic Coordinate Units = Decimal Degrees Temporal Time Type = UTCDate Type = JulianTemporal Range Type = Continuous RangePrecision of Seconds = 4Endsat Present Flag = Y

13. Range Date Time Range Beginning Date = 1998-06-08Range Beginning Time = 00:00:00Range Ending Date = 1998-06-08Range Ending Time = 23:59:59 Contact Contact Person Contact Person Container Role = InvestigatorContact Instructions = Send email if there is a problem.Contact Job Position = AIRS AnalystContact First Name = DenisContact Middle Name = N/AContact Last Name = Elliott Contact Person Address Contact Person Address Container Street Address = 4800 Oak Grove DriveCity = PasadenaState Province = CaliforniaPostal Code = 91109-8099Country = USA

14. Telephone Telephone Container Telephone Number = 818-354-2117Telephone Number Type = Voice Email Electronic Mail Address = Denis.A.Elliott@jpl.nasa.gov Contact Organization Contact Organization Container Role = InvestigatorContact Instructions = Send email if there is a problem.Contact Organization Name = JPL AIRS Project Office Contact Organization Address Contact Organization Address Container Street Address = 4800 Oak Grove DriveCity = PasadenaState Province = CaliforniaPostal Code = 91109-8099Country = USA Organization Telephone Organization Telephone Container Telephone Number = 1-818-354-2117Telephone Number Type = Voice

15. Organization E-Mail Electronic Mail Address = Denis.A.Elliott@jpl.nasa.gov Discipline Topic Parameters Discipline Topic Term Variable 1 Earth Science Radiance or Imagery Microwave Sensor Counts DisciplineTopicParameters Processing Level Processing Level Description = Radiometric CountsProcessing Level ID = 1A Platform Platform Container Platform Short Name = AquaPlatform Long Name = First EOS Polar Orbiting Satellite, 1:30 PM Ascending Equator CrossingPlatform Type = Spacecraft

16. Platform Characteristic Platform Characteristic Container Platform Characteristic Name = EquatorCrossingTimePlatform Characteristic Description = Local time of the equator crossing and direction (ascending or descending)Platform Characteristic Data Type = varcharPlatform Characteristic Unit = Local Mean Time Platform Characteristic Value Class Platform Characteristic Value = 1:30 PM, ascending Platform Characteristic Container Platform Characteristic Name = OrbitInclinationPlatform Characteristic Description = inclined orbitPlatform Characteristic Data Type = floatPlatform Characteristic Unit = degrees Platform Characteristic Value Class Platform Characteristic Value = 98.2 Platform Characteristic Container Platform Characteristic Name = OrbitalPeriodPlatform Characteristic Description = orbital period for a 16-day, 233-orbit repeat cyclePlatform Characteristic Data Type = floatPlatform Characteristic Unit = minutes

17. Platform Characteristic Value Class Platform Characteristic Value = 98.88 Instrument Instrument Container Instrument Short Name = AMSU-AInstrument Long Name = Advanced Microwave Sounding UnitInstrument Technique = Cross-Track Scanning Multichannel Microwave SoundingNumber of Sensors = 15 Operation Mode Class Operation Mode = 1.      Normal 2.      Calibration 3.      Test  Instrument Characteristic Instrument Characteristic Container Instrument Characteristic Name = IFOVInstrument Characteristic Description = instantaneous field of viewInstrument Characteristic Unit = degreeInstrument Characteristic Data Type = float Instrument Characteristic Value Class Instrument Characteristic Value = 3.3

20. Sensors

25. Additional Product Specific Attributes Inventory Metadata ECS Data Granule Size MB ECS Data Granule = DOUBLELocal Granule ID = STRINGDay Night Flag = STRINGProduction Date Time = DATETIMELocal Version ID = STRING

26. Measured Parameter Measured Parameter Container Parameter Name = STRING QA Flags Automatic Quality Flag = STRINGAutomatic Quality Flag Explanation = STRINGOperational Quality Flag = STRINGOperational Quality Flag Explanation = STRINGScience Quality Flag = STRINGScience Quality Flag Explanation = STRING QA Stats QA Percent Missing Data = INTEGER Orbit Calculated Spatial Domain Orbit Calculated Spatial Domain Container Orbit Number = INTEGERStart Orbit Number = INTEGERStop Orbit Number = INTEGEREquator Crossing Longitude = DOUBLEEquator Crossing Time = TIMEEquator Crossing Date = DATE

27. Collection Description Class Short Name = AIRAASCIVersion ID = 1 Input Granule Input Pointer = STRING Spatial Domain Container Horizontal Spatial Domain Container Zone Identifier Class Zone Identifier = Other Grid System Bounding Rectangle West Bounding Coordinate = DOUBLENorth Bounding Coordinate = DOUBLEEast Bounding Coordinate = DOUBLESouth Bounding Coordinate = DOUBLE Range Date Time Range Beginning Time = TIMERange Ending Time = TIMERange Beginning Date = DATERange Ending Date = DATE PGEVERSIONCLASS PGE Version = STRING

28. Associated Platform Instrument Sensor Sensor Platform Name Platform Name Instrument Name Instrument Name Sensor Name M Aqua STRING AMSU-A STRING STRING Sensors Associated With This Product Product Specific Metadata NUMBADDATA = INTEGERNUMSPECIALDATA = INTEGERNUMPROCESSDATA = INTEGERNUMMISSINGDATA = INTEGER

29. NUMLANDSURFACE = INTEGERNUMOCEANSURFACE = INTEGERNUMGEOQA = INTEGERORBITPATH = INTEGERSp Starting Path = INTEGERSp Starting Row = INTEGERSp Ending Row = INTEGERSCANLINECOUNT = INTEGER NUMTOTALDATA = INTEGERAIRSGRANULENUMBER = INTEGERAIRSGRANULECYCLENUMBER = INTEGERNODETYPE = STRINGPRODUCTGENERATIONFACILITY = STRINGNUMFPE = INTEGERNUMSUNGLINT = INTEGERLONGRANULECEN = INTEGERLATGRANULECEN = INTEGERLOCTIMEGRANULECEN = INTEGER

30. Metadata File (.met) • Metadata File (.met) • Built around the 'parameter = value' form • Provides the inventory level and unparsed metadata attribute values for each data granule file; this information is used by the SDSRV to populate the inventory metadata database • Each inventory and unparsed metadata element in the .met is fully described in the MCF • If the attribute is not defined in the MCF, it can not be set in the .met. • .met File Naming Convention • Ought to end with “.met” • Example: AIRS_L1b20000524190442v1.met

31. Ingest Granule Metadata Generation Source Metadata Configuration File Target Metadata Configuration File Inventory Tables Science Data Server Inventory Metadata (ASCII File) Metadata File ECS Ingest Processing Data Granule

32. Processing Granule Metadata Generation (Generated by Science Data Server) Inventory Metadata (Generated by Processing) Target Metadata Configuration File (ASCII File) Process Control File Product SDP Toolkit Inventory Metadata Archive Metadata Product Generation Executable Scientific Data Set Scientific Data Set Inputs (HDF-EOS)

33. ECS Data Distribution Metadata (ASCII File with .met extension) Metadata Tables To User Data Granule Data Server Subsystem Storage Management