Data and Metadata Integrity

1. Data and Metadata Integrity AIRS DST Roles Monitor AIRS Products generation Monitor AIRS Distribution Monitor AIRS Subscription Operational QA flag Science QA flag QA Update Metadata Check DFA – duplicate granules

2. AIRS DST Primary & Secondary Roles Primary role: Our primary role with respect to Science Quality Assessment (QA) is updating the inventory metadata to reflect the AIRS Science Team’s interpretation of the science quality of the data managed in our archives. Secondary role: Our secondary role involves ensuring the basic usability of the data by evaluating appropriate tools in the public domain and providing user-support. This is especially relevant, for instance, in the context of four-dimensional data products that cannot be handled by a majority of the COTS and custom software packages. We are open to the idea of providing useful feedback from our end-users and customers which would be relayed to the Science Teams via our established User-Support model.

3. AIRS DST Role in Ensuring the Basic Usability of the Data • Following successful production, ingest & archival of the data products, we ensure: • AIRS Data products are visible and orderable via the WHOM and EDG user interfaces • Integrity of the product & its metadata are not compromised • Data products are functional and usable • All the appropriate technical & science links on-line • Provide information on public-domain & COTS software • Keep ECS-Science and GDAAC User Services informed • Keep the Science Teams in the loop

4. Issues to monitor in the near future • ECS-EDG Profiles; Seamless User Registration, integration of summary browse functionality • Restricted Granule Access; Temporal restriction flag • Delete from Archive/Delete-Effective Date changes • ESDT update capability; Mandatory status implications • QA Flag :: QA Stats :: QA Data Planes – Logical mapping • AIRS Processing and Reprocessing • AIRS Products Public Release

5. The AIRS Data Support Team assists the Data Producer by performing data integrity checks. These checks fall into three categories: production, ingest and distribution checks. • Production • Production tools are helpful in checking the production at each stage. Data gaps due to data/system problems can be determined by these tools. Data integrity checks are also performed in the production stream and metadata validation to identify problems with science algorithms and corrupted data, and to ensure proper database population of attributes. • Data Ingest • Data ingest checks includes metadata checks for file size, data time, and production time. • Distribution • Distribution tools are designed to help many aspects of distribution system including the statistics of the data distributed to different users on different media. These tools are helpful in determining the most popular data products and media.

6. Data production and distribution tools are listed as: • AIRS L0 Data Monitor • AIRS Data Production Monitor • AIRS Data Production Request Monitor • AIRS Shipping Monitor • AIRS Data Distribution Monitor • AIRS Data Archive Monitor • AIRS Data Inventory Monitor • Other QA Support • Application of MUSE/QAMUT (QA Metadata Update Tool) • Identifying problems through tool development and team player’s knowledge of the data

7. QA Metadata Update Scripts (MUSE) MUSE is a C-Shell script with a number of built-in menus that allow querying and identifying granules before performing QA updates to either the Science or the Operational flags and their Explanations in the metadata tables. These updates are performed via SQL commands directly interacting with the Science Data Server database tables. MUSE can be run from any directory, but it needs to have a subdirectory called Muse_Files, which contains four files: MUSE_Rc MUSE_QUERY.sql MUSE_UPDATE.sql MUSE_QFCHECK.csh

8. QA Metadata Update Scripts (MUSE) Continue MUSE MAIN MENU 1. Load Saved Query Values 2. Set SYBASE Values 3. Set Query Values 4. Run Query 5. Quit The Set SYBASE Values sub-menu has the following options: MUSE MAIN MENU 1. Load Saved Query Values 2. Set SYBASE Values 3. Set Query Values 4. Run Query 5. Quit The Set SYBASE Values sub-menu has the following options: MUSE SYBASE MENU 1. Set Server Name (Sybase server for the SDSRV database) 2. Set Database Name (SDSRV database Ex: EcDsScienceDataServer1) 3. Set Login / Password (Sybase login & password with update privileges) 4. Set Working Directory (Directory location for writing user output files Ex: /home/bhaskar/MUSE/work) 5. Set Quality File Directory (Directory location for the quality explanation text files Ex: /home/bhaskar/MUSE/Quality_Files) 6. Return to Main Menu MUSE SYBASE MENU 1. Set Server Name (Sybase server for the SDSRV database) 2. Set Database Name (SDSRV database Ex: EcDsScienceDataServer1) 3. Set Login / Password (Sybase login & password with update privileges) 4. Set Working Directory (Directory location for writing user output files Ex: /home/bhaskar/MUSE/work) 5. Set Quality File Directory (Directory location for the quality explanation text files Ex: /home/bhaskar/MUSE/Quality_Files) 6. Return to Main Menu

9. QA Metadata Update Scripts (MUSE) Continue The Query values are set via the Set Query Values sub-menu with the following options: MUSE QUERY MENU • 1. Set ShortName (ESDT Short Name) • 2. Set VersionID (ESDT Version ID) • 3. Set Quality Flag Value (Qualifier: Passed/Failed …) • 4. Set Quality Flag Explanation File (Name of the text file in the Quality File Directory which contains the quality explanation narrative. Ex: Explanation.txt • 5. Set Quality Flag Type (S = Science, O = Operational) • 6. Set Acquisition Range (If entered, the query can be restricted by a beginning and ending acquisition date range. Ex: Jan 1 2000 12:01 AM – Dec 31 2000 11:59 PM) • 7. Set LocalGranuleId (If entered, the query can be restricted to a single . LGID or multiple LGIDs using the “%” wildcard. The default is ALL LGIDs) • 8. Restore Default Values (5-7) (Restores default values to options 5 through 7) • 9. Return to Main Menu • 4. Set Quality Flag

10. QA Metadata Update Scripts (MUSE) Continue When the query is run via Run Query on the Main Menu, the following results menu is displayed: MUSE RESULTS MENU 1. Save Query Parameters(Allows the user to save the user-defined query parameters which can be loaded and run at a later time) 2. Review Query Results On-line(QA Flag and Explanation values before update) 3. Run the Update (Updates the SDSRV database for each granuleID – ParameterName combination returned in the query) 4. Review Update Results On-line (QA Flag and Explanation values after update) 5. Return to Main Menu

11. QA Metadata Update Future Enhancements • Supporting multiple LGIDs as input (up to 50,000 LGIDs) • Functionality to handle multiple Measured Parameters • Error handling capabilities • Log writing and maintenance capabilities • Supporting MUSE to run as a Stored Procedure • Adding a Graphical User Interface

12. Key AIRS/AMSU/HSB Data Products • Temperature profiles through the atmosphere • Temperature profiles through the atmosphere • Water vapor profiles through the atmosphere • Total surface • Water vapor profiles through the atmosphere • Total precipitable water, cloud liquid-water content, precipitation indication, cloud-ice indication • Heights of the tropopause and stratopause • Cloud-top pressure and temperature, fractional cloud cover, cloud spectral properties, and cloud type • Ozone profile through the atmosphere • Total column atmospheric ozone, methane, carbon monoxide, and carbon dioxide • Sea and land surface skin temperature • Infrared and microwave surface emissivity • Longwave and shortwave radiative fluxes at the top of the atmosphere and at the surface

13. Aqua’s Delta II Rocket, Ready for Launch Rendering by Reto Stockli