1 / 16

Tool Working Group Report

This report provides an overview of the Tool Working Group's activities, including updates on core PDS tools development, prioritization of activities, and coordination efforts. It also includes information on recent meeting discussions, tool presentations, and plans for future development.

lavery
Download Presentation

Tool Working Group Report

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Tool Working Group Report PDS MC F2F Meeting November 8, 2018

  2. Topics • Overview • October 2018 Meeting Report • November Plans Tool Working Group Report

  3. Overview • EN focuses on developing and maintaining core PDS tools • This includes generation, validation, transformation and inspection of PDS products • Updates are released every six months on the current build schedule • Point builds are released periodically when a major issue is addressed • Tool Working Group (TWG) is prioritizing development activities and working PDS-wide coordination. Tool Working Group Report

  4. TWG Members • D. Crichton, Chair • E. Law as alternate • E. Guinness, Geosciences • Joni Johnson, Atmospheres • Todd King, PPI • Rishi Verma, IMG • Eric Palmer, PSI • Tanya Lim, PSA • Matt Tiscareno, RMS • Michael Cayanan, EN as ex-officio advisor Tool Working Group Report

  5. TWG Activities • The group is tracking status for actively developed tools at the EN and the DNs • Demos of DN tools raises awareness to other Nodes (e.g., OLAF, ELSA) • A traceability matrix is maintained to map Level 3 requirements to tools • Prioritizations for validation and transformation determined • New requirements and functionality will be run by the group • All of the above is captured in the following Google doc: • https://docs.google.com/spreadsheets/d/18oqtg3DEo2KrgvBOWLSOuqF2uZtq2XmByJwUknYSZUQ/edit#gid=1340660799 • The group is actively involved in beta testing of the core tools • Plan to incorporate these tests into the regression test suite where appropriate Tool Working Group Report

  6. October 2018 Meeting Report • Tool Presentation by PPI Node • Discussed Build 9a Tool Status • Discussed development prioritizations for Validate Tool and Transform Tool Tool Working Group Report

  7. PPI Node Tool Presentation • PPI presented a set of pds4-tools: • pds-collection-builder: extracts information from PDS4 labels in a directory and generates a collection inventory • pds-doi-request: extracts information from a PDS4 label and generates a DOI request which can be used with Interagency Data (IAD) web services • pds-ldd-doc: generates Github Flavored Markdown (GFM) documentation for a Local Data Dictionary specification file • Available on github • https://github.com/nasa-pds/pds4-tools Tool Working Group Report

  8. Build 9a Tool Status • Build 9a was completed on 9/28/2018 • TWG participated in the Beta Testing of the tools [1] • Validate Tool release includes: • Enhanced reporting • Support for objects greater than 2GB • Addressed number of issues related to content validation • Transform Tool release includes: • Initial support for transforming PDS4 images to FITS • Addressed number of issues handling various PDS3 to PDS4 transformations • Inspect Tool • Made available to the TWG as a prototype • Official release planned for end of year 2018 [1] https://pds-engineering.jpl.nasa.gov/content/build_9a_tool_test_information Tool Working Group Report

  9. Validate Tool Prioritizations • Build 9b • Add spot checking of content validation (skip every nth record) • Add check for un-labeled files in a bundle • Add check that logical identifiers are formed correctly • products have their parent collection LID as its base • collections have their parent bundle LID as its base • Add check for existence of referenced local identifiers within a label • Build 10a • Add spot checking of content validation (skip every nth file) • Add Context Collection validation rules • Add Collection Inventory table validation Tool Working Group Report

  10. Transform Tool Prioritizations • Build 9b • Add Array_3D_Image to FITS transformation • Finish wrapping Transform Tool as a web service • Updated the Transform Tool priority matrix [1] for feedback with Nodes on prioritizations moving forward • Received feedback from Geosciences Node so far [1] https://docs.google.com/spreadsheets/d/18oqtg3DEo2KrgvBOWLSOuqF2uZtq2XmByJwUknYSZUQ/edit#gid=635102082 Tool Working Group Report

  11. November Plans • Discuss Transform Tool priorities • Discuss Inspect Tool Node testing • Discuss Validate Tool priorities Tool Working Group Report

  12. Backup

  13. Validate ToolChanges since Build 8b in Point Build • PDS-549: Added a summary of messages section to the reports • PDS-556: Support for Arrays greater than 2GB • PDS-557: Updated to read in only the portion of the data file corresponding with the object description • Updated to validate against version 1A10 of the IM • PDS-563: Fixed issue where it was incorrectly flagging negative float values as errors during table content validation • PDS-564: Fixed to handle gap datetimes during data content validation • PDS-566: Updated to support the new start_bit_location/stop_bit_location elements based on changes to version 1A00 of the IM • PDS-568: Updated to no longer do field format validation for binary tables • PDS-570: Support multiple collection products in the same directory • PDS-571: Remove checking of valid files and directories at the root of a bundle • PDS-572: Disable check that every file in a bundle is referenced by a label • SW-31: Support ASCII_Real values without leading zeroes (due to CCB-213) • SW-32: Support date time values with microseconds precision (due to CCB-226) Tool Working Group Report

  14. Transform ToolChanges since Build 8b • PDS-493*: Support multi-line comments in target PDS3 labels • PDS-500*: Support parenthesis in description fields for target PDS3 labels • PDS-539: Support explicit FILE objects for target PDS3 labels • PDS-540: Gracefully handle unfound data objects for target PDS3 labels • PDS-541: Gracefully handle missing RECORD_BYTES during PDS3 to PDS4 label transformations • PDS-551*: Fixed an issue transforming MRO HiRISE image products • PDS-553*: Support SFDU statements in PDS3 labels when doing PDS3 image transformations • PDS-560: Correctly map bit columns to field bit elements for PDS3 to PDS4 label transformations due to a change in version 1A00 of the IM • PDS-566: Updated to support new start_bit_location/stop_bit_location elements due to change in version 1A00 of the IM * Modifications to the underlying VICAR IO library. Tool Working Group Report

  15. Completed Transformations Tool Working Group Report

  16. TO-DO Transformations Tool Working Group Report

More Related