UTG4 – Archiving and processing planetological data Simone Silvestro

1. UTG4 – Archiving and processing planetological data Simone Silvestro INAF Osservatorio Astronomico di Capodimonte, Napoli

2. Introduction: INAF instrument & missions Data processing: USGS ISIS & INAF pipelines Archciving: NASA PDS/ESA PSA & others

3. Introduction 15 Instruments (11 missions) Ongoing Terminated Future 1) VIR 2) VIRTIS (1) 3) VIRTIS (2) 1) MAJIS 2) JANUS 3) Ma-MISS 4) Micro-MED 5) SYMBIO-SYS 1) NOMAD 2) CaSSIS 3) MARSIS 4) PFS 5) EFD 6) JIRAM ExoMars 2016: NOMAD, CaSSIS Mars Express: PFS, MARSIS ExoMars 2020: Ma-MISS, Micro-MED Cassini: VIMS JUICE: MAJIS, JANUS Venus Express: VIRTIS (1) Rosetta: VIRTIS (2) Juno: JIRAM BEPI-Colombo: S-SYS DAWN: VIR CSES: EFD

4. Introduction Different bodies Mars Jupiter & Satellites Saturn & Satellites Venus Asteroids & Comets Earth Mercury 1) NOMAD 2) CaSSIS 3) PFS 4) MARSIS 5) Ma-MISS 6) MicroMED 1) JIRAM 2) MAJIS 3) JANUS 1) VIRTIS (1) 1) VIR 2) VIRTIS (2) 1) VIMS 1) EFD 1) SYMBIOSYS

5. Introduction Instrument types Radar GPR Camera/Spectrometer Others 1) NOMAD 2) CaSSIS (stereo) 3) PFS 4) Ma-MISS 5) JIRAM 6) MAJIS 7) JANUS (stereo) 8) VIRTIS (1) 9) VIRTIS (2) 10) VIR 11) SYMBIO-SYS (stereo) 12) VIMS 1) Micro-MED 2) EFD 1) MARSIS

6. Introduction: INAF instrument & missions Data processing: USGS ISIS & INAF pipelines

7. Processing Processing pipelines *USGS ISIS/Propietary pipeline ONLY propietary pipeline/ISIS TBD 1) VIRTIS (1) – Venus Exp. 2) VIRTIS (2) – Rosetta 3) VIR – DAWN 4) CaSSIS – TGO ExoMars 2016 1) MAJIS - JUICE 2) JANUS - JUICE 3) JIRAM - Juno 4) NOMAD – TGO ExoMars 2016 5) Ma-Miss – ExoMars 2020 6) Micro-MED 7) PFS – Mars Express 8) MARSIS – Mars Express 9) SIMBIO-SYS – Bepi Colombo 10) EFD - CSES *Integrated Software for Imagers and Spectrometers

8. Processing Propietary pipelines • INAF/IAPS: • NOMAD (internal Python scipts) • PFS (internal tools) • MARSIS (in-house proc. pipeline – Matlab scripts) • VIR (internal propietary calibration pipeline) • VIRTIS (1) (in-house proc. pipeline, Python reading software freely available) • VIRTIS (2) (IDL processing pipeline) • INAF/OAPD: • CASSIS (high-level products – digital terrain models – GIS interface for request/targeting) • SIMBIO-SYS (high-level products – digital terrain models)

9. Processing USGS ISIS

10. Processing USGS ISIS Example of a processing pipeline in USGS ISIS • Three processing steps valid for most cameras/spectrometers: • Level 0 – Data ingestion • Acquire and convert image data files (PDS format) to the ISIS3 image format (cubes) • Add information to the ISIS3 image in order to compute geometric properties such as latitude/longitude range and illumination angles of the image • Level 1 (A and B) – Radiometric Calibration and Noise Removal • Convert raw pixel numbers to reflectance (irradiance/solar flux or I/F) • Remove noise • Level 2, 3 – Projection • Geometrically rectify to a map projection

11. Processing NASA processing levels • 0–reconstructed, unprocessed instrument and payload data at full resolution, with artifacts removed • 1A – reconstructed, unprocessed instrument data at full resolution, time-referenced, and annotated with ancillary information, including radiometric and geometric calibration coefficients and georeferencing parameters (e.g., platform ephemeris) computed and appended but not applied to Level 0 data • 1B – level 1A data that have been processed to sensor units • 2 – derived geophysical variables at the same resolution and location as Level 1 source data • 3 – variables mapped on uniform space-time grid scales, usually with some completeness and consistency • 4– model output or results from analyses of lower-level data (e.g., variables derived from multiple measurements)

12. Introduction: INAF instrument & missions Data processing: USGS ISIS & INAF pipelines Archciving: NASA PDS/ESA PSA & others

13. Archiving • Global repositories • NASA PDS • ESA PSA • Local repositories • ASI SSDC • INAF • Others • CNSA • IKI ROSCOSMOS Allthe data produced by the instruments listed in this ppt should be NASA PDS/ESA PSA compliant

15. Archiving NASA PDS – 8 main nodes 6 science discipline nodes 2 support nodes

18. Archiving PDS Standards • PDS Data product = Data object + label file describing the object • Data files: ASCII or binary format • Primary data object: set of results from a scientific observation (TABLE, SPREADSHEET, IMAGE, SERIES, SPECTRUM, QUBE) • Secondary data object: any data used for processing or interpreting the primary data object(s) (HISTOGRAM, PALETTE, HEADER) • Label files: written in Object Description Language (ODL) in the format «keyword = value», defines both the physical and logical structure of the constituent data object(s)

19. /* File Format and Length */ RECORD_TYPE = FIXED_LENGTH RECORD_BYTES = 800 FILE_RECORDS = 860 /* Pointer to First Record of Major Objects in File */ ^IMAGE = 40 ^IMAGE_HISTOGRAM = 840 ^ANCILLARY_TABLE = 842 /* Image Description */ SPACECRAFT_NAME = VOYAGER_2 TARGET_NAME = IO IMAGE_ID = "0514J2-00" IMAGE_TIME = 1979-07-08T05:19:11Z INSTRUMENT_NAME = NARROW_ANGLE_CAMERA EXPOSURE_DURATION = 1.9200 NOTE = "Routine multispectral longitude coverage, 1 of 7 frames" /* Description of the Objects Contained in the File */ OBJECT = IMAGE LINES = 800 LINE_SAMPLES = 800 SAMPLE_TYPE = UNSIGNED_INTEGER SAMPLE_BITS = 8 END_OBJECT = IMAGE OBJECT = IMAGE_HISTOGRAM ITEMS = 25 ITEM_TYPE = INTEGER ITEM_BITS = 32 END_OBJECT = IMAGE_HISTOGRAM OBJECT = ANCILLARY_TABLE ^STRUCTURE = "TABLE.FMT" END_OBJECT = ANCILLARY_TABLE END PDS label example Primary data object Secondary data object

20. Archiving PDS Standards • Processing levels • 1– raw data: data with telemetry embedded • 2 – edited data: corrected for telemetry error, tagged with time and location of acquisition, expressed in instrument units, also called Experimental Data Record (EDR) correpsonds to NASA level 0 data • 3 – calibrated data: data values expressed in physical units, no resampling, edited data can still be reconstructed (NASA level 1A) • 4 – resampled data: data resampled in space/time and calibrated, also called Reduced Data Record (RDR), edited data cannot be reconstructed (NASA level 1B) • 5– derived data: derived results (maps, reports, graphics, etc. NASA Levels 2 through 4). • 6, 7, 8 – ancullary/correlative data and user descriptor needed to fully carachterize and process the data product(s)

21. Archiving ESA Planetary Science Archive (PSA)

22. Archiving ESA Planetary Science Archive (PSA)

23. Archiving ASI Space Science Data Center (SSDC)

24. Archiving ASI SSDC- MATISSE Web based tool to access and visualize data from planetary exploration instruments

25. Archiving Summary • Future missions • SIMBIO-SYS (Bepi-Colombo): raw and calibrated data on ESA PSA, high-level product(s) on ASI SSDC • Ma-MISS: raw and calibrated data on ESA PSA, NASA PDS and IKI ROSCOSMOS • MicroMed: TBD • MAJIS/JANUS (JUICE): calibrated data on ESA PSA, raw and high-level data product(s) on local rep. (INAF/IAPS or ASI SSDC) • Ongoing missions • NOMAD (ExoMars TGO): non-calibrated data on local rep. (INAF/IAPS), calibrated data on ESA PSA • CASSIS (ExoMars TGO): raw and calibrated data on ESA PSA/NASA PDS, high-level product(s) (DTMs) o local rep. (INAF/OAPD) • PFS (MarsExpress): raw data on local rep. (INAF/IAPS), calibrated data on ESA PSA/NASA PDS • MARSIS (MarsExpress): EDR and RDR files on ESA PSA/NASA PDS, raw data at INAF/IAPS • JIRAM (JUNO): EDR/RDR files on the NASA PDS, high-level producs on ASI SSDC • EFD (CSES): raw/processed data at ASI SSDC downloaded from CNSA • Terminated missions • VIR (DAWN): processed data at NASA PDS, raw at INAF/IAPS • VIRTIS (1) (Venus Express): raw/calibrated data at ESA PSA • VIRTIS (1) (Rosetta): raw/calibrated/high-level product(s) at ESA PSA and NASA PDS (ESA requested the availability of high-level products – spectral and temp. maps – on the PSA) • VIMS (Cassini): raw data on the NASA PDS with documentation. Calibrated data not available

26. Archiving Summary • Future missions • SIMBIO-SYS (Bepi-Colombo): raw and calibrated data on ESA PSA, high-level product(s) on ASI SSDC • Ma-MISS: raw and calibrated data on ESA PSA, NASA PDS and IKI ROSCOSMOS • MicroMed: TBD • MAJIS/JANUS (JUICE): calibrated data on ESA PSA, raw and high-level data product(s) on local rep. (INAF/IAPS or ASI SSDC) • Ongoing missions • NOMAD (ExoMars TGO): non-calibrated data on local rep. (INAF/IAPS), calibrated data on ESA PSA • CASSIS (ExoMars TGO): raw and calibrated data on ESA PSA/NASA PDS, high-level product(s) (DTMs) o local rep. (INAF/OAPD) • PFS (MarsExpress): raw data on local rep. (INAF/IAPS), calibrated data on ESA PSA/NASA PDS • MARSIS (MarsExpress): EDR and RDR files on ESA PSA/NASA PDS, raw data at INAF/IAPS • JIRAM (JUNO): EDR/RDR files on the NASA PDS, high-level producs on ASI SSDC • EFD (CSES): raw/processed data at ASI SSDC downloaded from CNSA • Terminated missions • VIR (DAWN): processed data at NASA PDS, raw at INAF/IAPS • VIRTIS (1) (Venus Express): raw/calibrated data at ESA PSA • VIRTIS (1) (Rosetta): raw/calibrated/high-level product(s) at ESA PSA and NASA PDS (ESA requested the availability of high-level products – spectral and temp. maps – on the PSA) • VIMS (Cassini): raw data on the NASA PDS with documentation. Calibrated data not available

27. Archiving High-level data products DTMs Spectral maps Geological maps

28. Archiving Propietary pipelines • INAF/IAPS: • NOMAD (internal Python scipts) • PFS (internal tools) • MARSIS (in-house proc. pipeline – Matlab scripts) • VIR (internal propietary calibration pipeline) • VIRTIS (1) (in-house proc. pipeline, Python reading software freely available) • VIRTIS (2) (IDL processing pipeline) • INAF/OAPD: • CASSIS (high-level products – digital terrain models – GIS interface for request/targeting) • SIMBIO-SYS (high-level products – digital terrain models)

29. Archiving CaSSIS – Data repository @ OAPD • Each Monday data download from mirror in UaBern • Automatic shape files generation of the acquired FoVs

30. Archiving CaSSIS – Data repository @ OAPD Data extraction with GIS tools • GIS software (ArcGIS, QGIS) allows very fast data visualization, extraction and exploitation • In the context of the CaSSIS mission, there are a large amount of data which will be impossible to exploit without the aid of GIS-based tools • White polygons are some of the footprints of all the CaSSIS observations, which have been collected in a shapefile (i.e a database of observations) • The shapefile can be queried, in order to rapidly extract observations at a given location (red dot) • Queries can be very specific (i.e. filters, incidence, emission, phase, local time), in order to adapt to the scientific requirements • The shapefile gathers all the CaSSIS data in one visualization, which can be integrate with data from other instrument on Mars, thus allowing the science team to fully exploit all existing data

31. Archiving CaSSIS – Data repository @ OAPD Data extraction with GIS tools • GIS software (ArcGIS, QGIS) allows very fast data visualization, extraction and exploitation • In the context of the CaSSIS mission, there are a large amount of data which will be impossible to exploit without the aid of GIS-based tools • White polygons are some of the footprints of all the CaSSIS observations, which have been collected in a shapefile (i.e a database of observations) • The shapefile can be queried, in order to rapidly extract observations at a given location (red dot) • Queries can be very specific (i.e. filters, incidence, emission, phase, local time), in order to adapt to the scientific requirements • The shapefile gathers all the CaSSIS data in one visualization, which can be integrate with data from other instrument on Mars, thus allowing the science team to fully exploit all existing data

32. Conclusions Most of the instruments are cameras/spectrometers, each institute has developed different processing pipelines. Documentation on how processing the data can be found on PDS/PSA There is not a common policy on how to release high-level data producst (different missions = different policies) High-level data products are stored on different repositories (in INAF or at the ASI SSDC) My personal opinion... It would be nice to have, at least for the high-level data products a common repositoriy (in INAF) where data could be easily accessed A user-friendly (GIS-based?) interface (NASA Giovanni, JMARS, Mars Dune/Crater Database) would help to disseminate the usage and outcome from the space missions where INAF is involved

33. NASA Giovanni

34. JMARS: Java Mission-planning and Analysis for Remote Sensing (ASU) Available for Earth, Ceres and Vesta too!

35. Mars Global Digital Dune Database (MGD3)