1 / 24

Workshop on Research Metadata in Context – Sept 7/8, Nijmegen, Netherlands

Metadata Frameworks in EUFAR Matt Freer, EUFAR. Workshop on Research Metadata in Context – Sept 7/8, Nijmegen, Netherlands. Overview. EUFAR Introduction Airborne Science Metadata Overview EUFAR Implementation Summary. EUFAR Integrating Activity of the EC FP7

malory
Download Presentation

Workshop on Research Metadata in Context – Sept 7/8, Nijmegen, Netherlands

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Metadata Frameworks in EUFAR Matt Freer, EUFAR Workshop on Research Metadata in Context – Sept 7/8, Nijmegen, Netherlands

  2. Overview • EUFAR Introduction • Airborne Science Metadata Overview • EUFAR Implementation • Summary

  3. EUFAR Integrating Activity of the EC FP7 Budget 8 M€ Duration 4 years (2008-2012) 33 Partners 7 instruments and 22 aircraft open to Trans-national Access www.eufar.net

  4. Jülich (DE)‏ ULEI (DE)‏ STFC (UK)‏ USZ (HU)‏ UCAM-DCHEM (UK)‏ UWAR (PL)‏ COSINE (NL)‏ IRSN (FR)‏ COMAT (FR)‏ VKI (BE)‏ UZH (CH)‏ WU (NL)‏ USBE (CZ)‏ TAU (IL)‏ GFZ (DE)‏ PML (UK)‏ PTB (DE)‏ ONERA (FR)‏ Météo-France (FR) Coordinator MetOffice (UK) DLR (DE)‏ NLR (NL)‏ Enviscope (DE)‏ CNRS (FR)‏ NERC (UK)‏ INTA (ES)‏ FUB (DE)‏ KIT (DE)‏ AWI (DE)‏ CNR (IT)‏ UNIMAN (UK)‏ VITO (BE)‏ 14 aircraft or instruments operators and 18 experts in airborne measurements EUFAR Introduction

  5. EUFAR Introduction EUFAR objectives: • Provide scientists with access at equal terms to the most complete range of research infrastructures • Develop trans-national access to national infrastructures • Reduce redundancy and fill the gaps • Promote the use of research infrastructures, particularly among young scientists from countries where such infrastructures are lacking • Strengthen expertise through exchange of knowledge, development of standards and protocols, construction of databases and joint instrumental research activities

  6. Airborne Metadata Overview • Aircraft in EUFAR measure large range of atmospheric data: • Aircraft parameters • Altitude/Lat/Long • Roll/Pitch/Yaw • State parameters • Temperature • Pressure • Humidity • Chemistry • CO2 • O3 • NOx • Microphysics • Water content • Ice crystal habits • Particle concentration • Particle size distributions • Radiation • Long wave • Short wave • Winds • Hyperspectral imaging • Land use • Reflectivity

  7. Airborne Metadata Overview Challenges: Heterogeneity: Instruments Measurements Data systems Processing techniques Data streams Collaborations on projects, research and data production between varied countries and institutions (Europe, USA, etc) Many different levels of data (raw, intermediate, processed, final, etc)‏ All of these necessitate clearly defined, robust protocols for describing data

  8. Airborne Metadata Overview External metadata Gives users broad overview of data Location Project Responsible party Quality Aids in data classification, search, retrieval, archival Embedded metadata Makes files self-describing Can define metadata globally or per-variable Reduces errors in usage Helps automate data processing tasks Stores logistical information (calibration, maintenance and other historical sensor information)‏

  9. EUFAR Metadata Conventions • Draft version of proposed conventions has been created and circulated to members for comment • At the International Conference on Airborne Research in the Environment (ICARE) in Toulouse, Oct 2010, version 1.0 of these standards will be finalized • EUFAR proposes: • External • INSPIRE • Embedded • NetCDF • CF Conventions • IWGADTS

  10. External Metadata – INSPIRE Directive • Infrastructure for spatial information in Europe (INSPIRE)‏ • Directive entered into force May 2007 by the European Commission • Ensures compatibility and usability of spatial data infrastructures with the adoption of common implementing rules (e.g. Metadata)‏ • Implementation rules available at http://inspire.jrc.ec.europa.eu • INSPIRE Principles • Data should be collected only once and kept where it can be maintained most effectively • Spatial information should be combined into seamless product • Data should be scalable to multiple levels – detailed enough for investigation, general enough for strategic purposes • Geographic information should be readily available at all levels • Available data, metadata and usage conditions should be clear and easy to find

  11. INSPIRE Metadata elements • INSPIRE Metadata regulation mainly based on ISO 19155-2 • Includes Information on • Data set or series • Classification (scientific field)‏ • Geographic location • Temporal reference • Quality and validity • Conformity • Constraints related to access and use • The responsible organisation Descriptions of different image and instrument processing steps List has been expanded within EUFAR by additional quality indicators

  12. INSPIRE Metadata rules implementation into EUFAR (1)‏ Identification • Resource title: EUFAR project acronym • Resource abstract: EUFAR abstract • Resource type: dataset or series • Resource locator: www.eufar.net • Unique resource identifier: EUFAR file name specification • Coupled resource: not applicable • Resource language: eng Additional: • List of related available files • Platform: aircraft all sign • Instrument: list of instruments incl. basic sensor characteristics (e.g. spectral coverage)‏

  13. INSPIRE Metadata rules implementation into EUFAR (2)‏ • Classification • Topic category: EUFAR main scientific field • Spatial data service type: not applicable • Keyword • Keyword value: EUFAR scientific theme • Originating controlled vocabulary:GEMET - INSPIRE themes, version 1.0 2008-06-01 • Geographic Location • Geographic bounding box Additional: • Area information: Country, Province/Region, Local name • Temporal Reference • Temporal extent: date of data acquisition • Date of publication: date of entry into EUFAR-DB • Date of last revision: date of last revision • Date of creation: date of data processing

  14. INSPIRE Metadata rules implementation into EUFAR (3)‏ Quality and Validity • Lineage • Spatial resolution Additional: • Processing information (e.g. processing level)‏ • Calibration information (e.g. date of spectral calibration)‏ • Acquisition information (e.g. overall heading)‏ • Data quality layers for hyperspectral imagery • Sensor calibration and system correction • Image data artefacts and processing errors • GPS / IMU related errors, geometric correction • Atmospheric correction and atmospheric conditions quality control flags attitude and position hyperspectral image black body information, …

  15. INSPIRE Metadata rules implementation into EUFAR (4)‏ • Conformity • Specification: INSPIRE implementing rules • Degree: degree of conformity (true or false)‏ • Constraints related to access and use • Conditions applying to access and use: EUFAR consortium agreement • Limitations on public access: EUFAR consortium agreement • Organisation responsible for the establishment, management, maintenance and distribution • Responsible party: bureau@eufar.net • Responsible party role: pointOfContact • Metadata on metadata • Metadata point of contact: e.g. DLR, opairs@dlr.de, PAF • Metadata date: date when the metadata record was created or updated • Metadata language: eng

  16. Embedded metadata Tangent: EUFAR recommended file format NetCDF (network Common Data Form)‏ Wide use in atmospheric science community APIs for most common programming languages (C, C++, Fortran, Java, Python, Matlab, IDL) NetCDF is: Binary Portable Scalable: Multi-dimensional array oriented Subsets of large datasets easily accessible Appendable Self describing Data model: Dimensions Variables Attributes

  17. Embedded metadata • Tangent: NetCDF (network Common Data Form)‏ • NetCDF is: • Binary • Portable, i.e. machine independent • Scalable, I.e. multi-dimensional array oriented • Appendable • Self describing • Wide use in atmospheric science community • Data model: • Dimensions • Variables • Attributes

  18. EUFAR NetCDF Metadata Conventions EUFAR conventions for NetCDF extend existing, common standards: NetCDF Climate and Forecast (CF) Conventions Broad conventions, applicable across atmospheric and earth science Interagency Working Group for Airborne Data and Telemetry Systems (IWGADTS) Conventions US-based group Tailored for aircraft

  19. EUFAR NetCDF Metadata Conventions • Specifies file structure • Names for time dimensions • Time for 1Hz signal • Time25 for 25Hz signal • etc • Order of dimensions • Definition of time coordinate – follow CF conventions • Defines standard file attributes • Conventions • Title • Institution • Source • History • References • Comment

  20. EUFAR NetCDF Metadata Conventions • Defines standard variable attributes • units • Long name • Standard name • Valid range • Sampled rate • Category • Calibration coefficients • Dependencies • Processor • Provides standard vocabularies based on CF Conventions • standard names • units

  21. EUFAR NetCDF Metadata Conventions • EUFAR NetCDF Metadata Conventions do not: • Define or standardize variable names Why? • Unique and varied instruments and measurements between aircraft and institutions • Can be multiple measurements of the same quantity Example: NCAR C130 Pressure measurements

  22. NetCDF metadata rules for EUFAR • Example: NCAR C130 Pressure measurements • PSFD • PSFDC • PSFRD • PSFRDC • PCAB • Solution: Data Discovery • Define global attribute ‘air_pressure’ • The value of this attribute indicates recommended field • Example: reference:air_pressure = PSFDC • Data discovery helps users and software choose correct measurement } All pressure: Which one to use?

  23. NetCDF metadata rules for EUFAR • EUFAR standard data discovery fields • Highly recommended • Latitude • Longitude • Altitude • Recommended • Wind speed • Upward air velocity • Air temperature • Platform speed • Dew point temperature • Air pressure • Platform roll angle • Platform pitch angle • Platform orientation • Ground speed E/W • Ground speed N/S

  24. Summary • EUFAR conventions: • Proposal has been circulated • Version 1.0 being released in Oct 2010 • External - INSPIRE • General description • Archival • Search • Embedded - NetCDF • Data usage • Processing • Accuracy

More Related