1 / 21

In-situ data workflow using OGC Sensor Web Enablement (SWE)

In-situ data workflow using OGC Sensor Web Enablement (SWE). GEO Data Technology Workshop Bartolomeo Ventura (1) , Andrea Vianello (1) , Mattia Rossi (1) , Daniel Frisinghelli (1) , Roberto Monsorno (1) , Armin Costa (1) , Roberto Roncella (2) , and Alexander Jacob (1)

hollye
Download Presentation

In-situ data workflow using OGC Sensor Web Enablement (SWE)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. In-situ data workflow using OGC Sensor Web Enablement (SWE) GEO Data Technology Workshop • Bartolomeo Ventura(1), Andrea Vianello(1), MattiaRossi(1), Daniel Frisinghelli(1), Roberto Monsorno(1), Armin Costa(1), Roberto Roncella(2), and Alexander Jacob(1) • (1) EuracResearch – Institute for Earth Observation • (2) National Research Council of Italy - Institute of Atmospheric Pollution (CNR-IIA)

  2. Summary • From heterogeneous data to standardization concept • Data work flow • Use case: LT(S)ER Matsch – Mazia initiative

  3. Heterogeneous Data how to handle multiple data? Especially if they belong to different sensors at different scale with different characteristics Need of a standardization In recent years the market of the sensors has grown quickly. It is possible to find more and more efficient and intelligent devices together with a notable decrease in their costs. BUT Data sharing and Interoperability

  4. Real problem Absence of a standardization for input data • Different data organization • with respect to the same sensor Different sensors and stations • Different names for input files Different data organization

  5. Standardization concept The ability to share the same data resource with multiple applications or users. OGC has defined the Sensor Web Enablement (SWE) initiative that introduces: • standard for encoding sensor observation Observations&Measurements (O&M); • standard for describing sensor characteristics Sensor Model Language (SensorML); • web service to accessSensorML and O&M: Sensor Observation Service (SOS) The 52°North SOS is a reference implementation of the OGC Sensor Observation Service specification (version 2.0), an interoperable interface for publishing and querying sensor data and metadata.  The web application of the 52°North provide an API to insert data in the database (PostgreSQL Database) using post method and operations defined in the SOS standard.

  6. Data lifecycle • It provides a high-level overview about the management of data subdivided in different stages according to specific tasks with the aim of a successful organization and preservation of data for use and sharing

  7. Data lifecycle - Acquire: collect data from ground stations South-Tyrol network of ground stations for collecting both images (RGB and Infrared) and 75 other physical parameters such as Normalized Difference Vegetation Index (NDVI), Surface Temperature, or Photosynthetically Active Radiation.

  8. Data lifecycle – Harmonize: data check; data validation; data clean It is not unusual for data to have discrepancies in consistency or format. • It is important to create a cohesive dataset that is ready to be organized.

  9. Data lifecycle - Organize: automatic data storage in a SOS implementation of 52°North OGC standard • The 52°North SOS provides a RESTful API for inserting data in the database using • POST method and the operations defined in SOS standard: • InsertSensor: to register sensors in the DB following SensorML standard. • InsertObservation: to upload measurements into the DB following O&M standard. Considering the big number of records to be collected, it is necessary to have a solution able to speed up not only the InsertSensor but also the InsertObservation operations

  10. Data lifecycle - Organize: automatic data storage in a SOS implementation of 52°North OGC standard The innovation of the methodology consists in facilitating the insertion of the observations in the database with a semi-automatic procedure. • Only a minimal interactionof the Data provider/User is required: • Standardization of the input file • to have an header describing the data column; • an established date and time format i.e. YYYY-MM-DDThh:mm; • the time-stamp is always stored in the first column; • a predetermined naming convention i.eXXX_FOI_XXX.csv • Creation of the configuration file • XML file containing all information about the considered FOI. • A raw metadata version of the FOI.

  11. Data lifecycle - Organize: automatic data storage in a SOS implementation of 52°North OGC standard Two versions of the Insert_Observation operation have been implemented: serial approach: the process reads every folder and .csv file of the sensor consequently one after the other; parallel approach: the process analyses multiple folders simultaneously and inserting multiple observationsat the same time assigning to each of the available computer's cores the first available station category folder. The scripts (GNU GPL v3.0) are available in the GIT repo: https://gitlab.inf.unibz.it/SOS/SOS-data_upload/tree/master

  12. Data lifecycle - Analyze: R, Python clients to access time series The RESTful API, provided by the SOS service, allows access to a complete dataset The R package MonalisR, communicates with the SOS API to download data in JSON format, giving a complete overview of the database content. User can add its own R scripts to customize the output https://gitlab.inf.unibz.it/earth\_observation\_public/MonalisR

  13. Data lifecycle - Share: results, data, metadata A customizable Graphical User Interface (GUI) is available. • The web application communicates with the SOS service in real time and allows interactive visualizations of the extracted dataset. A web application offers a faster way to search and visualize data by creating ad-hoc queries using GUI, without writing codes

  14. Data lifecycle - Share: results, data, metadata The availability of metadata in the sensor web can be very useful for discovering the right sensor, together with its spatial and temporal information. The configuration file allows to have both the XML file for the InsertSensor operation in the SOS service, and an XML file for a CSW catalogue, compliant with the OGC standards as well as INSPIRE and ISO-19139 using an XSL transformation: The main advantage of using an XSLT consists in having the possibility of adapting all the information dealing with the sensor in the corresponding standard to be used or directly customized to the requirements of a specific project.

  15. Integration with GEOSS Platform IN-SITU DATA • SERVICE TYPE: OGC Sensor Observation Service (SOS) • SERVICE VERSION: 2.0 INTEROPERABILITY TESTS • DISCOVERY: GEO Discovery and Access Broker (DAB) successfully harvested metadata records from SOS service • 14 Observation Offerings (series) • 14 Features of Interest (datasets) • ACCESS: GEO Discovery and Access Broker (DAB) is able to find the following Observation Data • Air temperature • Relative Humidity • Precipitation • Wind Speed • …

  16. Integration with GEOSS Platform

  17. Use Case: LT(S)ER Matsch – Mazia initiative The Long-Term Ecosystem Research (LTER) Europe is an initiative which aims to study how ecosystems react, in the long term, when influenced by different factors (belonging to different fields) at different scales (local, regional, continental, and global). The Institute for Alpine Environment is involved in the LT(S)ER Matsch-Mazia initiative (Sstands for Social) focused over the Val Mazia-Matsch area in South Tyrol • The LTER Stations in Val Mazia-Matsch are individually equipped with diverse sensors, detecting various key environmental parameters: • surface and soil temperatures, • wind fluctuations and speed, • relative humidity, • soil water content, • potential of precipitation measurements. • At present, this network has collected more than 80 million observations from 30 ground stations, corresponding to 500 sensors http://www.lter-europe.net/, http://www.lteritalia.it/macrositi/it25

  18. Use Case: LT(S)ER Matsch – Mazia initiative SOS represents the common platform for sharing the collected data using SWE standards. • For this initiative, 30 configuration files are necessary. Once they are ready, the process to insert the corresponding sensors into the SOS can start and the remaining steps are completely automatic. • The system is able to verify if new input files arrive in the dedicated data folder. If new data are available, the corresponding Insert Observation file is created and automatically inserted in the database. • Considering the frequency of ground measurements, we set this automatic and customizable routine to start every 15 minutes, allowing a near real-time database update, as well as analysis-ready data for researchers. LT(S)ER public web application

  19. Conclusions • Key message • The adoption of the 52°North SOS implementation and the integration of a specific tool can automatize observations uploading and sensors registration with the least effort possible by using SWE standards • An OGC standard-compliant methodology, allowing the reuse of data outside the research context and, thus, avoiding duplication. • A flexible methodology, which can be applied to upload data into any SOS implementation. • Open source and custom software: python scripts can be improved, with respect to a more dynamic way to handle the parallelization processes. • Collected data are available in near real-time and accessible through a web application, offering a faster way to search through and visualize data by creating ad-hoc queries using GUI without writing codes. • Source code available in a dedicated GIT repository • What GEOSS can do? • To push more and more towards the use of standards

  20. Outlook Optimization of the creation of the configuration file by developing a GUI. Development of a driver to connect the OpenEO API to the SOS API, in order to access different services from different clients (such as R, Python, and Javascript) for big Earth observation cloud back-ends, in a simple and unified way. Use of SpatioTemporal Asset Catalog(STAC) specification as alternative solution for handling metadata.

  21. Thanks for the attention! Contact • bartolomeo.ventura@eurac.edu Eurac Research – Institute for Earth Observation Bartolomeo Ventura , Andrea Vianello et al., A Methodology for Heterogeneous Sensor Data Organization and Near Real-Time Data Sharing by Adopting OGC SWE Standards, ISPRS Int. J. Geo-Inf.2019, 8(4), 167; doi:10.3390/ijgi8040167

More Related