1 / 5

GTS MetaData Generation

Enhanced system automates harvesting, indexing, and processing of weather bulletins for accurate data retrieval and bulletin associations. Supports metadata synchronization, data replication, and portal updates.

lwilliam
Download Presentation

GTS MetaData Generation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GTS MetaData Generation Metadata Synchronization Data Replication Portal Node Catalogue white-list white-list Data Repository GTS MD GTS MD index index GTS cache MD generation Module MD generation Module Ingestion Module Retrieve Module MSS Feeding Module Local Data bases Metadata GTS Switch data data GTS data bases Central Support Office Central Support Office www.wmo.int www.wmo.int Information Classes Information Classes Volume C1 Volume C1 VGISC SIMDAT Technology 3 –Update the Catalogue & Portal : - Generated metadata files are harvested by the Node. - Relevant information is extracted from the metadata to build the portal descriptions of the products. - Relevant metadata blocks are indexed for free-text search via the portal. • 2 –Process Bulletins : • - Enrich the local copy of Volume C1 with decoded information. • - Apply a set of rules to filter bulletins. • Extract relevant information from bulletins definitions and information classes, build the metadata fragments & collect them into metadata files. • - Build the index of locally published metadata for the future association of metadata files with bulletin instances. Also store information necessary for the future processing of bulletin instances. 1 –Download remote inputs : - WMO FTP server : GTS bulletins Catalogue: WMO Volume C1, - central server (for homogeneity) : Information Classes (information needed to populate the Metadata tags)  WMO documents in program-readable formats,  information collected from several sources  association tables built for the purpose. - central server : White-list to deal the GTS Metadata publishing responsibilities M é t é o - F r a n c e D S I / D E V - J e a n - P i e r r e A u b a g n a c - 2 a p r i l 2 0 0 8

  2. GTS Metadata Generation GTS Metadata TimeGroup rule, unicity criterion fileIdentifier fragment (XML) responsibleparties DO (XML) DOdefault (XML) metadataConstraint fragment (XML) contact fragment (XML) retainE (XSLT) TTAAii (XSLT) fm C13table citation fragment (XML) Content Keywords (XML) Topic Category (XML) CodeForm rules abstract fragment (XML) « improved » Volume C1 (XML) topic fragment (XML) Content rules 1stdescriptiveKeywords fragment (general hierarchical keywords: discovery) (XML) www.wmo.int RE to provide content keywords for grid point bulletins 2nddescriptiveKeywords fragment (hierarchical content keywords) (XML) Volume C1 (XML) extent fragments (XML) NoaaStations Volume A (XML) vgisc fragment (XML) XSLT motor Interpretation Rules & Information Classes • Information extraction rules match regular forms for the fields of Volume C1 definitions. They could be proposed as instructions for Volume C1 declarations. • New formats developed for WMO sources could be proposed as alternate formats for those sources. • Information collected from sparse sources or constructed from scratch show a need that our classes could fill as first drafts. WMO Core Profile version 1.0 of the ISO19115 standard M é t é o - F r a n c e D S I / D E V - J e a n - P i e r r e A u b a g n a c - 2 a p r i l 2 0 0 8

  3. GTS Ingestion Metadata Synchronization Data Replication Portal Node Catalogue white-list Data Repository GTS MD index index (Replication) GTS cache GTS cache MD generation Module Ingestion Module Ingestion Module Retrieve Module MSS Feeding Module Local Data bases Metadata GTS Switch data data GTS data bases GTS Switch Central Support Office www.wmo.int Information Classes Volume C1 VGISC SIMDAT Technology 3 –« Forward » bulletins : - The Node regularly prompts the data repository for the list of newly available products, as they may be concerned by a subscription request. - Newly inserted bulletins concerned by replication should be proposed to the replication mechanism. 2 –Process (Instances of) Bulletins : - GTS bulletins with abbreviated headers matching an entry in the metadata index are inserted into the local database. - Merging the metadata index built during the local generation with (all or part of) the index built during generation at another site enables the local ingestion of remotely published bulletins. 1 –Listen to the GTS flow : - GTS bulletins collected into files according to WMO FTP Protocol appear in one or several repositories (channels of the Message Switching Service). - GTS bulletins are extracted & filtered according to their Abbreviated Header. M é t é o - F r a n c e D S I / D E V - J e a n - P i e r r e A u b a g n a c - 2 a p r i l 2 0 0 8

  4. Retrieve GTS product Metadata Synchronization Data Replication Portal Portal Node Catalogue white-list Data Repository GTS MD index GTS cache GTS cache MD generation Module Ingestion Module Retrieve Module Retrieve Module MSS Feeding Module Local Data bases Metadata GTS Switch data data GTS data bases Central Support Office www.wmo.int Information Classes Volume C1 VGISC SIMDAT Technology • 3 –Retrieve bulletins : • - The request is forwarded by the Node to the Retrieve module and translated into database extraction criteria. • Extracted products are returned to the Node. 2 –Define the request : - Instances of a given GTS bulletin can be requested by selecting an hourly time-slot & a date slot. The metadata defines the choices offered for each parameter: the proposed hours reflect the regular transmission times of bulletin instances, the date slot is bounded by the local database storage depth policy. - GTS metadata offer both retrieve options: one-time only pull & subscription. 1 –Discover the product : - GTS bulletins can be discovered on the portal under a variety of keyword hierarchies (sorted by type and / or location, by content keywords) as defined in the metadata. - GTS bulletins can also be discovered by free-text search (content of indexed metadata tags). M é t é o - F r a n c e D S I / D E V - J e a n - P i e r r e A u b a g n a c - 2 a p r i l 2 0 0 8

  5. MSS Feeding Metadata Synchronization Data Replication Portal Node Catalogue white-list Data Repository GTS MD index (Replication) GTS cache GTS cache MD generation Module Ingestion Module Retrieve Module MSS Feeding Module MSS Feeding Module Local Data bases Metadata GTS Switch data data GTS data bases GTS Switch Central Support Office www.wmo.int Information Classes Volume C1 VGISC SIMDAT Technology • 2 –Forward Bulletins : • - Declare locally published but remotely ingested bulletins to the Node. • Forward the newly inserted bulletins to the local MSS: sort the bulletins by WMO Code Form, collect them into files (WMO FTP protocol) and deposit them in the local MSS repository. 1 –Filter products forwarded by the replication : - Refuse products already present in the local database (match sought on the bulletin identifiers independently of the ingesting data repository). - Accepted products are inserted into the local database and retain the detail of their ingesting data. M é t é o - F r a n c e D S I / D E V - J e a n - P i e r r e A u b a g n a c - 2 a p r i l 2 0 0 8

More Related