1 / 7

Different types of ingestion

Different types of ingestion. Ingestion for replication Essential data Oriented replication and subscription Added services Integration of the legacy databases Parsing of the GTS flow Global products Geo-localisation products. Added Services 1: Access to GTS legacy database. Advantages

stephaniefisher
Download Presentation

Different types of ingestion

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Different types of ingestion • Ingestion for replication • Essential data • Oriented replication and subscription • Added services • Integration of the legacy databases • Parsing of the GTS flow • Global products • Geo-localisation products GTS Ingestion Simdat Meeting 05-02-08

  2. Added Services 1: Access to GTS legacy database • Advantages • Not mandatory to parse and crack the GTS files • Use of the legacy databases behind the switch (outside vgisc area) • Database of flat files in folders mapped on the short header TTAAiiGGggg • Database of flat files in folders mapped on the hour or the minute • Disadvantages • It’s impossible to have a dynamic metadata state (no notification) • Catalogue Synchronization and replication impossible • static representation of GTS products, no standard GTS database • Existing sofware in Meteo France • Database of flat files in folders mapped on the short header TTAAiiGGggg • FTP access (Jakarta Java Classes) GTS Ingestion Simdat Meeting 05-02-08

  3. Other DB GTS Database NWP DB GTS Database GTS Collections Metadata GTS Database NWP Metadata GTS Switch DR GISC CN Remote Switch Remote CN GTS Ingestion Simdat Meeting 05-02-08

  4. Added Services 2: GTS database in the VGISC area • Advantages • The Data Repository owns its GTS Data Bases (Data , Metadata) • Dynamic metadata state (what is present in the database) • Individual messages database • The real-time push harvesting of metadata is possible • The GISC parser waits and stores. It works on solicitation. • Disadvantages • Parsing and cracking the GTS files • Management of the Metadata -> what structure ? • Hard work with strong knowledge on GTS messages GTS Ingestion Simdat Meeting 05-02-08

  5. GTS Ingestion Simdat Meeting 05-02-08

  6. Experience: GTS anomalies • Collections • Stations that do not belong to the country • A few TTAA headers not compliant with the tables (Analyses for example) • Different collections with the same header, BBB integrated (need rules to choose the good one) • Individual messages • The same messages (same station, same type, same time) are different in different collections GTS Ingestion Simdat Meeting 05-02-08

  7. Other DB NWP DB GTS Database GTS Database GTS Database NWP Metadata GTS GISC Parser GTS Switch DR GISC GTS Collections Metadata GTS Messages Metadata CN Remote Switch Remote CN GTS Ingestion Simdat Meeting 05-02-08

More Related