1 / 26

T.C. TURKISH STATE METEOROLOGİCAL SERVICE DEPARTMENT OF RESEARCH AND INFORMATION TECHNOLOGIES

T.C. TURKISH STATE METEOROLOGİCAL SERVICE DEPARTMENT OF RESEARCH AND INFORMATION TECHNOLOGIES. TSMS ARCHIVING SYSTEM . Mustafa Sert October 2011. Data Flow Hardware Software Services. OUTLINE. Data Flow. Telecomm. Section. NWP Section. Research Section. Remote Sensing Section.

lonna
Download Presentation

T.C. TURKISH STATE METEOROLOGİCAL SERVICE DEPARTMENT OF RESEARCH AND INFORMATION TECHNOLOGIES

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. T.C.TURKISH STATE METEOROLOGİCAL SERVICEDEPARTMENT OF RESEARCH AND INFORMATION TECHNOLOGIES TSMS ARCHIVING SYSTEM Mustafa Sert October 2011

  2. Data Flow Hardware Software Services OUTLINE

  3. Data Flow Telecomm. Section NWP Section Research Section Remote Sensing Section Observation Messages (SYNOP,METAR..) MSS Server GTS Data Model Data Model Data Radar / Satallite Data MSSQL DB 206 awos stations SYBASE ASE DB Preproc/ MARS/ HSM Servers TUMAS WEB Portal ORACLE DB 150 awos stations SYBASE IQ DW 199 awos Stations (Building) ORACLE DB (building) IT Section

  4. Data Inventory • Climatological data (1926-Today) • Ravinsonde data (1971-Today). • Synoptic data(1980-Today) • AWOS data (2003 -Today) • Aerodrome AWOS data (2003-Today) • Metar data (2000-Today) • Taf,Sigmet,Airmet data (2008-Today) • Open Screen Observation Data (1999-Today) • Sea Surface temperature data(Beginning– Today). • Inversion analyse data (2006-Today) • Ozon/UV Data (2006/1997-Today) • Radar Data (2008 – Today) • Satallite Data (2008 – Today) • Model Data (2006 – Today)

  5. Data Inventory

  6. Data Inventory

  7. Data Inventory

  8. Hardware • 1 DB Server (HP DL380 G4, Intel Xeon 2 CPU 3.4 Ghz, 6 GB RAM, 2 * 146 GB Internal, 1.8 TB SATA- 1.3 TB SCSI External Disk) • 1 DW Server (HP ML370 G4 2*3.0 Ghz, 10 GB RAM, 2 * 146GB Internal, 1,2 TB SATA External Disk) • 1 DPP Server (IBM x3650 2 * Dual core Intel Xeon 3.0 Ghz, 24 GB RAM, 2*146 GB Internal, 3 TB FC External Disk) • 1 MARS Server (IBM x3755 3 * Dual Core AMD 2.80 Ghz, 24 GB RAM, 2*146 GB Internal, 2.5 TB FC, 10 TB SATA External Disk) • 1 WebPortal Server (IBM x3650 2 * Dual core Intel Xeon 3.0Ghz Processor, 24 GB RAM, 2*146 GB Internal, 0.5 TB FC External Disk) • 1 HSM (TSM) Server (IBM p series P560 16 * CPU 1.5 Ghz Power 5 Processor, 32 GB RAM, 2*146 GB Internal, 1 TB FC External Disk)

  9. Hardware • 2 IBM N5500 Disk System (2 * 11 TB FC, 2*11 TB SATA Disk, 2 SAN Switch) • 1 IBM TS3500 Enterprise Tape Library ( 12 x IBM 3592 700 GB Teyp Drive, Total 290 TB Tape Media (415 Data Cartridge x 700 GB)

  10. Hardware TÜMAS System Web Portal Server Sybase ASE Server Sybase IQ Server TUMAS Web Portal presents the data from databases and Archive System DPP Server MARS Server Source Data Archive/retrieve data on/from MARS or retrieve metadata Disk System 11 TB FC 11 Terabyte SATA Data Acquire from different sources and pre-process, (format conversion) Hierarchical Storage Mgmt. Server (TSM) Manages the data on disk and tape systems. Teyp Library 290 Terabyte Storage Area Network

  11. Software • Operating System ; Generally Suse Linux 9 / 10, except TSM Servers (AIX 5.4) • Database ; Sybase ASE, Sybase IQ, Empress DB and mysql (in prprocessing server) • Hierarchical Storage Management Software; IBM Tivoli Storage Manager 5.5 • Web Application Software; IBM WebSphere Portal 6.1 • Spesific Software ; MARS ( The Meteorological Meteorological Archival and Retrieval System, from ECMWF) • Spesific Software ; EmosLib, Preprocessing software, Metview/magics, GribAPI from ECMWF

  12. Software MARS • Meteorological Archival and Retrieval System • Manages meteorological fields • Operational data • Research data • Special projects • Not a file system • Users are not aware of the location of the data • An archive, not a database • Metadata online • Data offline • Model Data ; GRIB • Observations; BUFR

  13. Software MARS Components Client/Server architecture Clients: workstations, supercomputers Servers: supercomputers, dedicated servers Several databases Tape library

  14. Software MARS at ECMWF • 12.3 PetaBytes of data in ~5 million files, for more than 55 · 109meteorological fields • ~ 200 GigaBytes of metadata • 50 · 106 fields added daily (14 Terabytes) • 500 active users/day executing ~700000 requests/day • ~ 19 Terabytes retrieved daily • Analysis, forecast and observations since 1957 (ERA-40) • Operational forecast since 1985

  15. Software • MARS A meteorological language Retrieve,date = 20010101/to/20010131,parameter = temperature/geopotential,type = forecast,step = 12/to/240/by/12,levels = 1000/850/500/200,grid = 2/2,area = -10/20/10/0

  16. Software MARSWhat is a field? • Class (Operations) • Type (Forecast) • Stream (Daily Archive) • Version (1) • Date (1978-01-12) • Base Time (12 UTC) • Step (240 hours) • Parameter (Temperature) • Level (1000 hPa)

  17. Software Request References References Files References Data MARS Server Server Data Data Data Files File Store MARS Architecture Meta-Data Meta-Data Request

  18. Services TUMAS (Turkey Meteorological Archiving System) • Web – based application developped by Java. • Users can access data online from TUMAS • Internal / External users should be a TUMAS member for retrieve data. • External users from non-governmental organizations should pay for data. • Data size for database ~ 1 TB • Data size for TSM (Disk - Tape Library) ~ 70 TB

  19. Services The data presented in www.tumas.dmi.gov.tr can be grouped into 5 titles Turkish Meteorological Data Archiving System TUMAS

  20. Services Turkish Meteorological Data Archiving System TUMAS

  21. Services Turkish Meteorological Data Archiving System TUMAS TUMAS has been designed to meet the increasing demand of storing and presenting of meteorological data (observations, radar, satellite, NWP, etc.) TUMAS is the main repository of meteorological data at Turkish State Meteorology Service(TSMS). It contains terabytes of data which are generated by TSMS. TUMAS use Sybase RDBMS and MARS software which is used in ECMWF for achieve and retrieve data which are stored in different media ( disk storage system,tape library ). MARS software is installed in TUMAS and mainly works archieve/retrieve data from/to disks or tapes. http://www.tumas.dmi.gov.tr

  22. Services Turkish Meteorological Data Archiving System TUMAS • In TUMAS – mars section, all data is archieved in bufr and grib format. Because MARS software does not support data except bufr or grib format. • TUMAS archieving data includes: • observation data (mars, sybase db) • raw and product data obtained from satellite • raw and product data obtained from meteorological radars • input / output data obtained from local and global weather forecast models.

  23. Services Turkish Meteorological Data Archiving System TUMAS • TUMAS system consists of 4 modules. These modules are Preprocessing, Mars main system, HSM and WebPortal. • 1.Preprocessing module : includes below • collection of all data • decode data • convert data to bufr/grib format • data quality control • write to database to utilized by MARS • 2. MARS module : • MARS is a client/server based application software to archieve data into disk storage system and tape systems. MARS has a interface script language to archieve/retrieve data. MARS receives data from preprocesing and sends these data to archieve storage system by using HSM interface.

  24. Services Turkish Meteorological Data Archiving System TUMAS 3. HSM module : data are archieved in several media (disk and tape systems). To manage data which are stored in this storage system, HSM (hierarchical Storage Management software is utilized. In TUMAS, IBM Tivoli Storage Management software is used. 4. WebPortal : Portal can access to database data for observations. MARS data is presented to users by webmars module. Webmars is data web-based presentation system. All archieved data is presented under several authorization and authentication mechanisms. In TUMAS, SUSE Linux 9 platform is used for mars,webPortal and preprocessing modules. AIX 5 Unix platform is used for HSM module.

  25. Future Plans • Installing a new server for climate models regCM and precis. (8 processor, 10 core /proc, 256 GB RAM, 50 TB SAS disk) (2011 Q4 – 2012 Q1) • Installing a virtualization platform (2012 Q3) • Developping a new web portal and grow mapping, visualisation. (with open source software)

  26. Questions?msert@dmi.gov.tr Turkish State Met. Service Department of Research And Information Technologies Information Technology and Statistics Division October – 2011 Alanya

More Related