1 / 42

Ecmwf observations data handling

Ecmwf observations data handling. Milan Dragosavac. IFS. Analysis. Forecast. ODB. Data transport. Data transport. Data acquisition. MDB. RDB. FDB. PDB. MARS. General applications. Display. Print Plot. Data acquisition. Acquire data Process data Archive data ( not MARS)

kapila
Download Presentation

Ecmwf observations data handling

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ecmwf observations data handling Milan Dragosavac WMO BUFR training course Langen 17-20 April 2007

  2. IFS Analysis Forecast ODB Data transport Data transport Data acquisition MDB RDB FDB PDB MARS General applications Display Print Plot WMO BUFR training course Langen 17-20 April 2007

  3. Data acquisition • Acquire data • Process data • Archive data ( not MARS) • Outgoing ( send data outside Ecmwf) WMO BUFR training course Langen 17-20 April 2007

  4. Acquisition process • GTS data • ftp • Other data WMO BUFR training course Langen 17-20 April 2007

  5. Acquisition outgoing • Qscat satellite data in BUFR form • SSMI brightness temperatures • TOVS 1B calibrated and converted into BUFR format WMO BUFR training course Langen 17-20 April 2007

  6. MEB feed • Restores any old data • Join segmented bulletins • Write bulletins in MEB • statistics WMO BUFR training course Langen 17-20 April 2007

  7. Message data base • “STREAM”_YYYYMMDD • AIRC_20070216 • AIRS_20070216 • AMAP_20070216 • GRIB_20070216 • 75 streams in total • One stream per day WMO BUFR training course Langen 17-20 April 2007

  8. Message data base attributes • TIME character( 4,1) • OBSTYPE character( 4,1) • TTAAii character( 6,1) • BBB character( 3,1) • CCCC character( 4,1) • _SOURCE character(80,1) • _CHECKSUM longinteger • _TIMESTAMP character(20,1) WMO BUFR training course Langen 17-20 April 2007

  9. Message data base attributes continued • _data_size longinteger • _bulk_size longinteger • _owner character( 3,1) • _last_access time(0) • _access_number longinteger • _compress character( 1,1) • _data bulk(20,0,512,1) WMO BUFR training course Langen 17-20 April 2007

  10. Message data base indices • UNIQUE I_timestamp_SYNO_20070215 on (_TIMESTAMP) • UNIQUE I_comp_SYNO_20070215 on (TIME, TTAAii, BBB, _CHECKSUM) • NORMAL (2,62) I_OBSTYPE_SYNO_20070215 on (OBSTYPE) WMO BUFR training course Langen 17-20 April 2007

  11. Message data base • 100 Gbytes allocated + 150 Gbytes for METOP data • Over 22 Gbytes / day • Saved on ECFS according to OBSTYPE • All data can be reprocessed if Data base is lost • HP cluster 4 CPU 64 bit processors • 4 operational machines + 1 for development • Raid disks 1.2 Tbytes WMO BUFR training course Langen 17-20 April 2007

  12. MEB data arrival • Monitoring arrival times of different data types • Analysis cycle • Average number of days WMO BUFR training course Langen 17-20 April 2007

  13. Message data base data availability • Data availability according to synoptic hour WMO BUFR training course Langen 17-20 April 2007

  14. MEB x-interface • Allows MEB search • Find corresponding BUFR observation in REB • Expand BUFR and GRIB data • Write data into file WMO BUFR training course Langen 17-20 April 2007

  15. MEB x-interface • REB BUFR observations corresponding to MEB bulletin WMO BUFR training course Langen 17-20 April 2007

  16. Preprocessing suite • MEB – message data base feed task • deqc – decoding, quality control and BUFR creation, writing into REB • ex00 – extraction for analysis • bc00 – extraction for boundary condition suite • wmo – ftp WMO station list WMO BUFR training course Langen 17-20 April 2007

  17. Decoding • Decoding • Quality control • BUFR creation • Global grib fields creation • Writing into REB WMO BUFR training course Langen 17-20 April 2007

  18. REB extraction for analysis • 14 files created • Conventional observations • Satellite data • Sea surface temperatures • Ice analysis WMO BUFR training course Langen 17-20 April 2007

  19. REB – report data base • One table per observation type per day • AIRC_20070215 • ATOV_20070215 • B002_20070215 • CREX_20070215 • TEMP_20070215 • SSBT_20070215 • QSCA_20070215 • statistic WMO BUFR training course Langen 17-20 April 2007

  20. REB attributes • TIME character(6,1) • IDENT character(9,1) • _BUFR_TYPE integer • _BUFR_SUBTYPE integer • _LATITUDE1 float(6) • _LONGITUDE1 float(6) • _COR1 integer • _QC integer WMO BUFR training course Langen 17-20 April 2007

  21. REB attributes continued • _TIMESTAMP character(20,1) • NR_SUBSETS integer • _data_size longinteger • _bulk_size longinteger • _owner character(3,1) • _last_access time(0) • _access_number longinteger • _data bulk(20,0,512,1) WMO BUFR training course Langen 17-20 April 2007

  22. REB indices • UNIQUE I_ts_SYNO_20070215 on (_TIMESTAMP) • UNIQUE I_comp_SYNO_20070215 on (TIME, IDENT, _BUFR_SUBTYPE, _LATITUDE1, _LONGITUDE1 • NORMAL (2,15) I_extract_SYNO_20070215 on (TIME, _BUFR_SUBTYPE, IDENT) WMO BUFR training course Langen 17-20 April 2007

  23. REB reports data base • Just bellow 22 Gbytes / day • 100 Gbytes allocated + 150 Gbytes for METOP data • All data can be re-decoded at any time • Data can be accessed through MARS request and Metview • Archived in MARS WMO BUFR training course Langen 17-20 April 2007

  24. REB x-interface • Search REB • Find original bulletin in MEB • Expand BUFR and GRIB data • Write data to file • Access WMO station list data base WMO BUFR training course Langen 17-20 April 2007

  25. BUFR x-interface • Expands BUFR message • Any subset can be viewed • Filtering features WMO BUFR training course Langen 17-20 April 2007

  26. WMO Data Base • WMO_a_master • WMO_a_master_manual • WMO_a_master_tape WMO BUFR training course Langen 17-20 April 2007

  27. WMO_a_master attributes • Region_country integer not null • Block smallint not null • Station_index integer not null • Station_subindex smallint not null • Principal _station character(1,1) • Station_name text(20,20,30,1) • Latitude real • Longitude real WMO BUFR training course Langen 17-20 April 2007

  28. WMO_a_master attributes continued • Pressure_elevation integer • Pressure_elevation_flag character(1,1) • Ground_elevation integer • Ground_elevation_flag character(1,1) • Pressure_level smallint • Surf_prog_00 character(2,1) • Surf_prog_03 character(2,1) • Surf_prog_06 character(2,1) • Surf_prog_09 character(2,1) • Surf_prog_12 character(2,1) WMO BUFR training course Langen 17-20 April 2007

  29. WMO_a_master attributes continued • Surf_prog_15 character(2,1) • Surf_prog_18 character(2,1) • Surf_prog_21 character(2,1) • Ua_type_00 smallint • Ua_type_06 smallint • Ua_type_12 smallint • Ua_type_18 smallint • Modify_flag character(1,1) not null WMO BUFR training course Langen 17-20 April 2007

  30. WMO_a_master indices • UNIQUE I_wmo_a_master ON (block,station_index, station_subindex) WMO BUFR training course Langen 17-20 April 2007

  31. WMO station list WMO BUFR training course Langen 17-20 April 2007

  32. ODB Observational data base • Interface to analysis • Data base contains unpacked bufr data and original bufr data • Loading done in parallel with load balance achieved on supercomputer due to bufr features • Analysis updates ODB • Analysis bufr feedback files created • Archiving of feedback files in MARS WMO BUFR training course Langen 17-20 April 2007

  33. Loading ODB with observations • Tasks running in parallel • 2-16 processors used per task • Load balancing achieved WMO BUFR training course Langen 17-20 April 2007

  34. ODB to BUFR • odb2bufr family WMO BUFR training course Langen 17-20 April 2007

  35. Odbviewer WMO BUFR training course Langen 17-20 April 2007

  36. Odbviewer WMO BUFR training course Langen 17-20 April 2007

  37. Odbviewer WMO BUFR training course Langen 17-20 April 2007

  38. Odbviewer WMO BUFR training course Langen 17-20 April 2007

  39. Odbviewer WMO BUFR training course Langen 17-20 April 2007

  40. Odbviewer WMO BUFR training course Langen 17-20 April 2007

  41. Odbviewer WMO BUFR training course Langen 17-20 April 2007

  42. Future • Sharp increase in volume of observations expected • IASI data with 8641 channels per observation • Data bases have crucial role in observations handling • Binary data representation allows easy handling of observations as bulk objects WMO BUFR training course Langen 17-20 April 2007

More Related