WHY TABLE DRIVEN CODE FORMS?

1. WHY TABLE DRIVEN CODE FORMS? Langen, 17 April 2007 (Joël Martellet, WMO, World Weather Watch, Data Processing and Forecasting Systems)

2. Why table driven codes?:Plan • The past: fixed alphanumeric codes • and the present: fast evolution of science and technology • Solution: table driven codes • Objectives of this seminar • What is different?

3. WHY WMO CODES? • A WMO “code” is a way of representing data which are exchanged internationally between countries = a data representation form • The Purpose is global exchange of meteorological (or associated to meteorology or hydrology, e.g. marine, oceanography) data in an unambiguous and most efficient way for use by many different applications in different countries • FUNDAMENTAL: IT IS THE RAW MATTER MAKING METEOROLOGY • The design of the Codes should take into account operational constraints

4. WMO DATA REPRESENTATION The design of a data representation system for real time exchange and processing has to take into account operational constraints • THE PAST: the Constraints decades ago: • Telecommunication lines with low bandwidth (50 Bits/s) led to: • character type • abbreviated coding • parameter representation often translated by code rather than the exact value • Manual operation: • human readability • not too frequent • Solution: • FIXED ALPHANUMERIC CHARACTER CODES

5. CONSIDER THE WORLD WEATHER WATCH DATA FLOW (OLD!): Before the meteorological codes were seen! at all stages of the data flow! Teletype Paper Manual plotting Forecaster Observerand encoding on Teletype

6. The old WMO “codes” were developed by data type, e.g. TEMP, SYNOP, SATOB, etc. • And they have a «fixed» format. FM 42 AMDAR FM 41 CODAR FM 35 TEMP FM 13 SHIP FM 88 SATOB FM 87 SARAD FM 86 SATEM FM 85 SAREP

7. Nowadays: faster technical evolution • Expanding requirements for applications: e.g. NWP and Global Climate Studies • Increasing volume of data (e.g. satellite) and complexity of observations • Higher accuracy of the measurements • Higher resolution required for observations (e.g. Radiosonding): • In time (more frequent) • In space (vertical resolution) • All radiosonde observations required (flight of the sonde, like an aircraft) for very high resolution non-hydrostatic models.) • New types of data observed and exchanged (e.g. ozone, radiology, sea and water level , etc…) • More frequent requested changes for new data types

8. And how to modify a traditional “fixed” code like inserting a new group: • Such a modification would require a corresponding update to all software programs that encode or decode such reports, if not: the software would either give incorrect values or fail completely. • The reason is that the fixed format and the coding conventions which define the data have to be translated and built fully into the coding and decoding programs. It is this fact that renders the traditional alphanumeric code forms incapable of accommodating new types of data.

9. Difficulties, if not impossibilities: • Changing requirements but fixed codes • Change implies long time scale and costs, because automation came!: • software fixes needed at all stage of the processing chain • equipment made by manufacturers • training required, etc. • Approval of code changes by WMO official body: CBS, and then implementation by Member Countries take from 2 to 4 years, or even more • This is incompatible with the modern fast rate of evolution of science and technology

10. Then blocking ! • Current (but old) Traditional Alphanumeric Codes (TAC) prevent the exchange of critical environmental information to NMHSs and their customers • There is an unnecessarily inefficient costly use of resources for information exchange • They will increasingly constrain NMHS capabilities to exchange more accurate and timely information • They will become increasingly more costly to sustain expansion and growing scientific needs.

11. WHY TABLE DRIVEN CODES?Operational meteorology evolves: • More automation: • more computer processing • Higher bandwidth (9.6kb/s, 64kb/s, 1 Mb/s) • More frequently: new parameters (e.g. satellite, oceanography) • Higher resolution required in time and space • Higher accuracy- more precision required

12. Today the meteorological code does not need to be seen at all!CONSIDER THE WORLD WEATHER WATCH DATA FLOW (ALL AUTOMATED): Automatic decoding, displaying program Forecaster Automatic Weather Station (AWS) automatic encoding

13. Different functions within the data flow: • Not to confuse the functions of: • Data acquisition • Data collection • Data transmission • Data reception • Data visualization • The format used to represent data could be different at each stage, if it is more efficient.

14. The format used to represent data could be different at each stage, if it is more efficient. Not to confuse the functions of: • Data acquisition: AWS sensors value format (or observer reporting data) • Data collection: Encoding in a data representation form the observer report (or the AWS sensors values) for national collection • Data transmission: Keeping format or Encoding with a view to perform international transmission • International transmission process: If possible do not change the data representation, just move a mail “envelope” • Data reception: Decode the data representation form to feed data base or other processing applications • Data visualization: Visualize the data values for a human reader = convert to the most appropriate display format for the user

15. The WMO concern is the International Exchange and the agreed formats to transmit data between countries. • National formats could be different than WMO recommended standards if more appropriate. • But we need new efficient formats for international exchanges.

16. SOLUTION:Need for Data Representations which offer: • EXPANDABILITY (ability to add easily new parameters, to change accuracy) • SELF DESCRIPTION (auto description of content) • FLEXIBILITY (ability to vary the content) • SUSTAINABILITY (old archives readable) • COMPRESSION (for binary digital exchange)

17. SOLUTION:Need for Data Representations which offer: • EXPANDABILITY (ability to add easily new parameters, to change accuracy): • BUFR, CREX, GRIB Edition 2 (GRIB2) • SELF DESCRIPTION (auto description of content) • BUFR, CREX, GRIB Edition 1 (GRIB1), GRIB2 • FLEXIBILITY (ability to vary the content) • BUFR, CREX, GRIB2 • SUSTAINABILITY (old archives readable) • BUFR, CREX, GRIB1, GRIB2 • COMPRESSION (for binary digital exchange) • BUFR, GRIB1, GRIB2

18. BUFR:Binary Universal Form for the Representation of meteorological data designed in early eighties- approved for operation implementation on November 1988- Used for archives of all data types and operational exchange of satellites data, ASDAR, AMDAR ,wind profilers, tropical cyclone data, ARGOS data: buoy, XBT, XCTD, sub-surface floats, and starts to be used for translating Traditional Alphanumeric Codes (TAC: SYNOP, SHIP, TEMP, CLIMAT, etc..). CREX: Character form for the Representation and Exchange of data designed in the nineties (it is the image of BUFR in character) - approved for operational implementation on May 2000- designed for data types which do not have Traditional Alphanumeric Codes (TAC) and where BUFR cannot be used - Used for operational exchange of ozone, radiological, tide gauge, hydrological and soil temperature data , squall lines (Africa) and tropical cyclone information (also it is a good tool to understand BUFR - could be an interim solution before BUFR, if binary connection and binary processing not possible)

19. Objectives of the seminar • To understand fully the structure of BUFR • To understand how to convert Traditional Alphanumeric Codes (TAC) data into BUFR • To understand how to initiate and have approved additions to the Table Driven Code Forms • To understand how to use and implement in the data processing chains some of the existing decoder and encoder software • To understand how to organize the migration internationally and therefore nationally (migration plan) • To understand what to do to start operationally the migration within the World Weather Watch System

20. TABLE-DRIVEN CODES FORMAT • In a table driven code form, like in a Traditional Alphanumeric Codes, there is also a fixed structure, but it applies only to the shape of the «container» (or code layout or structure) rather than to the content of the «container». These structure rules are the same for whatever the type of data: “only one physical format for all data types”. The presence and form of the data are described within the «container» itself. This is the concept of: • SELF-DESCRIPTION. • In order to accomplish it, there is a section: THE DATA DESCRIPTION SECTION in a BUFR, CREX or GRIB message, where the type and form of the data, contained within the message, are defined. • Once this section is read, the following content: the transmitted data (in the DATA SECTION) can be understood (and decoded).

21. THE DATA DESCRIPTION SECTION • This section will in fact contain a set of “pointers” (called “descriptors”) which refer to elements which are listed in predefined and internationally agreed TABLES (kept in the official WMO Manual on Codes and on the WMO web server). In the TABLES, the list of descriptors (elements) defines how every “datum” is and SHALL BE coded. • Hence: the name: • “TABLE DRIVEN CODE FORMS”. • All the “datum” to be transmitted must be defined in the tables of the WMO Manual (e.g. name, unit (m/s, °K,…), size: 6 bits, 9 bits,…).

22. Descriptors in theData Description Section In the MESSAGE: The Data Description Section: I see the List of Descriptors In the WMO MANUAL: Set of TABLES containing set of Descriptors Each Descriptor defines how a parameter (an element) Shall be coded: if I know that,I can decode the data in the Data Section (indeed, the data must be listed in the same order as the descriptors in the Description Section) and the Tables must be readable by the decoder (program or human)

23. General structure of a TDCF message Table driven codes generally have this structure  • Indicator: GRIB/BUFR/CREX • Identification: Date, time, originator, edition number, table version number ... • Optional section: e.g. more Metadata, private data … • Data description section: What sort of data follows (pointers to WMO tables identifying the data to be transmitted) • Data section:Actual data here • Closure: “7777”

24. When there is a requirement for transmission of new parameters or new data types, items are simply added to the WMO defined tables (to be agreed by CBS) and new pointers (descriptors) will be included in the data description section to reference the new items, if they must be exchanged. In principle, table driven codes can transmit an “infinity of information”. There is total FLEXIBILITY. • Definition of new «codes » as such, is no longer necessary. Expansion of WMO tables is sufficient = EXPANDABILITY. • Edition number of the format (physical structure of the message) and version number of the WMO tables recorded in the identification section at a fixed for ever place enable the safe processing of archived data = SUSTAINABILITY.

25. BUFR versus CREXIN THE DATA SECTION: • IN BUFR, AN ITEM (PARAMETER CORRESPONDING TO A TRANSMITTED DATUM IN A REPORT) WILL BE TRANSLATED INTO A SET OF BITS. • IN CREX AN ITEM WILL BE TRANSLATED INTO A SET OF CHARACTERS (BYTES). CREX IS THE IMAGE IN CHARACTERS OF BUFR BIT FIELDS. THUS, IT IS ALSO VERY SIMPLE TO READ A CREX MESSAGE. • IN BUFR AND CREX THE PARAMETERS ARE SIMPLY LISTED AS THE USER REQUIRES. THE DATUM ARE LAYED OUT ONE AFTER THE OTHER.

26. BUFR versus CREXCompression(1) • BUFR has an advantage over CREX: it offers condensation (packing) or compression, therefore voluminous data (ex. satellites, ACARS, wind profilers) will require less resources for transmission and stocking. • But the big disadvantage is that man cannot read it!

27. BUFR versus CREXCompression(2) • BUFR provides efficient data packing via: • “BUFR compression” part of BUFR (and GRIB) specifications, like run length compression. Compression is performed by an algorithm which is defined within the code regulations of BUFR (and GRIB). • And somehow data condensation by: • Use of combined data descriptors, called “common sequences” • Use of scale and reference values for descriptors of datum. • CREX is not designed for packing efficiency, obviously not fit for bulky data

28. BUFR versus CREX • CREX, provides flexibility and human readability. • It requires for the transmission of big or many reports a substantial amount of characters. (However, if considered as a bit stream, and by applying compression algorithms available today, it can be compressed somehow, like any computer file) . • CREX tables will have the same parameters as BUFR tables, and similar rules, but CREX will be simpler. • There is no packing, no associated data (flags, substituted values). • CREX is the image of BUFR and provides a standard (but not the best) for visualisation (read and decode ) of BUFR information.

29. VERY SIMPLE CREX EXAMPLE CREX++ T000104 A000 B11001 B11002++ 260 0246+ 030 0137++ 7777 • Data Description Section: • CREX WMO Master Table 00, Edition1, Version 4, Surface data, wind direction, wind speed • Data Section: • wind 1: direction 260 degrees, speed 24.6 m/s • wind 2: direction 30 degrees, speed 13.7 m/s

30. CODING OF SQUALL LINESIN WEST AFRICA IN CREX:Observations (3 points) and forecasted trajectory and evolution: CREX++ T00020412// A007001 P00012000 U00 S001 Y20050823 H1830 D16060++ 2005 08 23 17 50 1500 –01000 070 00010 1900 –00840 1100 –01220 02 0038 0300++ 7777 DATA DESCRIPTION SECTION: T00020412// = 00 = Master table for meteorology 02 = CREX edition number 04 = BUFR edition number 12 = version table of BUFR/CREX // = No local table A007001 = 007 = Synoptic features 001 = Squall Line P00012000 = 00012 = Originating Centre = Dakar 000 = No sub-centre U00= 00 = Sequence number of message, = 0 = first S001= 001 = number of sub-sets in the report = 1 Y20050823= Date of the message = year, month, day H1830= Hour of the message = hour, minute D16060 = Common sequence defining Squall line (type 1)

31. CODING OF SQUALL LINESIN WEST AFRICA IN CREX:Observations (3 points) and forecasted trajectory and evolution: CREX++ T00020412// A007001 P00012000 U00 S001 Y20050823 H1830 D16060++ 2005 08 23 17 50 1500 –01000 070 00010 1900 –00840 1100 –01220 02 0038 0300++ 7777 DATA SECTION (Part 1): Time of observation: D01011 Date: 2005 = Year 08= Month 23= Day D01012 Hour: 17= Hour 50= Minute Position of Squall Line Centre: B05002 Latitude: 1500= 15 deg. North B06002 Longitude: -01000 = 10 deg. West B19005 Direction of moving feature: 070= 070 degrees B19006 Speed of moving feature: 00010= 10 m/s

32. CODING OF SQUALL LINESIN WEST AFRICA IN CREX:Observations (3 points) and forecasted trajectory and evolution: CREX++ T00020412// A007001 P00012000 U00 S001 Y20050823 H1830 D16060++ 2005 08 23 17 50 1500 –01000 070 00010 1900 –00840 1100 –01220 02 0038 0300++ 7777 DATA SECTION (Part 2): Amplitude of feature, from most external point to centre point: North side: B05002 Latitude: 1900= 19 deg. North B06002 Longitude: -00840= 8 deg. 40 West South side: B05002 Latitude: 1100= 11 deg. North B06002 Longitude: -01220= 12 deg. 20 West B20048 Evolution of feature: 02 = Intensification B11041 maximum burst expected: 0038= 38 m/s B13055 intensity of rain expected: 0300= 300 mm/h CODE TABLE 0 20 048 Evolution of feature Code figure 0 Stability 1 Diminution 2 Intensification 3 Unknown 4-14 Reserved 14-99 Not used

33. EXAMPLE FOR SYNOP:A SYNOP message from an RA VI high altitude station - the station VAN (Turkey): SMTU10 LTAA 230600 AAXX 23064 17170 11665 60907 10025 21030 38444 48605 52026 69942 70262 83540 333  21035 3/105 42002 70025 83635 86359= KSMDii LTAA 230600 CREX++ T000104 A000 D07222++ 17 170 1 2005 11 23 06 00 3845 04332 16690 16620 08444 ///// 0026 02 08500 01605 14 090 0035 025 -030 067 1500 002 06 02 075 07 03 0105 35 24 10 0002 01 03 06 0105 02 06 03 0270 12 -052 00002 00025 -0012 00004 00205 -0012 //// -0012 -0350++ 7777 Note: The underlined groups in the data section represent additional information that is not included in the SYNOP report.

34. The Table Driven Codes Forms (TDCF) BUFR/CREX • Offer great advantages compared to the traditional alphanumeric codes (TAC) (1): • Self-description, flexibility, expandability, sustainability, packing (BUFR), readability (CREX= image of BUFR in alphanumeric code ) • For new parameters or new data types, no need to change software, just additional table entries (this is fundamental in light of the fast evolution of science and technology: there are regular requests for representation of NEW DATA TYPES, METADATA, HIGHER RESOLUTION (TIME AND SPACE) AND HIGHER ACCURACY)

35. The Table Driven Codes Forms (TDCF) BUFR/CREX • Offer great advantages compared to the traditional alphanumeric codes (TAC) (2): • The systematic passing of metadata including geographical coordinates (latitude, longitude, height) in every report, which can be easily performed with the table driven codes, would alleviate the notorious WMO Volume A problems. • The Volume A, containing stations coordinates, is updated with too much delay, the WMO secretariat receiving sometimes with considerable delay and other times not at all, the updates that the Countries should send. The use of BUFR or CREX would solve the majority of the cases where there is a problem of wrong coordinates for a station. • The reliability of binary data transmission leads to expectation for an increase in data quality and data quantity received in meteorological centres. • More data and of better quality leading to better data assimilation, and consequently better products: better forecasts, better climate studies and generation of better products by data processing centres

36. SUMMARY:TABLE DRIVEN CODES, LIKE BUFR, GRIB 2 AND CREX OFFER: • SELF DESCRIPTION • FLEXIBILITY • EXPANDABILITY • SUSTAINABILITY • COMPRESSION (PACKING) FOR BUFR • EASY READABILITY FOR CREX • DATA OF BETTER QUALITY LEADING TO BETTER PRODUCTS • IT IS A MUST TO USE THEM IN THE 21ST CENTURY FOR THE SAKE OF SCIENCE EVOLUTION AND PROGRESS

37. THANK YOU FOR YOUR ATTENTION Questions???