WMO STRATEGY FOR MIGRATION TO TABLE DRIVEN CODE FORMS

1. WMO STRATEGY FOR MIGRATION TO TABLE DRIVEN CODE FORMS 18 April 2007 (Joël Martellet, WMO, World Weather Watch, Data Processing and Forecasting Systems)

2. OUTLINE • THE SITUATION IN 2007 • A MIGRATION WITH GOALS AND CONSTRAINTS • IMPACTS ON WORLD WEATHER WATCH DATA FLOW AND ASSOCIATED PROGRAMMES • MIGRATION PLAN • IMPACT ON MEMBER STATES • MIGRATION SCHEDULE • SOFTWARE AVAILABLE • STATUS • ISSUES

3. Observations data flow in 2007 • For the operation of the World Weather Watch, traditional meteorological and marine in-situ observations are still exchanged between Countries in Traditional Alphanumeric Codes (TAC). Although their total volume is only several tenth of megabytes (in comparison to about 1.5 GB for BUFR satellite data) per day, it is still (for many years) a required input for all meteorological applications - the number of observations for some types exchanged in one day (1 March 2007) is indicated: • FM 12 SYNOP (65077), FM 13 SHIP (7825), FM 18 BUOY (22016) FM 32 PILOT (855), FM 35 TEMP (1260), AIREP (23207) (from ECMWF Global Data Monitoring Report)

4. Data type Number of bulletins Volume in bytes AMDAR 4371 2 368 667 ATOVS retransmitted by EUMETSAT 7411 98 518 455 ATOVS data 236 300 507 877 149 AVHRR radiances 12 395 8 832 569 Buoy data 3 505 5 159 848 Wind profiler data 6 557 4 233 909 TRMM radiances 97 101 284 840 312 ACAR aircraft data 429 3 389 556 Geostationary radiances and winds 23 427 97 789 699 Tropical storm data 14 4 594 ERS2 wind and wave 982 3 224 970 Ice concentration data 24 11 012 HD winds Terra satellite 764 4 591 694 CREX ozone data 11 81 040 MSG radiances, winds 10 581 152 304 931 EUMETSAT HD winds 348 3 906 782 Goes HD winds 2033 15 995 910 EUMETSAT radiances 1 350 17 899 182 Jason radiances 6 8 322 GPS surface pressure 309 1 414 133 TAMDAR aircraft data 1724 809 590 Total 1 213 262 324 DATA in BUFR or CREX already on GTS (e.g. 26/09/2005)

5. Data type Single observation size Number of obs/day Volume in bytes SYNOP 332 65077 21 605 564 PILOT 400 855 342 000 TEMP 1500- 1260 1 890 000 CLIMAT SYNOP ? ? CLIMAT TEMP 600 370 222 000 AWS 264-320 SAREP 146 SATEM 190 30 047 5 708 930 SATOB 272 116 10 918 212 METAR 178 42000 7 476 000 TAF ? AMDAR 104 2 368 661 BUOY Up to 398 22016 8 762 368 SHIP Up to 332 7825 Up to 2 597 900 CLIMAT TEMP 600 370 222 000 ESTIMATE OF DATA VOLUME DAILY ON THE GTS OF TRADITIONAL OBSERVATIONS IF ALL MIGRATED TO BUFR: ABOUT 60 À 65 MBYTES

6. WMO decisions on Table Driven Code Forms • WMO Executive Council approved as operational codes: BUFR in 1988, CREX in 2000, GRIB Edition 2 in 2001 • The Commission for Basic Systems (CBS XII - December 2000) agreed to generate a PLAN for the migration from Traditional Alphanumeric Codes to Table Driven Codes and established an Expert Team on Migration to Table Driven Codes. • The Team developed a PLAN. • CBS Ext. (2002) endorsed the PLAN. • The XIV WMO Congress approved the PLAN. • http://www.wmo.int/web/www/WMOCodes/MigrationTDCF/SummaryMigraPlan_en.pdf • CBSXIII (March 2005) established the Coordination Team on Migration to Table Driven Code Forms (MTDCF).

7. MIGRATION GOAL • ULTIMATELY ALL OBSERVATIONS TO BE EXCHANGED IN BUFR (WHICH OFFERS MORE FEATURES THAN CREX, E.G. QUALITY FLAGS, ASSOCIATED VALUES, PACKING). • FOR SOME COUNTRIES, HOWEVER, CREX CAN BE AN INTERIM SOLUTION.

8. CBS defined constraints for the MIGRATION to Table Driven Codes BUFR/CREX • CBS REQUESTED: • IT SHOULD BE A SMOOTH TRANSITION WITHOUT NEGATIVE IMPACTS ON THE WORLD WEATHER WATCH OPERATIONS • NOT ALL AT ONCE, BUT PROGRESSIVE AND VOLUNTARILY, BY COUNTRY OR PRODUCER, BY DATA TYPE AND EVEN BY STATION OR PLATFORM. • THE PLAN FOR THE MIGRATION TAKES INTO ACCOUNT THESE PRINCIPLES.

9. THE BASIC PRINCIPLES OF THE PLAN ARE: • THE PLAN SHOULD ALLOW FOR EVERY WMO MEMBER TO MIGRATE. • ALL WMO MEMBER STATES SHOULD HAVE THE FREEDOM TO “SWITCH” INTERNATIONAL TRANSMISSION TO BUFR (OR CREX) WHEN THEY WANT AND WHEN THEY ARE READY TO DO SO. • THE DATA PRODUCER, NOT THE USER, IS THE INITIATOR OF THE MIGRATION PROCESS. • THE MIGRATION WOULD BE A LONG-TERM PROCESS WITH CONSIDERABLE FLEXIBILITY. • INCENTIVES FOR MEMBER STATES TO MIGRATE TO TABLE DRIVEN CODE FORMS SHOULD EXIST. THE BENEFITS OF THE MIGRATION HAVE TO BE CLEARLY EXPLAINED TO MEMBERS. • MANUFACTURERS OF OBSERVING SYSTEMS AS WELL AS DEVELOPERS OF PROCESSING SOFTWARE (INCLUDING WORKSTATIONS) SHOULD BE MADE WELL AWARE OF THE PURPOSE OF THE MIGRATION AND OF ITS BENEFITS.

10. CONSIDER THE WORLD WEATHER WATCH DATA FLOW:

11. EACH WMO MEMBER STATE should find:What is the level of automation? Teletype Paper Manual plotting Forecaster Observer encoding on Teletype

12. Automation for data processing: Automatic decoding, plotting,display program Forecaster Observer encoding on Teletype

13. Automation for data processing and some at observing level Automatic decoding, displaying program Forecaster Encoding program Observer Typing through an encoding program

14. Total automation: Automatic decoding, displaying program Forecaster Automatic Weather Station (AWS) automatic encoding

15. Different functions within the data flow: • Not to confuse the functions of: • Data acquisition • Data collection • Data transmission • Data reception • Data visualization • The format used to represent data could be different at each stage, if it is more efficient.

16. The format used to represent data could be different at each stage, if it is more efficient. Not to confuse the functions of: • Data acquisition: AWS sensors value format (or observer reporting data) • Data collection: Encoding in a data representation form the observer report (or the AWS sensors values) for national collection • Data transmission: Keeping format or Encoding with a view to perform international transmission • International transmission process: If possible do not change the data representation, just move a mail “envelope” • Data reception: Decode the data representation form to feed data base or other processing applications • Data visualization: Visualize the data values for a human reader = convert to the most appropriate display format for the user

17. WMO OBSERVATION DATA PRODUCERS(Global Observation System) • 188 NMCs PRODUCE TRADITIONAL OBSERVATIONS: • SYNOP, TEMP, PILOT, CLIMAT (also METAR; SPECI; TAF) • A CERTAIN NUMBER OF SPECIALISED COLLECTING CENTRES PRODUCE: • SATELLITE DATA (THE MAJORITY ALREADY IN BUFR) • AIRCRAFT DATA (AMDAR, ACARS (MANY ALREADY IN BUFR), AIREP) • SHIP DATA • BUOY DATA • XBT/CTD • SUB-SURFACE PROFILING FLOATS DATA • PRODUCERS WILL HAVE THE FREEDOM TO SWITCH TO BUFR WHEN THEY NEED (INTEREST OF THEMSELVES OR SOME OF THEIR USERS FOR NEW PARAMETERS, DATA TYPES, METADATA). HOWEVER, ONE HAS TO ENSURE THEIR OTHER USERS WILL STILL HAVE ACCESS TO THE DATA (ONE HAS TO ANALYSE DATA FLOW AND USER REQUIREMENTS)

18. The Global Observing System (GOS) and the concept of temporary double-dissemination (or dual transmission) for the migration : BUFR and CREX, or BUFR and TAC

19. Data conveyors: 31 RTHs (GTS)

20. Later: WMO Information System (WIS): Information collection data flow

21. WIS: Information distribution data flow Managed data communication network

22. WIS DATA FLOWS

23. Impact of the migration for a Regional Telecommunication Hub (RTH) or DCPC or GISC which receives data (routine operational transmission: store and forward or push system): • in Traditional Alphanumeric Codes (TAC) • retransmits in TAC • in CREX: • retransmits in CREX • in BUFR: • retransmits in BUFR • Conversion BUFR to TAC or BUFR to CREX (only if possible and easy and if associated NMCs really need it, that is: only if there is no double dissemination) • Message Switching Directories will have to be updated (Same MSS for BUFR or CREX bulletins as for TAC. Hence, new bulletin header system has been developed by WMO telecommunications experts (eg. TEMP, PILOT BUFR: 2 messages (1st) up to 100 hPa, (2nd) all measurements): • http://www.wmo.int/web/www/ois/Operational_Information/TDCF/Table1.html

24. Real time operational DATA USERS • The Global Data Processing System : • 16 Centres running operationally global models • 80 Centres running operationally regional or mesoscale models • 188 (all NMCs) Centres performing meteorological applications (about 30 are not automated!) • Who has a BUFR decoder? A universal one? • Who has a CREX decoder? A universal one?: • First priority of the migration: • universal BUFR and CREX decoder for all!

25. GDPS CENTRES: WMCs; RSMCs; WAFCs

26. GDPS CENTRES RUNNING MODELS GLOBAL DATA PROCESSING AND FORECASTING SYSTEMS

27. The migration requires preparatory corrective actions at GDPS Centres to avoid missing data: • Some Centres will have to ensure that their BUFR and CREX decoders in their processing chain is fully universal, to avoid missing observations. • Other Centres will have to implement from scratch a universal BUFR and CREX decoders, to be able to receive observations in BUFR or CREX. • Manufacturers of "turn key" work-stations inputting GTS data would need to be approached so that they include in their software universal BUFR and CREX decoders, either in the existing systems, or for the new systems currently in development or to be developed. • Centres, which are currently operating manually (about 30 over 188), will have to seriously consider automation with software including a universal BUFR and CREX decoder. Before automation is implemented, or before they have a “binary” connection, they could receive and understand CREX messages with training, and also encode and transmit observation reports in CREX. • For some Centres the Internet could allow access to data in binary formats that are not available over their GTS link.

28. ACTIONS ALREADY UNDERTAKEN (1) • CBS (Ext. 2006) HAS APPROVED TEMPLATES AND REGULATIONS FOR TRANSMITTING ALL TRADITIONAL OBSERVATIONS IN BUFR (Work of Dr Eva Cervena) • ESSENTIAL TO HAVE DESCRIPTORS FOR ALL PARAMETERS THAT CAN BE ENCODED IN A TRADITIONAL CODE FORM INCLUDING REGIONAL AND NATIONAL PRACTICES • AND TEMPLATES = LAYOUT OF THE FORMAT, LIST OF DESCRIPTORS TO BE TRANSMITTED (COMMON SEQUENCES) • TEMPLATES DEFINED FOR: SYNOP, SHIP, SYNOP MOBIL, PILOT, PILOT SHIP, PILOT MOBIL, TEMP, TEMP SHIP, TEMP MOBIL, TEMPDROP, CLIMAT, CLIMAT SHIP, CLIMAT TEMP AND CLIMAT TEMPSHIP • CBS has approved amendments for GTS exchanges to cope with BUFR and CREX bulletins, in particular Rules for transmitting NIL BUFR/CREX bulletins or reports for monitoring purpose>>>>>>>>>>>>>>>>>>>>>>>>>>>>

29. BUFR/CREX TEMPLATES have been defined and some are used for: • BUOY, Sub-surface float, TESAC and BATHY data • TRACKOB data (approved by CBS) • Single level Standard AMDAR (approved by CBS) • More complex Single level AMDAR data (approved by CBS) • Ascent/descent AMDAR profile data (approved by CBS) • Old AIREP • SAREP data (Part A: Information on tropical cyclone) (approved by CBS) • RADOB data (Part A: Information on tropical cyclone) (approved by CBS) • AUTOMATIC WEATHER STATION DATA OF: • - N-MINUTE PERIOD • - ONE-HOUR PERIOD • FOR TROPICAL CYCLONE TRACKS DERIVED FROM EPS • EPS GRAMS • Buoy data including directional and non-directional wave data (not validated) • VOS SHIP (not validated) • METAR/SPECI (not validated) and TAF (not validated)

30. CREX TEMPLATES have been defined and some are used for: • Bloemhof Flood Monitoring Code for Hydrology • Tide Gauge data • Total Ozone Measurement • Ozone Sounding Coupled to a Brewer Spectrophotometer • Ozone Sounding not Coupled to a Brewer Spectrophotometer • Tropical Cyclone information • Monitoring Information • CREX sequences for coding of Squall Lines in West Africa (approved by CBS) • CREX Template for synoptic reports from fixed land stations suitable for manual encoding (in validation) • New CREX templates for Tide Elevation (in validation)

31. THE WMO SECRETARIAT HAS DESIGNED A WEB PAGE WITH MUCH INFORMATION ON CODES AND MIGRATION TO TDCF. IT INCLUDES, AMONG OTHER INFORMATION, THE INTERNATIONAL MIGRATION PLAN AND A NEW GUIDE ON BUFR/CREX WITH 3 LAYERS : http://www.wmo.int/web/www/WMOCodes.html For training 3 levels have been defined: LAYER 1: FOR GENERAL PHILOSOPHY LAYER 2: FOR APPLICATION INTERFACING USERS, DATA AND TELECOMMUNICATION MANAGERS LAYER 3: FOR ENCODER/DECODER PROGRAMMERS WMO ORGANISED TRAINING SEMINARS (Level 1 and 2) : 2003 in RA1 (English), in RA III/IV, 2004 in RA1 (French), in East RA II/West RA V, 2005 in West RA II/East RA VI, 2007 in West RA VI (also a bit Level 3) (about 130 countries have been covered so far) ACTIONS ALREADY UNDERTAKEN (2)

32. ACTIONS EXPECTED FROM ALL WMO MEMBER STATES • DEFINE MIGRATION CONTACT POINTS (national focal point on Codes matters) (105 over 188 so far) does your country has one? • NOMINATE A NATIONAL STEERING GROUP ON MIGRATION TO TDCF (TO INCLUDE THE NATIONAL FOCAL POINT) • DEFINE A NATIONAL PLAN FOR MIGRATION TO TDCF FOLLOWING THE INTERNATIONAL PLAN - FOUND IN WMO WEB SITE: • http://www.wmo.int/web/www/WMOCodes.html • IDENTIFY IMPACTS OF MIGRATION ON NATIONAL OPERATION • IDENTIFY SOME NECESSARY (BUT USUALLY LIMITED) FINANCIAL AND STAFF RESOURCES • DEFINE and RUNNATIONAL TRAINING PROGRAMME ON TDCF • DEFINE AND PERFORM SOFTWARE ADJUSTMENTS: • 1)process BUFR/CREX bulletins in MSS • 2)introduce BUFR/CREX processing with decoder in PROCESSING CHAIN: • RECEPTION, DECODING, DISPLAY, ARCHIVING • 3)introduce BUFR/CREX encoder for the concerned data type at: OBSERVATION or/and CONCENTRATION site (test and check format) • 4) transmit BUFR/CREX Bulletins (with proper headers after notification to WMO Secretariat)

33. MIGRATION: WHAT ARE THE KEYS TO SUCCESS? (1) • MIGRATION TO TDCF WILL GO WITH AUTOMATION • CANNOT BE UNDERTAKEN WITHOUT • CAREFUL PLANNING • TRAINING • SOME RESOURCES: • COMPETENT STAFF • FINANCIAL • INTERNATIONAL COOPERATION (GROUPING FOR PROJECTS) • INTERNATIONAL ASSISTANCE

34. MIGRATION: WHAT ARE THE KEYS TO SUCCESS? (2) • TRAINING ON BUFR/CREX - WHAT IS IT?: • LEVEL 1: FOR GENERAL PHILOSOPHY • LEVEL 2: FOR APPLICATION INTERFACING USERS, DATA AND TELECOMMUNICATION MANAGERS • LEVEL3: FOR ENCODER/DECODER PROGRAMMERS • WMO SEMINARS: • SEMINAR FOR WMO REGIONS • TRAIN ALSO THE TRAINERS (WMO RMTCs, METEOROLOGICAL SCHOOLS, COLLEGES, UNIVERSITIES) • TRAINING WITHIN COUNTRIES: NATIONAL TRAINING PROGRAMMES • SOFTWARE PROJECT: • SOFTWARE HOUSE: A CENTRE DISTRIBUTING FREE BUFR/CREX (GRIB 2) DECODER and ENCODER SOFTWARE AND PROVIDING REMOTE ASSISTANCE FOR IMPLEMENTATION, DOCUMENTATION, ANSWERING QUERIES • ECMWF: encoder/decoder BUFR, CREX and GRIB1/2, transcoder CREX-BUFR (FORTRAN, C languages and UNIX, LINUX operating systems, on Windows if emulator UNIX CYGWIN) • NOAA/NCEP: software under UNIX • UNIX For WINDOWS: Brazil (INPE software), German software (BUFR), Slovak software (private company),

36. ACTIONS BY ASSOCIATED PROGRAMMES • THERE ARE OTHER PRODUCERS AND USERS • SATELLITE PRODUCERS PRODUCE ALREADY IN BUFR • OTHER PRODUCING CENTRES WILL START TO SWITCH TO BUFR • MARINE (JCOMM), NEED FOR BUFR ENCODERS, DOUBLE DISSEMINATION FOR USERS: • Service ARGOS has started dissemination in BUFR for buoys, XBT/CTD and sub-surface profiling floats • ICAO: ACARS, AMDAR TRANSMITTED IN BUFR, SIG WEATHER DATA TRANSMITTED IN BUFR, SIGMET IN BUFR BUT FAR AWAY FOR METAR/SPECI/TAF ! • USERS WILL HAVE TO BE PROVIDED WITH BUFR/CREX DECODERS

37. Schedule • ORGANIZED BY DATA CATEGORIES WHICH HAVE BEEN DEFINED. • DIFFERENT STEPS: • START OF EXPERIMENTAL EXCHANGE: data will be made available in BUFR (CREX) but not operationally, i.e. in addition to the current alphanumeric codes, which are still operational. Relevant conditions to be satisfied before experimental exchange may start: • Corresponding BUFR/CREX-tables and templates are available; • Training of concerned testing parties has been completed; • Required software of testing parties (encoding, decoding, viewing) is implemented; • START OF OPERATIONAL EXCHANGE: data will be made available in BUFR (CREX) whereby some (but not all) Members rely on them operationally. Still the current alphanumeric codes will be distributed (double dissemination or dual transmission ). Relevant conditions to be satisfied before operational exchange may start: • Corresponding BUFR/CREX-tables and templates are fully validated; • Training of all concerned parties has been completed; • All required software (encoding, decoding, viewing) is operational. • MIGRATION COMPLETE: at this stage the BUFR (CREX) exchange becomes the standard WMO practice. Double dissemination or dual transmission is terminated. For archiving purposes and at places where BUFR (CREX) exchange still causes problems, the alphanumeric codes may be used and disseminated on a local basis only.

38. MIGRATION SCHEDULE All dates above are meant as "not later than". However, Members and Organizations are encouraged to start experimental exchange, and, if all relevant conditions are satisfied, to start operational exchange as soon as possible. MIGRATION COMPLETE: For archiving purposes and at places where BUFR (CREX) exchange still causes problems, the alphanumeric codes may be used on a local basis only.

39. Status of the migration to BUFR/CREX (information on 3/2007) • BUFR is used for archives of all data types and operational exchange of satellites data, ASDAR, AMDAR ,wind profilers, tropical cyclone data, ARGOS data: buoy, XBT, XCTD, sub-surface floats, RADAR data and starts to be used for translating Traditional Alphanumeric Codes (TAC). • RA I: Some African countries are working seriously on the migration: e.g. Botswana, Ethiopia, Morocco, Mozambique, Senegal, Tanzania. Some are considering to use CREX (more metadata more precision and more flexibility compare to traditional SYNOP code) – Senegal produces squall line data in CREX • RA II: Japan Meteorological Agency is disseminating since November 2005 in BUFR all TAC data (SYNOP, SHIP, PILOT; TEMP, CLIMAT, etc..), BUOY, BATHY, TESAC; TRACKOB, tropical cyclone, wind profiler data and RADAR data - Hong-Kong produces AMDAR - China will produce SYNOP in BUFR in 2007. • RA III: Brazil and Argentina work seriously on development for implementation. • RA IV: USA produces internally upper-air data in BUFR, and sea level data in CREX. Mexico tested encoding of SYNOP, TEMP, CLIMAT and CLIMAT TEMP. Costa Rica works on implementation. • RA V: Australia produces tropical cyclone data in BUFR and in CREX will produce soon SYNOP in BUFR for Australia and Papua New Guinea and works for other data types - Fiji produces tropical cyclone data in CREX • RA VI: Czech Republic: SYNOP, TEMP and AWS data, Israel: TEMP, SYNOP and CLIMAT data, Jordan: SYNOP data, Germany: AWS data, Netherlands: AWS and TEMP data (Note: “TEMP data in BUFR” means radiosonde data including time and position of the radiosonde at each level) - Météo-France disseminates JASON 1 satellite data in BUFR.

40. ACTIONS PLANNED • Development (2007) of a data verification service to help testing encoding of BUFR messages • Development (2007) of a migration guidance document explaining how to start migration • Manufacturers are preparing: for radiosondes and for AWS data in BUFR • Training material (2007) to be put into the WMO web site

41. ISSUES • Misunderstanding: migration is not a drastic brutal change (as previously for TAC codes), but slow and smooth with dual dissemination (TAC and BUFR in parallel), TARGET DATES in the plan, Countries should plan migration targeting those dates • Lack of visibility for the migration, for its necessity • Initial purpose: for operational real-time exchange of data between NMHS (not for external users: e.g. OPMET) • Because it affects the whole meteorological observation data flow: it requires a lot of attention and commitment by NMHS • Misunderstandings- Confusions: • between visualization and transport format: • Exchange process = several steps: • producing (encoding), transport, decoding, DB storage, visualizing, archiving • between physical representation (or format) and envelop (wrapper) (e.g. XML and NetCDF are wrappers, BUT standards of physical representation within the general standard have to be defined). • XML and Net CDF are not alternative to BUFR and GRIB 2: the physical meteorological standards within XML and Net CDF would have to be entirely defined. Many years of work would be needed (it would be again other migration processes!). • THESE CONFUSIONS SLOW THE PROCESS FOR MIGRATION TO TDCF!

42. Summary: ACTIONS NEEDED • Countries not sufficiently informed, Secretariat needs to: • To send more circular letters with information on migration • Improved and make up-to-date WMO web site • Countries should develop their national migration plan and implement it (including national training) – The FIRST ACTION, if NMC is automated is to implement a universal BUFR/CREX decoder • Help for developing countries still needed(Migration goes with automation): • Training workshops • Assistance by expert missions • Pilot projects to demonstrate implementation • Development of encoding/decoding software under Windows would help • Advanced countries should be models and leaders for the migration • Bring more considerations on TDCF migration by WMO constituent body meetings (CBS, RAs, EC, Congress)- WMOBUDGET REQUIRED!

43. THANK YOU FOR YOUR ATTENTION Questions???