PROGRESS ON STREAMLINING THE FORECAST PROCESS VIA A KNOWLEDGE-BASED SYSTEM BMRC Seminar

1. PROGRESS ON STREAMLINING THE FORECAST PROCESS VIA A KNOWLEDGE-BASED SYSTEM BMRC Seminar 23 April, 2003. Harvey Stern (Victorian Regional Office) http://www.weather-climate.com/internetforecasts.html

2. Purpose Previously, reports on this work on streamlining the forecast process via a knowledge-based system available on the Internet have been presented at: • Internal Victorian Regional Office (VRO) Meetings; • A Bureau of Meteorology Research Centre (BMRC) Seminar; • American Meteorological Society (AMS) Annual Meetings. The purpose of the current presentation is to provide an update on how the work is progressing. To begin, maps depicting the area of interest, and an example illustrating the system’s output, are now given.

3. Location Diagram (1)

4. Location Diagram (2)

5. Example (NWP MSL Prognosis)

6. Example (The System’s Output)

7. Background An early version of the system (the PILOT VERSION) was presented to the 18th IIPS Conference (Stern, 2002). The system was developed for the small (227,000 sq km) southeast Australian State of Victoria. It was described as being capable of generating forecasts for public, aviation, marine and media interests, in languages other than English, and for more than 200 localities in Victoria. The large volume of output proposed (forecasts for than 200 localities) would be difficult (if not impossible) to generate utilising the current labour intensive systems. However, it would be straightforward (to generate) utilising automated systems.

8. “Bank” of forecaster experience A major benefit of the knowledge-based system is that it incorporates an extensive "bank" of forecaster experience. Ramage (1993) has proposed an "iterative" approach to "locking in" improvements in forecasting methodology. The automated nature of the system lends itself to Ramage's approach. The system's skill increases as new knowledge is incorporated into its operation. Hence, progress is gradually made towards the realisation of Ramage's dream.

9. Not “yet another” instrument of forecast guidance The system is not seen as "yet another" instrument of forecast guidance. Rather, its development is seen as a logical step along the path of having the computer replicate the best aspects of the manual side of the forecast process, by systematically "locking in" new knowledge. As Brooks (1995), wrote: "technology, which initially allowed humans to make routine weather forecasts, will soon close that avenue of human endeavour ... (and thereby permit) concentration on severe events". The main human-interaction is in utilising forecast verification data (after the event) to iteratively incorporate new additional forecaster knowledge into its algorithm.

10. November 2001 trial The PILOT VERSION's forecast performance (during November 2001) was evaluated for the city of Melbourne. The evaluation showed that, although superiority over climatology was achieved, the forecasts (on most measures) proved to be inferior to the official forecasts. The results of the November 2001 trial were analysed, potential improvements in the forecasting process (employed by the system) were identified, and those improvements were "locked in". The system (so modified), and now termed VERSION 1, was then subjected to another evaluation for the city of Melbourne, this time over 100 days (between late September 2002 and early January 2003).

11. Overall Accuracy (in 2001)

12. Modifications The PILOT VERSION operated by producing its predictions from a restricted number of discrete "forecast sets". The set that was chosen (by the system) was largely determined by the particular synoptic pattern suggested by the selected NWP model. The 2002/2003 modification, VERSION 1, utilises regression analysis to allow predictions to be selected from a continuous array of possible forecasts. The particular form of regression analysis employed is parameter enveloping. Parameter enveloping allows definition of how the various predictors impact upon, or envelope, the influence (on a predictand) of other predictors.

13. An illustration of parameter enveloping

14. Overall Accuracy (2002/2003)

15. RMS Error of Min Temp Forecasts

16. RMS Error of Max Temp Forecasts

17. % Correct Rain/No Rain

18. RMS Error of QPFs

19. A Forecasting Research Tool A simple illustration is now presented of how the system might be employed as a research tool to enhance understanding. The first Figure following depicts the MSL pressure (MSLP) distribution for Synoptic Type 41 (strong cyclonic SSW flow).

20. A Forecasting Research Tool (cont.) The Figure plots the annual march of PoP forecasts for strong cyclonic SSW flow, with 850 hPa temperatures of 0 deg C, & with 700 hPa RHs of 30% & 70%. One might ask why there is such a marked difference between winter & summer responses to the same situation.

21. A Forecasting Research Tool (cont.) An illustration is now presented of how one may proceed from the enhanced understanding achieved via the preceding analysis, to increase the potential accuracy of the forecasts generated. A study of Synoptic Type 41 cases suggests that the strength of the SSW flow between SE Australia and over waters to the west of that region might be significant. Regression analysis confirms that adding cyclonicity to the other predictors, cyclonicity proves to be the most significant.

22. A Forecasting Research Tool (cont.) The Figure plots the annual march of PoP forecasts for strong cyclonic SSW flow with RH=30% and the new equation. It reveals just how important cyclonicity is in assessing the likelihood of precipitation.

23. Need for Further Testing The 100-day verification trial suggests that substantial progress, albeit uneven, towards achieving computer replication of the manual forecast process, has been made. Specifically, the impact of the "locked in" improvements upon the skill displayed by the system has been considerable, particularly for Day-1 forecasts. Nevertheless, the performance of the system over the 100-day trial is so impressive, that one must entertain the possibility that it was simply a "fluke". For this reason, further testing needs to be undertaken.

24. Implications It was considered that, should further testing confirm the 100-day trial results, the implications would be profound. We could see ourselves at the "dawn" of the operational implementation of Ramage's approach to weather forecasting which, through computer replication of the manual process, would allow for: • Systematic incorporation of new procedures that may lead to a quantum leap in the accuracy of the forecast products (in a related piece of work, Ryan et al. (2003) have developed a system that archives the subjective inputs to the forecast process and makes them available for statistical analysis); • The opportunity to greatly increase the number and variety of such products.

25. Further Testing Some further testing of VERSION 1 did, indeed, confirm the 100-day trial results. A modest post-trial exercise, involving the derivation of forecasts for Day-1 only for the (now) 130 days ended February 1, 2003, yielded: • A CSS of 61.4 for the Internet Forecasts; and, • A CSS of 60.3 for the official forecasts.

26. VERSION 2 On the basis of the results of the aforementioned trial of VERSION 1, further knowledge was added to the system. VERSION 2, which extends the outlook from 6 to 7 days is now about to undergo a trial. Among the modifications included in the new version are: • The utilisation of a measure of uncertainty; • “Cyclonicity” as a term in the prediction equations; and, • An adjustment of the output when particularly high relative humidity is indicated. http://www.weather-climate.com/internetforecasts2.html

27. The Measure of Uncertainty The measure of uncertainty provides a "truer" result than conventional ensemble forecasting techniques. The measure is derived from the error distribution of actual forecasts, rather than (as in NWP ensemble forecasting) from an array of model output generated by imposing a random set of perturbations on the initial analysis. This leads to a set of equations of the form … Observed Departure from Normal = Function (Days Ahead, Forecast Departure from Normal) which are then inserted into the system. Conventional ensemble forecasting suffers from the disadvantage of the level of uncertainty in the initial analysis being unknown, whereas the uncertainty associated with a data base of actual forecasts is known precisely.

28. The Measure of Uncertainty (an example: Jan 25, 2003) Note, on next two slides, variation in max temp forecasts due to greater uncertainty at Day 7 than at Day 1

29. The Measure of Uncertainty (an example: Day 1)

30. The Measure of Uncertainty (an example: Day 7)

31. Philosophy It may be appropriate to ask, from a philosophical point of view, whether or not it may be premature, at this stage, to move towards computer replication of the manual forecast process. After all, the new National Digital Forecast Data Base (NDFD) of the U.S.A. National Weather Service (NWS) (Glahn & Ruth, 2003) allows for considerable manual involvement in its operation. A move to computer replication would result in a paradigm shift in the nature of the forecasting meteorologist's role. The role increasingly would become one of utilising sophisticated methodologies to analyse the output of the automated system, and implementing changes to it (consequent upon the analyses).

32. Developments in New Zealand However, having the computer replicate various aspects of the manual forecast process, in order to make possible the production of a greatly increased number and variety of forecast products, is already happening. For example, the highly competitive environment that the New Zealand weather service finds itself has resulted in it moving down this pathway (Linton & Peters, 2003).

33. Developments in the U.S.A. Furthermore, there is pressure in the U.S.A. to allow commercial operators to take over government's traditional role in the provision of weather services (excepting the delivery of urgent warnings to protect life and property). To illustrate, AccuWeather's 31 January 2003 Media Release "expressed regret that the (National Research Council) report did not recommend that the National Weather Service end its practice of issuing routine weather forecasts."

34. Some Concluding Remarks The likely impact of competition Regardless of philosophy and outside of the legislative framework, competitive pressures may determine the future. Private operators (and, also the general public) will realise that the new technologies allow for the development and implementation of forecasting systems capable of providing a breadth of output far greater than one could ever hope to produce utilising the current approaches.