Getting better value from forecasting software: -where can the improvements come from?

1. Getting better value from forecasting software:-where can the improvements come from? Paul Goodwin University of Bath Supported by: Robert Fildes, Kostas Nikolopoulos, Wing Yee Lee, Michael Lawrence

2. Project aim: To recommend practical improvements to forecasting software support systems through: • system design changes • process improvements • statistical modelling improvements

3. Main companies involved in research • Pharmaceutical company • Food manufacturer • Domestic cleaning products manufacturer • Major U.K. Retailer • Brewer

4. Company approaches to forecasting • Most has very large number of series to forecast - either monthly or weekly • All used a software system that embodied statistical time series forecasting methods • All judgmentally adjusted a large number of these forecasts -most adjustments were made by groups of managers in review meetings

5. Data supplied by companies • Software forecast • Final forecast (i.e. software forecast after judgmental adjustment) • Actual outcome We also sat in on forecasting review meetings and observed & discussed use of software with individual forecasters

6. Percentage of system forecasts that are adjusted

7. What we have found • Many examples of good practice • But some common issues. • How can software use & design address these issues?

8. On average did judgmental adjustments improve accuracy? • Where forecasts were adjusted (excluding the retailer):

9. But evidence that adjustment process can be improved… E.g. In one company: While adjustments lead to mean improvement in absolute % error of: 4.31% • Half improvements less than 0.37% • Only 51.3% of system forecasts improved through MI adjustment • What type of adjustments improve accuracy? • Which are the most damaging to accuracy?

10. 1. Only larger adjustments tend to improve accuracy… Data from one typical company…Company X * Size of adjustments measured as absolute adjustment as % of system forecast

11. 2. Negative adjustments are more effective, on average…. (Data is from all companies except retailer) Some very large errors distort the mean here

12. 3. Wrong direction adjustments are particularlydamaging… • E.g. Adjusted forecast System forecast Actual demand Mean absolute % error worsened by: 57.5% in company 1 67.8% in company 2 101.5% in company 3

13. Positive adjustments are more likely to be in the wrong direction… (Data is from all companies except retailer)

14. Smaller adjustments are more likely to be in the wrong direction… (Data is from all companies except retailer)

15. Optimism bias? • Final adjusted forecasts tend to be too high: by an average of18.1% • System forecasts that are unadjusted on average are to high by 44. 3% -so people seemed happy with these high forecasts (Data is from all companies except retailer)

16. The retailer

17. Distinguishing between forecasts and decisions • Make the forecast first e.g. “I think we’ll sell 200 units” • Then you can turn it into a decision e.g. “I think we ought to produce 250 units, in case demand is unexpectedly high”

18. Summary of data analysis • Avoid small adjustments -they tend to reduce accuracy • Be careful with positive adjustments -they are often made when the system forecast actually needs to be reduced • Beware optimism bias • Be careful to distinguish between forecasts & decisions

19. Can software help to solve the observed problems? • Analogies –to tackle optimism bias and improve accuracy adjustments • Profiles – to reduce wrong sided adjustments • Advice – to reduce small adjustments etc • Restrictiveness -to reduce small adjustments • Less severe restrictiveness -to deter trivial or unnecessary adjustments

20. Your views • After presentation of each idea please indicate your views on the questionnaire

21. Analogies E.g. a promotion campaign will take place next month • Software has access to a database of past promotions • It selects 3 most similar promotions to forthcoming promotion and displays their estimated effect on demand • It allows you to estimate the effect on demand of differences between forthcoming promotion & the selected similar promotions…

22. Adaptation judgment support Click here for the interface in the demo Similarity judgment support Memory support

23. Profiles of the effects of events E.g. a promotion campaign will take place next month for 3 weeks Our forecasts need to take into account the timing of its effects: Pre-promotion dip

24. Time series graph Free-plotting allowed Gallery of promotion demand profiles The demand profiles in the gallery are averages of the past promotions of the same promotion type, e.g. 40% Off. Experimental evidence has shown that using these profiles could improve forecast accuracy significantly regardless of the predictability of the environment

25. On screen guidance • System tells you when it thinks you should or should not make adjustments…

27. Restrictiveness System will not allow judgmental adjustments below a certain size…

29. Less severe restrictiveness System requires user to give a reason for adjustment before it allows the adjustment to be made…

31. Questions for breakout sessions • Are you satisfied with the way your software produces its statistical forecasts? If not what improvements would you like to see? • Do you think that your forecasting software could provide better support for including market information through your group's judgmental adjustments? If so how? • Are there any improvements you would like to see in the way your software can be used to analyse past errors and report forecasting accuracy?

32. End

33. Back to main slides Future promotion Step 3: Adaptation judgment support Step 2: Similarity judgment support Step 1: Memory support (Details of the three support features may be viewed by clicking on the labels.)