1 / 26

Against your better judgment? How organizations can improve their use of management judgment in forecasting

Research Grants GR/60181/01 and GR/60198/01. Against your better judgment? How organizations can improve their use of management judgment in forecasting. Robert Fildes, Lancaster University, UK Paul Goodwin, University of Bath, UK.

elvin
Download Presentation

Against your better judgment? How organizations can improve their use of management judgment in forecasting

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Research Grants GR/60181/01 and GR/60198/01 Against your better judgment? How organizations can improve their use of management judgment in forecasting Robert Fildes, Lancaster University, UK Paul Goodwin, University of Bath, UK

  2. In 2001, 40 international experts published a set of principles to guide best practice in forecasting. -see www.PrinciplesofForecasting.com • Some of the principles relate to the use management judgment in forecasting. -we identified 12 relevant principles • Six years on - is judgment being used in organizations in accordance with these principles?

  3. Our survey 144 forecasters attending 5 US practitioner conferences + 5 practitioners in 4 companies where we were carrying out case-study work In these companies we also observed the forecasting processes and collected data on forecasts & outcomes So: n= 149

  4. Details of respondents • Wide range of industries including: - food manufacturing, - telecoms, - insurance, - banking, - pharmaceuticals, - heavy-duty transportation, - real-estate, -cosmetics, -home-videos, -publishing, -toys, -greetings cards

  5. Responsible for forecasting between 1 and 34,000,000 different series [Median 400] Highest number was for a retailer who was responsible for forecasting demand for 75000 items in 450 different stores.

  6. Note: Several respondents indicated more than one lead time

  7. 89.5% of respondents agreed principal objective of their forecasting process is to produce as accurate forecasts as possible with resources available.

  8. Did forecasters stick to the principles? • The principles provide guidance on: 1. When to use judgment 2.Howto use judgment, and 3. How to assess the effectiveness of judgment.

  9. 1. When to use judgment Principle 1: Use quantitative rather than qualitative methods Principle 2: Limit subjective adjustments of quantitative forecasts Principle 3: Adjust for events expected in the future

  10. More judgment is being used than would be suggested by application of Principle 1 “Use quantitative methods...” and Principle 2“Limit subjective adjustments” Respondents rated importance of judgment, compared to statistical methods, in their forecasting (1=not used, 5 = very important). Mean response = 4.1 34.0% indicated that judgment was ‘very important’

  11. Results accord with Principle 3 -forecasters prepared to apply their judgment to take into account events that were expected in the future

  12. Other reasons for adjustments • 59.1% made adjustments to reflect relative benefits of over and under forecasting. -breaks a fundamental tenet of decision analysis: the need to distinguish between what we prefer and what we forecast. • 63.9% indicated that under forecasting was the most costly -broad thrust of evidence supports the existence of a tendency to forecast too high

  13. 1. When to use judgment (continued) Principle 4: Make forecast independent of organizational politics

  14. Other reasons for adjustments • 52.3% indicated that their forecasts were changed by senior management -in 28.2% of cases this was done without consultation. -consistent with a large no. of forecasts being judgmentally adjusted for political reasons. This would clearly transgress Principle 4 “Make forecasts independent of ..politics”

  15. 2. How to apply judgment Principle 5: Ask experts to justify their forecasts in writing

  16. Advantages of documenting rationale underlying judgment • In one study documenting rationale reduced the no. of unnecessary & damaging adjustments to stats. forecasts from 85% to 35% -People feel more accountable -Helps forecasters to reflect on their rationale. - Requires effort so people more reluctant to apply judgment where potential benefits are questionable. • Can allow people to learn about when the use of judgment is appropriate.

  17. 63.8% claimed that they did document rationale for judgmental adjustments Suggests widespread adherence to Principle 5 -But in our case-study companies most documentation was indecipherable

  18. 2. How to apply judgment (continued) Principle 6: Use structured procedures to integrate judgmental and quantitative methods Principle 7: Combine forecasts from approaches that differ Principle 8: When combining, start with equal weights

  19. 16.7%of respondentsused averages of statistical and judgmental forecasts • But none of our 4 case-study companies used structured methods or combined independent judgmental & statistical forecasts Little evidence of widespread adherence to Principles 6,7 & 8

  20. 3. Howto assess the effectiveness of judgment Principle 9: Compare track records of various forecasting methods. Principle 10: Seek feedback about forecasts

  21. Only44.3%said they reviewed whether judgmental adjustments to statistical forecasts improved accuracy. • Yet making judgmental adjustments can involve considerable amount of time and effort. • In one of our case companies: about 80 person-hours of management time spent in 17 forecasting review meetings each month Suggests widespread violation of Principle9 “Compare track records” & Principle 10 “seek feedback…”

  22. 3. Howto assess the effectiveness of judgment (cont.d) Principle 11: Use error measures that adjust for scale in the data Principle 12: Use multiple measures of forecast accuracy

  23. Not an error measure. Also some reported using R2 and AIC Widespread use of the MAPE suggests that many forecasters are in accord with Principle 11

  24. Only 30.2% of respondents reported using multiple error measures (Principle 12) • 26.2% did not indicate that they used any error measure at all.

  25. Did forecasters think their judgmental adjustments to statistical forecasts improved accuracy? -Suggests a median improvement of about 7 percentage points -In our 3 manufacturing case-study companies we found a median improvement of between 2.6 and 5 percentage points -Our retailer’s adjustments reduced accuracy considerably

  26. Conclusions • Many organisations rely too heavily on unstructured judgment rather than statistical methods • Many blurred forecasts with decisions & plans • Many forecasts are adjusted by senior managers often without consultation and possibly for political reasons. • Less than 50% of respondents reviewed whether their judgmental interventions improved accuracy • More than a third did not record reasons for these interventions. • Relatively few used multiple measures of accuracy. • More than a quarter did not indicate that they used any error measure at all.

More Related