Evaluation

1. Evaluation • The evaluation of Naive Forecasting Techniques relies primarily on the comparison of the forecasts with the corresponding actual values

2. Evaluation Methods • Mean Error (ME) • Mean Absolute Error (MAE) • Mean Squared Error (MSE) • Mean Percentage Error (MPE) • Mean Absolute Percentage Error (MAPE)

3. The ME can be very misleading. A ME value of zero can mean that the method forecasted the actual values perfectly (unlikely) or that the positive and negative errors cancelled each other out. It tends to Understate the error in all cases.

4. Evaluation Methods • Mean Error (ME) • Mean Absolute Error (MAE) • Mean Squared Error (MSE) • Mean Percentage Error (MPE) • Mean Absolute Percentage Error (MAPE)

5. MAE is a way of dealing with the Understatement of ME. By using the Absolute values of the error, the mean gives a better indication of the model’s fit.

6. Evaluation Methods • Mean Error (ME) • Mean Absolute Error (MAE) • Mean Squared Error (MSE) • Mean Percentage Error (MPE) • Mean Absolute Percentage Error (MAPE)

7. The MSE eliminates the positive/negative problem by squaring the errors. The result tends to place more emphasis on the larger errors and, therefore, gives a more conservative measure than the MAE.

8. The previous three measures are “series specific;” i.e., they only allow evaluation of the series that generated the errors. • The next two measures, by using the percentage of the error relative to the actual, are designed to allow comparison of the results with different models.

9. Evaluation Methods • Mean Error (ME) • Mean Absolute Error (MAE) • Mean Squared Error (MSE) • Mean Percentage Error (MPE) • Mean Absolute Percentage Error (MAPE)

10. The MPE is a relative measure of the forecasting error. It is subject to the “averaging” of the positive and negative errors.

11. Evaluation Methods • Mean Error (ME) • Mean Absolute Error (MAE) • Mean Squared Error (MSE) • Mean Percentage Error (MPE) • Mean Absolute Percentage Error (MAPE)

12. MAPE is a comparative measure that does not have the problem of averaging the positive and negative errors. It is relatively easy to use to communicate a model’s effectiveness.

13. Measurement of Forecasting Error • Mean Error (ME): The average of all the errors of forecast for a group of data. • Mean Absolute Error (MAE): The mean, or average of the absolute values of the errors. • Mean Square Error (MSE): The average of the squared errors. • Mean Percentage Error (MPE): The average of the percentage errors of a forecast. • Mean Absolute Percentage Error(MAPE): The average of the absolute values of the percentage errors of a forecast.

14. Year Actual Forecast Error 1 1402 2 1458 1402.0 56.0 3 1553 1441.2 111.8 4 1613 1519.5 93.5 5 1676 1584.9 91.1 6 1755 1648.7 106.3 7 1807 1723.1 83.9 8 1824 1781.8 42.2 9 1826 1811.3 14.7 10 1780 1821.6 -41.6 11 1759 1792.5 -33.5 Example: Nonfarm Partnership Tax Returns: Actual and Forecast with  = .7

15. Year Actual Forecast Error 1 1402.0 2 1458.0 1402.0 56.0 3 1553.0 1441.2 111.8 4 1613.0 1519.5 93.5 5 1676.0 1584.9 91.1 6 1755.0 1648.7 106.3 7 1807.0 1723.1 83.9 8 1824.0 1781.8 42.2 9 1826.0 1811.3 14.7 10 1780.0 1821.6 -41.6 11 1759.0 1792.5 -33.5 524.3 Mean Error for the Nonfarm Partnership Forecasted Data

16. Year Actual Forecast Error |Error| 1 1402.0 2 1458.0 1402.0 56.0 56.0 3 1553.0 1441.2 111.8 111.8 4 1613.0 1519.5 93.5 93.5 5 1676.0 1584.9 91.1 91.1 6 1755.0 1648.7 106.3 106.3 7 1807.0 1723.1 83.9 83.9 8 1824.0 1781.8 42.2 42.2 9 1826.0 1811.3 14.7 14.7 10 1780.0 1821.6 -41.6 41.6 11 1759.0 1792.5 -33.5 33.5 674.5 Mean Absolute Errorfor the Nonfarm Partnership Forecasted Data E

17. Year Actual Forecast Error Error2 1 1402 2 1458 1402.0 56.0 3136.0 3 1553 1441.2 111.8 12499.2 4 1613 1519.5 93.5 8749.7 5 1676 1584.9 91.1 8292.3 6 1755 1648.7 106.3 11303.6 7 1807 1723.1 83.9 7038.5 8 1824 1781.8 42.2 1778.2 9 1826 1811.3 14.7 214.6 10 1780 1821.6 -41.6 1731.0 11 1759 1792.5 -33.5 1121.0 55864.2 Mean Square Error for the Nonfarm Partnership Forecasted Data

18. Year Actual Forecast Error Error % 1 1402 2 1458 1402.0 56.0 3.8% 3 1553 1441.2 111.8 7.2% 4 1613 1519.5 93.5 5.8% 5 1676 1584.9 91.1 5.4% 6 1755 1648.7 106.3 6.1% 7 1807 1723.1 83.9 4.6% 8 1824 1781.8 42.2 2.3% 9 1826 1811.3 14.7 0.8% 10 1780 1821.6 -41.6 -2.3% 11 1759 1792.5 -33.5 -1.9% 31.8% Mean Percentage Error for the Nonfarm Partnership Forecasted Data

19. Year Actual Forecast Error |Error %| 1 1402 2 1458 1402.0 56.0 3.8% 3 1553 1441.2 111.8 7.2% 4 1613 1519.5 93.5 5.8% 5 1676 1584.9 91.1 5.4% 6 1755 1648.7 106.3 6.1% 7 1807 1723.1 83.9 4.6% 8 1824 1781.8 42.2 2.3% 9 1826 1811.3 14.7 0.8% 10 1780 1821.6 -41.6 2.3% 11 1759 1792.5 -33.5 1.9% 40.3% Mean Absolute Percentage Error for the Nonfarm Partnership Forecasted Data

20. Use of Error Measures To identify the best forecasting method • Use error measure to identify the best value for the parameters of a specific method. • Use error measure to identify the best method. • Use MSE and MAE for both of these situations. Note that MSE tends to emphasize large errors.

21. Use of Error Measures, continued Forecast bias is the tendency of a forecasting method to over or under predict. The mean error, ME, measures the forecast bias.