by Saša Žiković 16 th DEC, Dubrovnik, Croatia, 2010

Comments on “Incorporating Uncertainties into Economic Forecasts: anApplication to Forecasting Economic Activity in Croatia”Dario Rukelj and Barbara Ulloa by Saša Žiković 16th DEC, Dubrovnik, Croatia, 2010

Motivation • Although forecasts are by their nature probabilistic most economic predictions quote just a single value without giving attached probabilities. • As many countries started their policies of inflation targets, the incorporation of the uncertainty in economic variable predictions served to show that there is uncertainty about shocks to affect the economy. • The approach proposed by Garrat et al. (2003) regarding uncertainty forecasting, is applied in forecasting economic activity in Croatia. The incorporation of uncertainty about the unobserved future shocks on forecasts and robustness of forecasts to the choice of the parameters is done for Rukelj (2010) model. • Rukelj (2010) model is used assuming that it balances economic theory and consistency with data, and so has good forecasting capabilities.

Good points of the paper • The authors present density forecasts of economic activity in Croatia using stochastic simulations of the random errors with parametric and non-parametric approaches, as well as the evaluation of density forecasts, • Ecoometric are nicely and correctly done • Parametric and non-parametric methods are used in generation of shocks and the probability bands are constructed from the obtained set of simulated values. • I liked the use of bootstrap for non parametric errors I hope that the bootstrapping was applied to IID errors – since I don’t see any mention of this in the paper) • Introduces skewness in future distributions in order torepresent potential risks in Croatian economy • The study is legitimate as an applied study if the procedure is relevant to a Croatian policy question the authors have in mind.

Comments and suggestions • A bit difficult to read because of numerous errors - needs to checked by a professional... • The literature review is unsatisfactory in terms of the econometric methods used, the theoretical basis of the study and policy discussions. • The small macro-model (Rukelj 2010) is not properly introduced, evaluated and discussed. All we have is the author’s word that his own model performs well. • The methodology section is a not-so-competent summary of the appendix to Garett et al (2003). Equations 1-5 are essentially equations 27-31 in Gatett et. al. (2003), the only difference being the letters used in notation. Moreover the transcription is incomplete and wrong at some points. For example on page 5 the autors talk about ϕibut this set of coefficients do not exist in their notation, ϕiexist in Garett et. al.(2003). • In section 2.2 the number of replications are denoted of period T+h forecasts is denoed by both x and r interchangably. • In most places summation notation is not properly specified, no definitions and range for summation subscripts like x, h, t and s.

Rukelj (2010) SVAR in Croatia paper • Rukelj, D. 2010, "Modelling Fiscal and Monetary Policy Interactions in Croatia using Structural VECM" - very simple theoretical framework with 3 equations and 3 variables (government expenditures, M1 and velocity of money) and one common stochastic trend which drives the whole system. • There areNO characteristic Croatian features so I don’t understand why this would be a preferred model? • “...we chose the model proposed by Rukelj (2010) as it accurately accounts for Croatian’s macroeconomic features, and it performs well when forecasting” – Rukelj 2010 paper can only serve to establish benchmark parameters - the model is a basic SVAR. I don’t see any Croatian macroeconomic features and any sign of forecasting performance – there is absolutely no backtesting or comparison with any other model???

Comments and suggestions • Following eq. 3 on page 5 the predictive probability distribution function is not presented and similarly no predictive probability distribution function is presented for the case of parameter uncertainty after eq. 5. • The bullet points explaining how probability bands are defined look like they are adopted from somewhere because the notation does not follow the preceding equations (especially w.r.t. x and r superscripts). • Following equation 6, is not properly explained. No information is given on whether simulated errors differ with or without parameter uncertainty. The authors were better off just citing Gratett et al. (2003) inviting the curious reader to look at the original exposition of the technique. • The notation mistakes and lack of explanations of the procedure being followed make this section very problematic. There is no methodological contribution here. This is just an application of the Garett et al. methodology to Croatian data. • Why is there no comparison with other approches? You can use Clements and Smith (2000) methodology for competing models.

Comments and suggestions • Introduction of skewness produce very poor results – not U distr and increases the chances of observing a higher index in the time of crisis(???) – it is as useful as a fifth wheel! • Asymmetric shocks idea used by Bank of England in Inflation Report since 1996, but is there any asymetry? Have you tested for it - here the moments values are taken ad hoc and give strong positive skew – this is obviously the authors’ subjective view of the Croatian economy although I don’t understand why? • Why would the balance of risks be constant over time? – look at the current crisis • There have been a lot of advances since 1996 look at GP and skewed Student’s t distribution • What about structural breaks?

Comments and suggestions • Density forecasts suggest the economic activity index would have increased the first trimester of 2009, regardless the method for errors simulation. – SVAR needs to be improved only 3 variables (government expenditures, M1 and velocity of money) – overly simplistic – what about structural breaks, price and wage rigidities, exchange and interest rates, consumption, exports, imports, revenues, small open economy - influence of the state of the economy of the major trading partners, would have been particularly useful in predicting the effects of the current crisis on Croatia • Your bad forecasting results are not surprising if you notice that the calibration of your model was done on a time period which showed positive economic activity, without considering alternative states of the economy and/or changes in the global surrounding the results will always be equal to driving while looking at the rear view mirror.

Comments and suggestions • There is no dynamics in the generation of errors, no volatility modelling, the authors assume IID, how come? Authors are sure that there is no ARMA or GARCH terms in empirical errors? – look at your KS results. Density forecasts make most sense when combined with ARMA-GARCH framework • You only show the results - there is absolutely no discussion of the results - what are the implications of this results, what is the reason behind obtained results - this would be interesting to researchers and forecasters • The results are not very compelling because you can’t reject the H of outturns being U-dist in only 7/12 (only future uncertainty – parametric errors) and 5/12 (only future uncertainty – non-parametric errors) – and these are your best approaches! • What is the message of the paper – it is not worth including parameter uncertainty in forecasts? Why is this so? Make an argument! Provoke debate, attack the whole concept – BUT with strong arguments! What do the other authors say?

Comments and suggestions • limitations of KS test - tends to be more sensitive near the centre of the distribution than at the tails; and the distribution must be fully specified, i.e., if location, scale, and shape parameters are estimated from the data, the critical region of the K-S test is no longer valid. It typically must be determined by simulation. • ➨Lilliefors test -uses sample estimates of parameter values instead of parameter values that are assumed to be known • ➨ Instead of directly testing if the errors are IID U(0,1) you can transform observations to make them normal under the null hypothesis. This can be done by applying an inverse normal transformation to the uniform series. If xt is IID U(0,1), then zt = Ф-1(xt) is IID N(0,1). When the data is transformed to follow normal distribution, a wider array of powerful statistical tools can be applied, than under uniform distribution. You can test the null hypothesis that zt is IID N(0,1) against a fairly general first-order autoregressive process with a possibly different mean and variance.

Comments and suggestions • It is not clear why exactly you are introducing parameter and future uncertainty into the Rukelj 2010 model and why we should care whether the inclusion of only one type of uncertainty or both types is more appropriate. • Is the model a benchmark model for the Croatian economy? Is it used in policy design by the policy maker? • Is there an events scenario to be simulated using the models developed here? • Without the answers to these questions the finding that the model with future uncertainty marginally passes the KS test does not mean much. • It does not tie into any policy discussion or theoretical discussion. • Thus far you only showed that this approach is not good at forecasting economic activity in Croatia??? • What are the impacts of uncertainty on government policies?

Comments and suggestions • 4 pages of unexplained graphs in a16 page article on top of 3-4 pages which are practically transcribed from another article is not good use of space. This leaves very little room for original contributions. Moreover I am not following why the table on page 10 has 12 columns. I believe a summary measure could have been developed. Needless to say, that table needs a a title and a number. • In the references Billix and Sellin 1998 does not have proper citation info. • Figures would have been more informative if realized values were reported alongside the forecasts so that model performances could be visually compared.

Comments and suggestions • For your future research look at... 1) Measuring forecast uncertainty: A review with evaluation based on a macro model of the French economy (C. Bianchi, G. Calzolari) • Five alternative techniques have been applied to measure the degree of uncertainty associated with the forecasts produced by a macro-model of the French economy, the Mini-DMS developed at INSEE. They are bootstrap, analytic simulation on coefficients, Monte Carlo on coefficients, parametric stochastic simulation and re-estimation, a residual-based procedure. Due to the complexity and the size of the model (nonlinear and with more than 200 equations), several associated technical problems had to be solved. The remarkable convergence of results which has been obtained for all the main endogenous variables suggests that forecast confidence intervals are likely to be quite reliable for this model. 2) Norwegian RIMINI model (30 stochastic equations, and about100 exogenous variables)

To conclude • Correct econometric exercise. • If the goal is to formulate forecasts and policy responses this is a first step on a loooong journey! Comment about these comments... • “It is always easier to criticize than to write”!

A word of warning! • “Theoretical models are necessary tools in our attempts to understand and “explain” events in real life but whatever “explanations” we prefer, it is not to be forgotten that they are all our own artificial inventions in a search for an understanding of real life; they are not hidden truth to be discovered.” (Haavelmo, 1944) • “The disposition of present-day statistical theorists tosuppose that all ‘error’ distributions are exactly normal can be ascribed to their ontologicalperception that normality is too good not to be true” (Anscombe, 1967)

by Saša Žiković 16 th DEC, Dubrovnik, Croatia, 2010