Richard McKenzie OECD

1. Relative size and predictability of revisions to GDP, Industrial Production and Retail Trade – a comparative analysis across OECD Member countries Richard McKenzie OECD Workshop on Macroeconomic Forecasting, Analysis and Policy with Data Revision

2. The OECD Real-Time Data and Revisions Analysis Database • Full time series of data published every month starting from the February 1999 edition of the Main Economic Indicators for 21 key economic variables • Access OECD revisions analysis studies for GDP, Index of Industrial Production and Retail Trade Volume • Automated programs and detailed user guide allowing users to perform their own revisions analysis for any country / variable combination available in the database • Enables economists to test the performance of their econometric models in simulated real-time (out-of-sample testing using original first release data) • Other variables often used in econometric models that are not revised (e.g. financial variables, exchange rates) are available in a parallel interface

3. The OECD Real-Time Data and Revisions Analysis Database • Project originated from needs of Euro Area Business Cycle network • Survey of potential users from Central Banks on variables to include • Built the database by loading old CD ROMS into an SQL database

4. http://stats.oecd.org/mei/default.asp?rev=1

10. Workshop on Real Time Data Analysis

12. Promotion of the interface • Two OECD Statistics newsletter articlesand Statistics Brief • OECD Statistics working paper • First paper in a series, followed by additional papers for Real Time Data Workshop in Zurich, submitted to the journal of Business Cycle Measurement and Analysis, and now this workshop • Emails to working groups of statisticians, academics, central banks and economists through various networks

13. Promotion of the interface • Letter to heads of NSOs encouraging them to use the facility to perform their own revisions analysis • Related to promotion through the OECD Short-Term Economic Statistics Working Party • Integrated fully with statistics portal on OECD website • Download statistics show it is one of the most accessed databases in the OECD (between 6000 – 9000 views per month) • Updating is part of the monthly MEI process, automated procedure run by the IT area

14. Future developments • Include output gap from OECD Economic Outlook • Ensure ongoing periodic review • Possibly look to expand variable list or integrate with other sources • Hopefully IT performance and stability will continue to improve (enabling us to load metadata for each vintage)

15. What about the quality? • Comparison with Philadelphia Federal Reserve real time database for United States • Quick evaluation of GDP constant prices • OECD vintages start from February 1999 whereas the Fed start in 1965 • OECD vintage time series back to 1960, Fed to 1947 • OECD extracts at the beginning of the month, every month, Fed is middle of the quarter • OECD in whole numbers, Fed in $Billion • 3 vintages chosen randomly (Nov 01, May 03, Aug 06)

16. Analysis of revisions for short-term economic statistics • Quick overview of terminology • Purpose of revisions analysis • From both a user and producer of statistics perspective • Results from detailed analysis of GDP, Industrial production and Retail trade

17. Terminology Earlier estimate – often first published data Later estimate, many time intervals can be considered Revision, observable n times Mean absolute revision: Relative mean absolute revision:

18. Terminology Mean revision: =

19. Terminology / references • All revisions analysis is done for growth rates: • Month-on-previous-month (MoM): (Mt/Mt-1) -1or quarter-on-previous quarter (QoQ): (Qt/Q t-1) - 1 • Year-on-year (YoY): (Mt/Mt-12) -1 or (Qt/Qt-4) -1 • OECD approach is built on initial work done by Di Fonzo (2005) and UK Office for National Statistics. • Other key references for revisions analysis include Mankiw and Shapiro (1986) (new vs noise) , Rao et. al. (1989)

20. Purpose of revision analysis • Users (policy makers, analysts, forecasters etc.) • Robustness of first published data • Evidence of bias • Expected size of revisions over different time intervals (is this changing over time?) • Producers (Statistics offices) • Indicator of quality and reliability • Diagnostic tool to improve compilation processes

21. OECD Revision Analyses • GDP (constant prices) • Monthly vintages from May 1995 to June 2007 • 18 OECD countries • Index of Industrial Production • Monthly vintages from Feb 1999 to Feb 2006 • 25 OECD countries, Brazil, India, South Africa • Retail Trade Volume • Monthly vintages from Feb 1999 to April 2006 • 24 OECD countries and South Africa

22. Mean absolute revision to first published QoQ growth rates for GDP

23. RMAR to first published QoQ growth rates for GDP

24. Robustness of first published growth rates • First published estimate of GDP QoQ growth rate: • most comprehensive indicator of the current performance of a countries’ economy • First published estimate of IIP MoM growth rate • early indicator of the current state of the business cycle, expansion or contraction in production activity • First published estimate of RTV MoM growth rate • early indicator of current consumer demand

25. Robustness of first published growth rates • Relative mean absolute revision (RMAR) from revision analysis can help us assess robustness • RMAR to first published MoM or QoQ growth rate assessed on revisions after 1 year • Expected proportion of the first published growth rate that will be revised within one year • If greater than say 0.5, should policy makers / analysts really base decisions on these first published growth rates?

26. RMAR to first published data after one year: QoQ vs YoY growth rates for GDP

27. RMAR to first published dataafter one year: MoM vs YoY growth rates for IIP

28. RMAR to first published dataafter one year: MoM vs YoY growth rates for IIP (cont ….)

29. RMAR to first published dataafter one year: MoM vs YoY growth rates for RTV

30. Conclusion on robustness • Relative mean absolute revision (RMAR) from revision analysis can help users assess robustness of first published growth rates: • If MoM / QoQ are not robust, YoY may be more suitable for short-term analysis (but this can delay the identification of turning points …..) • Or may need to look at other estimators (trend estimates? – but need to test these for revisions too) • Put pressure on statistics office to improve their methods

31. Predictability of revisions and assessment of bias for mean revision • Ideally revisions should centre around zero over time (i.e. equally likely to be + or - ). • Easy to assess the statistical significance of the mean revision at different revision intervals • If a bias is found, what does this mean? • Could users / analysts exploit this information to improve on first published estimates or their forecasting models? • Does a ‘true’ or ‘final’ value of the economic variable we are using to assess an aspect of the economy exist?

32. Reasons for revisions and their timing (GDP focus) • Revisions in first few subsequent releases • Revisions to input source data (e.g. arising from sample surveys with late respondents, corrections of previous errors found etc.) • Revisions to models based on partial indicators or replacements of estimates based on models with actual data • Concurrent seasonal adjustment

33. Reasons for revisions and their timing (GDP focus) • Periodic revisions performed annually • Revision of seasonal models • Benchmarking to annual data sources (may also involve reconciliation with aggregate level supply and use tables) • Annual chain linking, rolling updates to base period • Periodic revisions at other frequencies • Benchmarking to 5 or 10 years census (may also involve reconciliation with detailed Input Output tables • Change to base year for constant price estimates

34. Reasons for revisions and their timing (GDP focus) • Major adhoc revisions: • Changes to compilation methodology (e.g. annual chain-linking) • Changes to conceptual definitions (e.g. SNA 93) • Changes to classifications (e.g. NAICS, NACE rev. 2)

35. When is an observed bias to the mean revision important? • Policy makers / analysts should only be interested in small number of subsequent revisions to first published data • Thus forecasters should only consider whether first estimates are efficient or biased based on a small number of subsequent revisions (probably not more than one year after first published data) • Consider statistical significance of mean revision after one year to first published GDP QoQ growth rates

37. Longer term revisions to GDP growth rates • Propose the conjecture that revision to GDP QoQ growth rates are more likely to be upwards the longer the period from first published data • Due to the systematic influence of changes in compilation methodology providing better estimates of volume and productivity backcasted through the series (e.g. ICT deflators and PPIs for service industries) • Possible tendency for conceptual and definitional changes to have a similar impact (e.g. capitalisation of software in SNA 93)

41. Other results • Mean of revisions after one year to first published MoM growth rates were statistically significant at 95% level for: • Greece, Belgium and India for Index of Industrial Production • Canada for Retail Trade Volume • For each of GDP (4 countries), IIP (8 countries) and RTV (4 countries) much higher incidence of bias to first estimates of YoY growth rates

42. Main conclusions • Revisions analysis provides essential information to both users and producers • Users to understand degree of robustness for first published data (RMAR) and any short-term bias • Producers to understand better the quality and as a trigger to improve processes • Assessment of bias in revisions must be treated with caution (especially for GDP) • Revisions to GDP growth may have a legitimate tendency to be positive in the longer term • OECD task-force on revisions policy and analysis

43. THE END