Richard McKenzie, Zakia Adam & Michela Gamba OECD

1. Impact and timing of revisions for seasonally adjusted series relative to those for the corresponding raw series OECD / Eurostat taskforce on performing revisions analysis for sub-annual economic statistics Richard McKenzie, Zakia Adam & Michela Gamba OECD

2. Item VI of task-force Terms of Reference “….. be able to break down the results of a revisions analysis study to quantify the impact of different (homogenous groups of) sources of revision. As a minimum the guidelines must describe how the effects of seasonal adjustment on revisions can be separated and quantified in relation to other reasons.

3. Description of OECD study • Compare revisions analysis for raw and SA series for selected countries, IIP & Retail trade • Size of raw and SA revisions at different intervals • Occurrence of revisions to raw and SA series • Bias in revisions to raw series relative to SA series • Revisions of raw and SA series for year-on-year growth rates • Provide a tool which would allow countries to perform similar studies with their data

4. Countries chosen • Balance of EU and non-EU countries • Include some countries where seasonal adjustment is performed by OECD (forward factor) • Most importantly, need vintage series for all monthly snapshots with few missing updates • Last criteria presented the biggest problem due to lack of availability of vintages of raw series

5. Index of industrial production • United States, Japan, Korea, Belgium, Czech Republic, Finland, France, Netherlands, Poland • Finland, Belgium, Netherlands – raw data is in fact working day adjusted • Belgium, Czech Republic, Poland – SA done by OECD (forward factor method)

6. Retail Trade Volume • Canada, Japan, Korea, Mexico, Norway, Sweden • SA done by OECD for Mexico

7. Size of raw vs SA revisions for month-on-previous-month growth rates • Consider relative absolute mean revision (RMAR) • Review for different revision intervals (is the relationship stable across time?)

12. Size of raw vs SA revisions for month-on-previous-month growth rates • Revisions of raw consistently smaller than SA by similar magnitude for different revisions intervals • Ratio differs across countries, WDA is evident (larger revisions relative to true raw series) • Similar picture for Retail Trade, but revisions to raw series even smaller in proportion of those to SA • Overwhelming evidence that majority of revisions to SA series is from process of seasonal adjustment • ….. But month on month growth rates of raw series are of little use to users / analysts …..

13. Occurrence of revisions for raw and SA series • How many months after first published values do we expect data to be revised? • Raw series should only be affected by late data, fixing errors etc. (especially if direct from one survey with no models etc.) • Revisions to SA series will depend on revisions to raw series and process of seasonal adjustment • Revisions in longer term may also affect raw series (e.g. rebases, methodological changes, benchmarking etc.)

16. Occurrence of revisions for raw and SA series • IIP • occurrence of revisions similar for raw and SA for USA, Netherlands, Korea, Japan • But raw less often revised than SA for Belgium, Czech Republic, Finland, France and Poland • Retail trade • Occurrence of revisions similar for Korea, raw a bit less for Japan and raw much less for Canada, Mexico, Norway & Sweden

19. Occurrence of revisions: raw vs SA • Some evidence that revisions to raw series less likely the further from first published data than for SA • More evident for Retail than IIP and likelihood of revisions to raw for Retail trade is lower • Reflecting again that Retail trade data more likely to be compiled from single survey in most countries • Further work needed to understand the impact SA method has on occurrence and size of revisions • OECD tool now developed to break down revisions on a month by month basis

20. Are revisions statistically significant? • Ideally revisions should centre around zero over time (i.e. equally likely to be + or - with mean revision close to 0) • If this is not true then reasons causing this tendency in the compilation process should be found • Should not expect a bias to exist in the raw series but not in the SA and vice versa

21. Mean revision statistically significant for raw but not SA (& vice versa) • IIP • Some evidence found for Belgium and Korea but very much on the margin • Retail trade • Some evidence for Canada but mean revision still very small • For Korea, raw series mean revision significant (at 10% and 5% level) for all revisions intervals but not for SA • Mean revisions smaller but growing with length of interval (SA is in same direction and close to SIG)

22. Which short term measures should we advise users to focus on? • Are first estimates of month-on-previous-month growth rates of SA series for IIP and Retail Trade reliable enough to enable informed decision making? • Not in most countries! First estimates of MoM growth rates revised by 2/3 initial value on average after one year (95% countries revised by more than 2/5) • Shows the value that revisions analysis can provide to users • Previous study suggested that users should focus on year-on-year growth rates for short-term analysis as these are more robust with respect to revisions

23. Revisions to year-on-year growth rates • Which series should year-on-year growth rates be calculated on? • OECD Data presentation handbook recommends YoY growth rates to be calculated from raw or WDA series if available • What about robustness of raw / WDA and SA YoY growth rates with respect to revisions?

27. Revisions to year-on-year growth rates: Conclusion / Future work • Some weak evidence to suggest that YoY are more robust with respect to revisions for raw series compared to SA • Thus supports recommendations in the handbook • More empirical research would be interesting for users, particularly for WDA series

28. Conclusions / Future work • Relative mean absolute revisions (RMAR) for month-on-month growth rates are much lower for raw relative to SA series • Process of seasonal adjustment accounts for more than half of RMAR for the SA series, sometimes up to 90% and indeed still more than 50% in comparison to WDA series • …… but mean average revision is still generally higher for RAW series and month-on-month growth rates for raw series are not much use for short-term analysis

29. Conclusions / Future work • Likelihood of revisions to raw series (MoM growth rates) drops considerably after the first month but tends to stabilise at lower level in following months • Likelihood of short-term revisions to SA series should depend on method used but more work needed to assess the relative impacts • New OECD tool allows analysis of revisions to raw series on a month-on-month basis up to 6 months from first published value

30. Future work • Disseminate results on OECD website, prepare paper and discuss implications for taskforce recommendations • Use detailed revisions triangles to try and identify and quantify other reasons for revisions? • OECD also conducting a study to look at the impact of improvements in timeliness on revisions

31. THE END