Revisions and the importance of metadata

1. Revisions and the importance of metadata OECD Short-Term Economic Statistics Working Party Meeting Paris, 25-27 June 2007 Andreas Lorenz* Deutsche Bundesbank Statistics Department *This presentation reflects the personal views of the author and not necessarily those of the Deutsche Bundesbank or its staff.

2. Outline • Introduction • Revisions analysis without metadata • Revisions analysis with metadata • Conclusion Revisions and the importance of metadata

3. 1 Introduction • Uses of revisions analysis • From the producer perspective: Instrument for quality monitoring • From the user perspective: Tool for building expectations about future revisions of preliminary data Revisions and the importance of metadata

4. Case study: German Index of Industrial Production (IIP) • Real time data from 1999 to 2006 • What does revisions analysis tell about the extent of revisions at different time periods? • Can this information be used to build expectations about future revisions of preliminary estimates? Revisions and the importance of metadata

5. 2 Revisions analysis without metadata a) Classical revision measures • Mean revision Revisions and the importance of metadata

6. Revisions and the importance of metadata

7. 2 Revisions analysis without metadata b) User perspective • How long has the user to wait for a stable month-on-month growth rate? • Depends on what is to be considered as “stable” • One possible operationalisation: “How long does the user have to wait until the month-on-month growth rate does not deviate by more than 0.5 percentage points from the final value?” Revisions and the importance of metadata

8. Number of months, after which AR = absolute revision; i = reporting month; j = final figure (t+24 months); x = mom growth rate Revisions and the importance of metadata

9. Results • The user has to wait on average about 6 ½ months for the month-on-month growth rate not to be revised any further by more than 0.5 percentage points from the final mom rate Revisions and the importance of metadata

10. 3 Revisions analysis with metadata • Metadata for the IIP for the overall period 1999 – 2006 • Only the largest firms report monthly • Smaller firms report quarterly • Reporting units for the monthly survey are selected once per year ( dying out sample within a year) • Monthly output figures are benchmarked with results from the full quarterly sample (smaller and larger firms) Revisions and the importance of metadata

11. Metadata regarding the timing of IIP releases • Preliminary release (T + 37 days; ~ 10% missing data) • First revision (T + 57 to 62 days; incorporation of late reports) • Quarterly revision (~2 ½ months after end of reporting quarter, sometimes later) • Annual revision (together with quarterly revision of Q4) • New base year and benchmark revisions (about every 5 years) Revisions and the importance of metadata

12. Revisions and the importance of metadata

13. Revisions and the importance of metadata

14. Revisions and the importance of metadata

15. Revisions and the importance of metadata

16. Metadata specific to single years and sub-periods • 1999: High quarterly revisions; later on: the expected quarterly revisions are included in the estimation of the first monthly estimates. This yields in general to smaller quarterly corrections. • 2002: Missing update of the sample for the monthly survey • 2005: New method for the imputation of missing values for the preliminary release (see next slide) Revisions and the importance of metadata

17. Estimation of missing values for the preliminary release • About 10% of the firms do not report in time for their figures to be included in the preliminary release of the monthly IIP • 1999-2004: The output of the previous month is used as an estimate for the missing value of the current month (no-change assumption). Calendar and seasonal effects are not taken into account. • Since 2005: The output of the firms of the corresponding NACE- Division which report in time is used as an estimate for missing values, which yields to smaller revisions. Revisions and the importance of metadata

18. Metadata for 2007 onwards • More firms report monthly • Reporting units for the monthly survey are updated monthly ( no dying out sample within a year) Revisions and the importance of metadata

19. Results • The first revision to the preliminary release does not follow a systematic pattern any more • Since the sample of the monthly survey is now updated concurrently, there is no systematic quarterly correction to be expected • The yearly revision is not predictable Revisions and the importance of metadata

20. Results • What can be learned in light of available metadata for building expectations about future revisions? • Nothing - the producer of official statistics has already “learned” and improved methodology • Revisions should not be systematic any more Revisions and the importance of metadata

21. 4 Conclusions • From the producer perspective: Revisions analysis have motivated the improvement of methodology which, in turn, lead to lower revisions • From the user perspective: Revisions analysis helpful for building expectations on future revisions (eg how long does a user have to wait for getting a stable mom growth rate) Revisions and the importance of metadata

22. 4 Conclusions • Caveat: Without knowledge of the metadata the use of results of revisions analysis for building expectations can be highly misleading • Metadata (sample methodology, estimation methods for missing values – which later may become available, timing of revisions) are of vital importance for building expectations about future revisions Revisions and the importance of metadata