Round table Long Series: Connection and Methodological Changes

1. Round table Long Series: Connection and Methodological Changes Craig McLaren (Office for National Statistics, UK) craig.mclaren@ons.gov.uk Rio de Janerio, 21-25th August 2006

2. Introduction • Thank you for inviting me here today • Dealing with methodological change from a time series perspective is a challenging issue • Users should have continuity of time series • Common issue across National Statistics Institutes • Talk about previous experiences and issues

3. Issues covered in slides • Smoothing in methodological change • Description of a generalised backcasting tool under development at ABS • Multivariate approach for short time spans • Lessons learnt from previous exercises • Upcoming issues at ONS and ABS

4. Dealing with methodological change? • Want to remove the effect of methodological change • E.g. Change to estimation method, classification structures • Improves the quality of data by increasing consistency and coherence across time • Should not remove effect of real world change • Appropriate to still reflect this in the time series • Two approaches • Edit unit records: typically used for Population data • Alter final estimates: typically used for Economic data • Consider monthly and quarterly time series with altering the final estimates

5. Motivation: Example • Level-shift caused by methodological change • Want to raise level of early part of series

6. Motivation: Example of seasonally adjusted before and after backcasting

7. Appropriate backcastingKnowledge required • Whether or not to backcast • Estimation and significance of impact • Estimate a suitable backcasting length • How far back does the change in series exist? • Was the change constant or gradual? • Determine if change is definitional or not • Measure the difference between the old and new series • For example: conduct parallel sample, do parallel estimation • Quality control • Assessing the magnitude of revisions

8. Parallel estimation Different scenarios • Different scenarios • Multiple time point overlap • One time point overlap • Can even have no overlap: use forecasting • Recompile historical data under new definition • High quality but expensive • How far back to go? • New data points • Model historical data and evaluate impact by intervention analysis • Smoothing back based on the impact assessment

9. Parallel estimationMeasuring impact • Impact can have different components • Trend break: more parallel estimates, more accuracy • Seasonal break: need at least a year + one period of parallel estimates • Usually assume no seasonal break • Multiplicative relationship: impact = 100*average(Onew / Oold) - 100 • Additive relationship: impact = average(Onew - Oold)

10. Parallel estimationSignificance of impact • Need to assess quality of impact estimate • Is impact significantly different from zero? • Need relative standard error for (Onew - Oold) • Should assess if statistically significant difference • Options for different levels • Aggregation level at which impacts are significant determines backcast process • Quality assurance given for this level and upwards • Lower level series adopt impact of higher level series

11. Aggregation structures • Directly backcast series • Lowest-level directly seasonally adjusted series • Indirectly backcast series • All other series formed by aggregation 1-D 2-D

12. Backcast objective • Objective function for ABS backcasting • Maintain the historical seasonally adjusted movement • Minimise methodological change effect to ensure the continuity of a time series • Need to avoid misinterpretation of a measurement change as a real world change • Aim: bound revisions to movements and maintain stable seasonal factors • Alternative objective functions equally valid

13. Shape of backcast factors • Directly backcast series • Multiplicative: exponential shape: Obt = Ot * xt/N • Additive: linear shape: Obt = Ot + tx • Index series only: infinite shape: Obt = Ot * x • Indirectly backcast series • Follow aggregation structure • No particular shape

14. Shape of backcast factors(continued) shape comparison real data: exponential

15. Shape of backcast factorsFinite versus infinite length • Infinite length: multiply whole series by constant • For directly backcast series: no revisions • For indirectly backcast series: small revisions • Good for index series • Usually bound length for conceptual reasons • New definition inappropriate long ago: e.g. new technology • Nonsensical to increase an old estimate too much • Risk management

16. Quality measure • Assess absolute change to the period to period movement in the seasonally adjusted estimates due to backcasting • Delta = • In general • Either percentage or quantity change • Multiplicatively seasonally adjusted: percentage • Additively seasonally adjusted: quantity

17. Quality measureDelta seasonally adjusted %-movements and the absolute differences, pre- vs post- backcasting ==> quality measure

18. Quality measureDelta (continued) • Maximum percentage change of movements in seasonally adjusted estimates pre and post backcast over a series • Used as quality measure • delta maximum (user specified) • Effectively a bound on revisions in seasonally adjusted movements • Compare maximum delta to delta maximum • Can be normalised to allow comparisons between series

19. Quality measureLength of backcast • Governed by choice of how much tolerance in the change to movements pre and post backcast • Smaller tolerance (delta) => longer backcast (typically) • delta maximum selection • Choose this so the backcast doesn't adversely affect published values • For example, at most one significant figure in published movements • In practice one common length for entire group of time series

20. Backcasting process backcast shape and length N want delta < threshold

21. Generalised backcasting toolAustralian Bureau of Statistics • Consistent ABS approach to backcasting • Standard shapes to smooth in impacts • Consistent diagnostics • Consistent language • Client areas can perform backcasts without input from Time Series Analysis experts • Streamlined process • Directly updates stored original estimates • Currently in development

22. Generalised backcasting toolProcess flow collection of series new backcast originals (overwriting old originals) generalised backcasting facility impacts clearance report settings final length selection diagnostics

23. Example: Generalised backcasting tool

24. Example: Generalised backcasting tool

25. Example: Generalised backcasting toolSeasonally adjusted estimates

26. Example: Generalised backcasting toolChange in seasonally adjusted movements

27. Example: Generalised backcasting toolClearance report

28. Multivariate approachDealing with methodological change • Title: Estimation of seasonal factors for a short time span using multi-level modelling • Number of overlap periods is typically short • This solution was used to assist with transition between two surveys and the results were used within ABS National Accounts • Joint work with Xichuan (Mark) Zhang, ABS

29. Multivariate approachAssumptions • New survey measures the same underlying activity as the old survey • Trend movement is the same for different surveys but may be at a different trend level • Seasonal factors are assumed to be different for different surveys • Can use multilevel models if there is a hierarchical structure

30. Multivariate approachModel • Mixed model i : old and new ... 1 k j : industry ... j totals k : state

31. Multivariate approachFinal model • Assume a log additive model for the time series decomposition

32. Multivariate approachOutline of steps • 1. Run full model • 2. Remove "trend" of each series • 3. Estimate seasonal factors • 4. Test if old and new surveys have same seasonal factors • 5. Convert from model into X-11 framework

33. Multivariate approachExample: real world application • Two surveys • 15 industries • 8 states and Australia • Data available: Four parallel quarter estimates over 2001

34. Example: New and old original data for 2 different industries

35. Example: Selected state results* seasonal factors are not significantly different between new and old survey

36. Multivariate approachComments • A mixed model (random and fixed effects) with multi-level modelling allowed realistic seasonal factors to be estimated for a short time series • Provided a framework • More cases like this will occur in practice • Further work by Carole Birrell and David Steel at University of Wollongong

37. General pointsPrevious lessons • Quality assurance of the new estimation methodology • Prepare users for revisions in time series estimates • One overlap time point is simply not enough to make a good impact assessment because of rotation effects • May required re-backcast once additional information available • Variance of impact assessments were not available

38. General pointsPrevious lessons (continued) • Managing statistical risk • Classification and estimation methodology changing at same time versus consecutive change approach • Different measurement methods: parallel estimates and parallel run • Rehearsal of backcast environment in relation to regular production schedule • Ensure additivity of estimates for shallow aggregation structure • Decision on new estimates should be published e.g. if the new estimate is good enough for publication or only for impact measurement purpose

39. Upcoming examplesOffice for National Statistics • Industry classification changes • Approximately once every 12 years • Strong link to European needs • Long agreement process • Currently in planning stage for upcoming changes • Broad time frame • 2007 Adding new industry codes to business register • 2008 Annual surveys selected on new codes • 2009 Short-term surveys selected on new codes • 2011 National Accounts first moves to new codes

40. Upcoming examplesOffice for National Statistics (continued) • Need to be able to re-construct results from old (2003) and new (2007) classifications • Options available: • Domain estimation • Conversion matrices • Parallel running for a limited period? • Problems of compiling and publishing results on two bases simultaneously • Constrain results

41. Upcoming examplesAustralian Bureau of Statistics • Range of issues • New industry classification • Generalised regression sample and estimation including using ABS Survey Facilities estimation approach • Building Activity Survey turnover stratification changes • … • Developing tools to assist • Generalised backcasting tool

42. Upcoming examplesAustralian Bureau of Statistics (continued) • Currently: creation of new frames and specification of backcasting facility • 2006 to 2009: Transition phase with dual frames involving parallel estimation based on top-up samples • September 2009 onwards: Implementation where sample design and estimation is based on new classification

43. Upcoming examplesAustralian Bureau of Statistics (continued) • Measuring the impact • For quarterly subannual surveys: 5 overlap points • For monthly subannual surveys: 13 overlap points • Methodology Division will provide advice on the significance of the impact, i.e. is there any real impact? • If no seasonal change then minimum of 3 overlap points • To calculate the trend factor for backcasting all new and old estimates are needed • (TrendNew / TrendOld)=(1/5)Sum(OriginalNew/OriginalOld)t

44. Some further information • ONS MD contact: gareth.james@ons.gov.uk • ABS MD contact: mark.zhang@abs.gov.au • McLaren and Zhang (2003) Estimation of seasonal factors for a short time span using multi-level modelling with mixed effects, Working Paper No. 2003/1, www.abs.gov.au