Methodological challenges in integrating data collections in business statistics

1. Methodological challenges in integrating data collections in business statistics Paul Smith Office for National Statistics

2. Outline • Data quality for different sources • quality measures for survey and administrative inputs • quality measures for outputs • Combinations of sources • familiar and more advanced situations • Mode effects • Models • Discussion

3. Statistical data collections - quality • Relevance • generally questions conform to desired concepts • may be tailoring for • practicality • consistency across collections even if concepts differ • Accuracy • affected by sampling • impacts from non-response, measurement error • Timeliness • generally relatively timely

4. Administrative data - quality • Relevance • questions conform to administrative (not statistical) concepts • few concessions to statistical needs • Accuracy • unaffected by sampling • processes to discourage non-response • treatment of measurement error differs by variable • Timeliness • generally slow

5. Differences between types of source • Sampling accuracy is measurable for surveys, not relevant for administrative data sources • confidence in quality reduced for admin data • balance of accuracy measures different • Building statistical requirements into administrative series • requires negotiation and agreement • VAT classification information in the UK • INSEE has statistical and accounting information well integrated

6. Questionnaire design • Questionnaire design principles mostly used in designing statistical collections • Administrative data seen as “forms” not “questionnaires” • less attention to question phrasing to obtain required answer • more on statutory requirements

7. Output data quality • Data quality from combined outputs can be challenging to measure • function of the qualities of the input sources, and the methods used to combine them • some well-known general approaches • development of measures needed for particular cases (eg from models)

8. Combinations of sources - 1 • Frame and sample information • Sampling frames typically derived from administrative sources • Multiple uses of frame information • sample design • sample selection • validation and editing • estimation and variance estimation • Quality easily derived – standard situation

9. Combinations of sources - 2 • Dual-frame surveys • More than one administrative source • Pension funds survey in the UK • Units • Business register • Challenges of population inflation if matching not perfect • Estimate probability that unit appears in sample from either source • use in appropriate weighting procedure • adjustment for P(in both surveys) depends on survey type

10. Combinations of sources - 3 • Multiple surveys • different periodicity • summary information monthly, detail annually • for example capital expenditure – quarterly breakdown, annual summary • Benchmarking • where short-period surveys small (and variable) and annual larger (and less variable) • Quality measures • account for sampling error in both sources • account for non-response and measurement errors in larger survey

11. Combinations of sources - 4 • Auxiliary information • If administrative concept not close to statistical concept, data may still be useful • Auxiliary information in estimation • not required to be correct, only correlated with outcome • the better the correlation, the better the accuracy • Auxiliary information in validation • use tax data to improve validation follow-up activity • Data confrontation • Use multiple sources to identify discrepancies • Balancing

12. Mode effects • Mode effects manifest in several ways • differences in contact rate • differences in response rate given contact • differences in question replies given response • Test differences through a designed experiment (van den Brakel & Renssen 1998, 2005) • evaluates whole-process differences (not individual steps) • non-response adjustment if good predictors for response amongst auxiliary data (var increases) • model-based adjustments for other changes

13. Temporal differences • Administrative data often have longer reference period than statistical requirement • Implies temporal disaggregation (model-based) – Dagum & Cholette 2006 • Quality implications • estimated data as inputs • sensitivity of model to interesting changes

14. Models for combining data • Full flexibility in combining data available through modelling approach • Models at boundary between statistical producer and user • Ideally statistical results insensitive to model assumptions • small area estimates • useful for social surveys • challenges for business surveys not yet resolved • modelling for unit structures - BRES

15. Discussion • Aim: more from existing sources • often imperfect matches • modelling only appropriate approach • subjective • robust to assumptions • sensitivity analysis • Mixed mode collections • usability and low cost • data combination • quality components harder to measure

16. for more details see the paper, or contact • paul.smith@ons.gov.uk