Surveys, administrative data or integrated models: A decision by quality indicators?

1. Jörg Enderer, Dieter Schäfer European Conference on Quality in Official Statistics Q2010, Helsinki, 4-6 May 2010 Surveys, administrative data or integrated models: A decision by quality indicators?

2. other administrative data data from statistical surveys Turnover tax data from fiscal authorities Employment data from Federal Employment Agency annually data:business register monthly data:short term statistics The project: Use of administrative data in short-term statistics

3. Reshaping STS - Results 2004 2011 Full replacement of surveys: Craft statistics (German particularity) Integrated model of smaller survey and administrative data: Service sector, wholesale trade, maintenance of motorvehicles S U R V E Y S Surveys unchanged, additional use for non-covered enterprises: Building installation Surveys unchanged: Manufacturing, site preparation and civil engineering, retail trade, hotels and restaurants

4. Favorable testing conditions • Availability of aggregated results and micro data for surveys and administrative data in terms of • periods • economic activities • regions • Extensive possibilities to compare the different methods for the same field • survey • administrative data • integrated models of survey and administrative data

5. Which quality criteria were relevant for the choice of the method? Could the criteria be described by quality indicators? By indicators included in ESQR? Did the indicators lead to a clear decision for a certain method? Questions (ex post)

6. Relevant quality criteria Important criteria for the tests Criteria independent from choice of method • Relevance • Accuracy • Timeliness • Punctuality • Comparability • Accessibility • Clarity • (Coherence)

7. Quality criterion: Relevance • Target: User needs have to be fulfilled • Surveys: Do we get what we want? Check of microdata => Accuracy issue • Administrative data: • Differences in definitions, statistical units, allocation to economic activities • Quantitative indicators: Differences in concepts in % • No uniform results for different activities and regions • Elimination of important differences by (complex) estimation models => Problems of accuracy? • Role of indicators: Information on big deviations of administrative data; no quantitative indicators for smaller deviations in definitions

8. Quality criteria: Timeliness and punctuality • Target: Comply with existing standards of timeliness • Indicators: Time lag in days; additional indicators for subprocesses • How does the fulfillment of the target affect the accuracy? • Administrative data: Suitable and automatic IT procedures had to be developed Higher time restrictions for editing and estimation methods • Administrative Data: In some cases target not achievable with reasonable accuracy, e.g. t+30 for manufacturing, retail trade • Role of Indicators: Minimum Standards

9. Quality criterion: Accuracy • Targets: Assure accuracy of the status quo • Different dimensions of accuracy: • Internal comparision: Revisions for each method • Comparison of results of different methods • Accuracy of estimations for administrative data • ESQR - Indicators: Average size of revisions, response rates, edit failure rates, coefficient of variation, coverage rates • Additional indicators for administrative data like misclassification rates, linkage rates, risk indicators for differences in statistical units (tax groups)

10. Accuracy: Limits of indicators • Size of revisions • Surveys: Correction of missing data and outliers • Administrative data: Correction of missing data, outliers and update of data Higher size of revisions does not necessarily mean poorer quality • Does higher edit failure rate mean less accuracy? • Different indicators are difficult to weight against each other: Coefficient of variation against misclassification? • Role of indicators: Vital but not without sound interpretation

11. Decision-making by indicators? • Minimum standards for single quality criteria are important • Minimum standards for relevance and timeliness can be reached at the expense of accuracy • Accuracy is very complex and difficult to judge for one method (esp. non sampling errors) • Comparison of accuracy for different methods even more difficult • ESQR – Indicators are very important, but administrative data need partly other (additional) indicators • Quantitative indicators need background information for sound interpretation • Composite/overall quality indicators are no way out

12. Decision-making process in the project The decision was a social process managed by written summaries, discussions between producers and communication with main users Standards on relevance, timeliness and punctuality must be met Minimum standards for accuracy must be ensured, e.g. acceptable size of revisions estimation rates coefficients of variation In addition pros and cons have to be balanced.

13. Many thanks for your attention Any questions? Dieter Schäfer and Jörg Enderer E-Mail: datenqualitaet@destatis.de