Assessing Quality for Integration Based Data

1. Assessing Quality for Integration Based Data M. Denk, W. Grossmann Institute for Scientific Computing

2. Contents • Introduction • Data Generating Processes • Data Quality for Integration Based Production • Assessing Quality for Integration Based Data • Conclusions

3. Introduction – Aspects of Quality • Quality is discussed from two different points of view • The Processing View • What methods can be used in production of statistics ? • Specific statistical techniques for specific statistics • Development of models of best practice or standards • The Reporting View • How should Quality reports look like?

4. Introduction – Reporting View • Numerous formats for Quality Reports • SDSS, DQAF, Fed Stats, StaCan,…. • Logic of the proposals according to so called hyperdimensions • For example ESS: • Institutional Arrangements • Core Statistical Processes • Dimensions for Statistical Output • Inside the hyperdimensions so called quality dimensions • Relevance, Accuracy, Timeliness, Accessibility,……

5. Introduction – Reporting View • Not so much agreement about the dimensions • Possible Reason: Different methods / levels of Conceptualization • Concepts of mental entities • e.g. quality dimensions in DQAF • Concepts as meaning of general terms • e.g. quality elements in DQAF • Concepts as units of knowledge • e.g. quality indicators of DQAF • Concepts as abstracts of kinds, attributes or properties • measureable quantities like sampling error, …

6. Introduction – Reporting View • Stronger matching of the processing and the reporting view seems necessary • Starting point can be attributes and properties of statistical processes necessary for assessing quality • From basic quality concepts we build higher level elements by aggregation • Prerequisite for definition of necessary basic quality concepts: • Empirical analysis of different production processes • Final result is a User Oriented Quality Certificate

7. Data Generating Processes • We can distinguish two broad classes of data generating processes • The survey based data generating process • The integration based data generating process

8. Data Generating Processes – Survey based • Most considerations about reporting quality start from the traditional survey process • Characteristics of the traditional survey process • One well defined target population (e.g. persons) • A rather homogeneous method for data collection (e.g. questionnaire) • A more or less linear sequence of processing steps (e.g. data cleaning, data editing, data imputation, output) • Final Output is one Output File

9. Data Generating Processes – Integration based • Many Statistics do not follow such a linear production scheme • Examples: Indices, numerous balance sheets, National Accounts, …. • Common characteristic: Data are produced from many different sources • Let us call such processes as integration based processes • Data produced in such way are called integration based data

10. Data Generating Processes – Integration based • Characteristics of integration based data processing • Population: • The underlying population may be split into segments • Example: Expenditures for education: government, private enterprises, households • Many times more than one population is involved, possibly also one population at different times • Example: calculation of indices

11. Data Generating Processes – Integration based • Characteristics of integration based data processing • Data collection: • Data collection is different for different segments and populations • Many times the collected data are the output of already existing data products • Main processing activities are alignment procedures making the different sources comparable • Output may be a set of organized Data Files

12. Data Generating Processes – Workflow View • Workflow for Survey Process

13. Data Generating Processes – Workflow View • Workflow for Integration Based Process

14. Data Quality for Integration Based Production • Two important aspects of data quality • Content quality • Are the measured “concepts” really the target “concepts” ? • Production quality • Are the used methods sound?

15. Data Quality for Integration Based Production – Content Quality • Main reasons for lack of content quality • Slight difference in the measurements of the variables (“concepts” ) in case of reuse of already existing data • Example: • Transport of goods on Austrian rails • Transport of goods according to data from railway authorities (taking not into account that transport may use partly German rails) • Slight differences in the definition of the segments in the underlying population

16. Data Quality for Integration Based Production – Content Quality • Conclusion: Using data already collected for other purposes gives often only proxy variables for the intended variables • Question: Is this in coincidence with your mental concept of the term “Non-Sampling Error”? • Manuals of international organizations are many times rather vague with respect to such problems

17. Data Quality for Integration Based Production – Content Quality • Possible Strategies for Solution • Statistical Models for aligning the concepts • More detailed description of the concepts by using additional variables characterizing the differences as formal properties of the data • More detailed description of the underlying populations by using additional variables characterizing the differences

18. Data Quality for Integration Based Production – Processing Quality • Elements of processing quality • Quality of methods used for the different components of the integration based statistic • This implies that we do not have one method of collection, one editing, one imputation,… but many activities of that kind • Quality of methods used in the integration process • Alignment of variables in order to overcome differences in concepts • Standard activities like plausibility, editing, imputation necessary for the integration activities

19. Assessing Quality for Integration Based Data • If we know the quality of all the components used in the integration process we have to think about transmission of quality in the integration steps • Starting point should be an “Authentic Data System” • All data used in the integration process • Quality information about the different data sets of the system

20. Assessing Quality for Integration Based Data • Distinguish two types of quality transmission • Quality compilation • Methods for representing quality of the overall product • Quality calculations • Algorithms for assessing quality • In both cases we need • Methods for assessing quality • Models of best practice / standards

21. Assessing Quality for Integration Based Data – Quality Compilations • In some cases the best we can do is better representation of the quality dimensions of the used components • Distribution of quality indicators • Concentration of quality indicators

22. Assessing Quality for Integration Based Data – Quality Compilations • Example: Coverage for integration based data • Structure of integrated sources together with coverage information

23. Assessing Quality for Integration Based Data – Quality Compilations • Coverage distribution

24. Assessing Quality for Integration Based Data – Quality Compilations • Coverage concentration with respect to target concept

25. Assessing Quality for Integration Based Data – Quality Calculations • Methods will be in most cases not formulas but advanced statistical procedures for different quality dimensions • Examples: • Measurement of accuracy using variances, standard errors or coefficient of variation • Could be done by using bootstrap (e.g. applied for indices by NSO-GB) • Simulation techniques • Sensitivity analysis (“robustness”)

26. Conclusions • Assessing quality of integration based statistics needs • Clear separation of content based quality and processing based quality • Better documentation / representation of complex production processes, Usage of Workflow Models • Documentation of the authentic data file • Definition of best practice / standards for integration processes • Algorithms for calculation quality dimensions • Methods for representation of quality indicators