1 / 24

Some considerations on developing a DWH for SBS estimates

Some considerations on developing a DWH for SBS estimates. Orietta Luzi – Mauro Masselli Istat - Italy march 2013. The rationale of DWH. the complete use of all the information (survey and administrative data) we have on the whole or about the entire target population;

derica
Download Presentation

Some considerations on developing a DWH for SBS estimates

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Some considerations on developing a DWH for SBS estimates Orietta Luzi – Mauro MasselliIstat - Italymarch 2013

  2. The rationale of DWH • the complete use of all the information (survey and administrative data) we have on the whole or about the entire target population; • to build up a platform in which we integrate data and processes (from capturing to integrating data, from checking data to estimating results to disseminating estimates). • theadvantagesin cancellation of sampling errors from one side and process integration and standardization on the other, exceed the disadvantagesdue to increasing non sampling errors and the partial loss of control on administrative data  

  3. goals • First step: To establish a common set of estimates (micro/macro) among SBS and NA on observed economy • Second step: Integration of other surveys on business (structural – ICT,R&D, externalò trade….. and STS) Implications • Revision of sampling designs of SBS surveys • Revisions of production processes

  4. Business Register • BR central role as Selection List and “frame” • The target population is identified with all the enterprises listed in Business Register. • For each unit BR contains two kind of variables: • classification variables (NACE, legal Status, splits and joins, current status, etc..) • content variables (e.g. the total number of persons employed, subtotals of different kind of workers, labour costs, an estimation of turn over ….). • We assume that the classification variables and the variable “persons employed” and the implicit binary variable “existence of business” are by itself target variables and call them Z; they are kept by BR as they are and do not enter in any procedure of data treatment.

  5. Target variables The target variables can be divided into two groups: • A set of “basic variables” X* needed for the estimates required by the SBS - EU Regulation and by NA estimates ; • The remaining variables Y* needed only for NA to be estimated conditionally to the first set

  6. Sources • the administrative sources: tax file, balance sheets, social security worker’s data, fiscal authority survey • SBS surveys at moment, other structural business surveys in the next future

  7. Administrative data • How to asses the quality? Some results from essnet admin data • Essentially: • Definitions how much close are to SBS ones • Data analysis • On overlapping data set • To identify biases analysis of distributions models on relationships between data sources

  8. Administrative data • Advantages: costs, completness • Disadvantages: stability over time – data can be changed for internal decision of the producing administration • Operational definitions • Data indicators from overlapping Agreements with producers data sets Redisign sample surveys

  9. From the collected variables to the target ones For each enterprise, some of the X* variables may exist in one or more of the S sources in different combinations, according to the dimension, the social security rules, the fiscal status etc. only for the sampled respondents units we have a complete set of target variables and these variables are set equal to X*. The variables Ai reported in source “i” may coincide or may approximate the corresponding X*; in the second case it could be possible to “correct” some of them obtaining a set of more precise Xi “estimate” of X*.

  10. From Ai to Xi xij=aij in case of “good” fitting xij= f(aij…..) otherwise,

  11. The matrix X

  12. The matrix X* by establishing a hierarchy between sources

  13. Macro-operators

  14. Donor methods • Randomly • By models • Eg the projection estimator • By calculating a new variable to be used as a distance between donor and recipient • Latent variables model In all the methods we can use ex ante domains or can identify the appropriate variables to build up the donor domains

  15. Establishing coherence:modify data of source i by data of source j • Change some var Xi • Check the impact on the other var Xi • Modify other var Xi • asses Xi E&I rules Outliers detection and removal

  16. A simplified example Source i BR Persons employed and labour costs • Personsemployed > • Turnover valueadded labourcosts • Intermediate costs • Services • Valueadded/personsemployed ?

  17. Sources Hierarchy • Ex ante - Based on • How definitions of source i is close to SBS ones • BR/social security data • SBS sample survey • Balance sheets • Fiscal authority survey • Tax files • Prevoius and current data analysis

  18. Correct A data to obtain X data xi,k = f(ai,k,ai,m…) By data analysis on overlapping data sets By definitions Other considerations

  19. How to fill in the matrix X*to obtain the matrix X** except for the group M1, survey respondents, in all the other cases we have a number of X* variable smaller than K (the needed target variables). So for obtaining the estimates we can consider two options: • a massive imputation of missing values at micro level • an estimation of missing X* at macro level

  20. Micro integration Z, X(1), X(2) …X(S) Selection of X*; E&I; coherence among different sources BR Micro Z X* Calculating X(1)…X(S) Survey Micro integration Z, A(1) A(2)…A(S) Massive imputation E&I; coherence among different sources on imputed units Micro data treatment in the single sources admin sources Micro Z X**Y* Massive imputation Estimation of variables Y* SBS estimation Micro NA treatment NA estimates micro approach

  21. Micro integration Z, X(1), X(2) …X(S) BR Selection of X*; outliers detection; Calculating X(1)…X(S) Micro Z X* Survey Micro integration Z, A(1) A(2)…A(S) Summing up by domains; inconsistencies clean up Micro data treatment in the single sources admin sources Domain D estimates X**Y* Estimation of variables Y* SBS estimates NA estimates macro approach

  22. Cross section and longitudinal approach At moment the cross-sectional approach. However the longitudinal approach has the significant features • using “variations” is the logic adopted by NA estimating procedures • we have “more information” to dealing with. • implication all the functions regarding the data control and imputation procedures could be developed considering both cross sectional and longitudinal “rules

  23. Metadata Generally speaking, we can roughly divide them in three broad sets: • Metadata needed to manage the data • the information related to process and procedures, • the wider documentation related to the different topics in developing the DWH. • Sustainability different tools for managing

More Related