1 / 13

ESSnet DWH - Metadata in the S-DWH

ESSnet DWH - Metadata in the S-DWH. Harry Goossens – Statistics Netherlands Head Data Service Centre / ESSnet Coordinator hct.goossens@cbs.nl. Questionnaire stocktaking. No NSI answers ‘YES’ on all these four questions:

kamala
Download Presentation

ESSnet DWH - Metadata in the S-DWH

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ESSnet DWH - Metadata in the S-DWH Harry Goossens – Statistics Netherlands Head Data Service Centre / ESSnet Coordinator hct.goossens@cbs.nl ESS-net DWH

  2. Questionnaire stocktaking • No NSI answers ‘YES’ on all these four questions: • Do you have a single coherent system which covers most of your data in the production of business statistics ? • Is your metadata currently integrated into your data systems ? • Is your data input for current needs integrated into your data systems ? • Are your current output requirements integrated into your data systems ? • No NSI has a finished DWH and metadata system ESS-net DWH

  3. Conclusion Stocktaking Overall daily practice: • All NSI’s find metadata (highly) important • Mostly NO metadata systems operational (yet some in development) • Most NSI’s struggle with metadata • Often capacity problem, ‘extra work’ • Need for guidance on metadata ESS-net DWH

  4. Data & Metadata Data are qualitative orquantitative informationcollected through observation Metadata are data about /describing data. Statistical Data & Metadata Statistical data are data from surveys and/or administrative sources, used to produce statistics Statistical metadata are data about / describing statistical data or better: about STATISTICS Metadata definitions ESS-net DWH

  5. Metadata for a DWH • Technical metadata • Structural informationHow to physically find and use logical data • Process descriptionsHow data flows in the DWH • Authentication rulesWho may do what ? Business metadata • Definitions and descriptionsHelp the end-user interpret and evaluate the data ESS-net DWH

  6. Metadata for statistics production • Structural metadata • Act as identifiers and descriptors of the data: Identify, use, and process data matrixes and data cubesNames of databases, columns, dimensions Reference metadata • Describe the contents and the quality of the data: • Include conceptual, methodological and quality metadataAlgorithms, definitions, Q-indicators • Source: METIS ESS-net DWH

  7. Metadata categories • A metadata item is either • Structural (technical) or Reference (business) Other mutually exclusive categories: • active  passive • structured free - form • standardised non standardised • centralised local ESS-net DWH

  8. The Statistical DWH Data Warehouse Statistics production Statistical Data Warehouse • A central ‘statistical data store’ for managing all availabledata of interest, regardles of its source, enabling the NSI to: - produce necessary information (= statistics !)- (re)use available data to create new data / new outputs- execute analysis and perform reporting ESS-net DWH

  9. Metadata for a S-DWH • Emphasis / focus on: • Active, Structural and Structured metadata • Reference metadata (common to all statistics production) and • Process metadata Describe expected or actual outcome of one or more processesusing evaluable and operational metrics • Quality metadataSource quality, methods used, usability/restrictions • Tracing informationWhich surveys/registers contributed to a specific output ? ESS-net DWH

  10. Metadata standards in a S-DWH • What should be standardised ?Contents, formats, repository, software • Which level of standards should be used ?International/Eurostat, National/NSI, DWH internal • How should a standard be interpreted ?Complete adherence, compatible • How strict adherence should be required ?Mandatory, recommended • Should some components be prioritised ?Big bang, evolution ESS-net DWH

  11. Metadata Quality • The more data, the more need for metadata • The S-DWH contains lots of data, making it dependent on its metadata • Correct, high-quality metadata are vital for its use and for metadata governance: • No metadata  useless data • Bad metadata  misused data • Good metadata  useful data ESS-net DWH

  12. Metadata as a design tool • Metadata is a complex issue and central to the concept and implementation of the data warehouse. The Project needs to consider how guidance may/can be given to ensure that metadata systems allow all the gains of the S-DWH to be exploited effectively. • Metadata has a role to play in the abstract design process, independent of any specific structure. The S-DWH model has implications for the way metadata is collected, transmitted and used. If this is the case: the process design could be determined entirely within the metadata requirements and provide automatic consistency between technical architecture and metadata needs ESS-net DWH

  13. SGA II: WP 1 - Metadata • Fitting S-DWH in current metadatamodels and standards: • Building a framework which defines metadata requirements and roles in the S-DWH context • Study on the use of metadata models and standards: define the various functionalities of a metadata system to facilitate and support the operation of the S-DWH • Provide recommendations and guidelines on the governance of metadata management in the S-DWH • Keep it manageable & practical !!! ESS-net DWH

More Related