1 / 23

Standardising and Industrialising Statistical Metadata Flows in Statistical Production Process: Practical Steps

The presentation outlines initial steps taken by ABS towards standardising and industrialising end-to-end flows of statistical metadata within the statistical production process. It discusses the context, vision, practical implementation, and challenges faced in creating a harmonised methodology for metadata management. The project aims to create an environment where statistical information can be easily accessed and utilized for evidence-based decision-making. Key topics include the role of metadata in census data, business transformation requirements, and the integration of statistical workflows. The presentation also touches on international progress towards an industry architecture for official statistics production.

aarmas
Download Presentation

Standardising and Industrialising Statistical Metadata Flows in Statistical Production Process: Practical Steps

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Standardising & industrialising “end to end” flows of statistical metadata within the statistical production process Initial practical steps at the ABSHelen TooleJennifer MitchellAlistair Hamilton

  2. Structure of presentation • Context • ABS IMTP (Information Management Transformation Program) • Nascent international progress toward “industry” architecture for production of official statistics • Metadata Registry/Repository (MRR) &Statistical Workflow Management (SWM) • Vision • Proof of Concept (PoC) • Metadata “Census” • Learnings & next steps

  3. IMTP Vision An environment in which Australian Governments and the Community can easily find, access, and combine statistical information which can then be used confidently as an evidence base for policy, to target service delivery and to inform decision making

  4. IMTP • Drivers : Changed needs, changed expectations, changed opportunities (eg data deluge) & threats (eg maintain relevance) • Standardising & industrialising production is fundamental to delivering suites of outputs that are extensive, timely, flexible, sustainable and readily integrated for multifaceted analysis • A necessary enabler, although not sufficient on its own • Required transformation is multifaceted • Business model, business process & practice, business applications, organizational structure, wider national statistical system • Producers face shared challenges, & opportunities, internationally • Strategic vision of the High-level group for strategic developments in business architecture in statistics • The case for an international statistical innovation program - Transforming national and international statistics systems

  5. Business Process & Information • Enterprises require “process-centric” & “data-centric” perspectives on their core business • This point is explored in more detail in GSIM presentation (Session VI) • In our industry various classes of information are • the core product (eg statistics), and • the core raw material (eg data) • “Process” & “Information” are pillars for standardisation & industrialisation within IMTP • Appears fundamentally similar to thinking from Statistics Netherlands around steady states and transformations, including information/metadata to describe & drive transformation • Relevance of METIS CMF (Common Metadata Framework) Part C • Metadata and the Statistical Business Process

  6. Strategic VisionFrom the High Level Group on Business Architecture in Statistics (HLG-BAS)The road to industrialisation & standardisation

  7. Model (for Proof of Concept) of how vision might be actualised locally GSIM Common Reference Model GSBPM SEMANTIC REFERENCE MODEL Conceptual DDI SDMX Simulated Forms Builder MRR Simulated Registry Search SWM Practical Harmonised Methods Harmonised Technology Unresolved discussion in ABS : Where would CORA + CORE constructs be positioned?

  8. MRR + SWMfuture business context Integrated “Statisticians’ Workbench” for Statistical Production (“Process Dashboard”) Applications and services supporting statistical production Statistical Workflow Management System (SWM) [Enables metadata driven processes & ensures efficient flows of metadata in production process] Metadata* Registry/Repository (MRR) [Register & store all metadata used (input, output, guide, enabler) in statistical production process] * More generally “Statistical Information” – including data and metadata Diagram borrows heavily from Statistics Sweden’s presentation at MSIS 2011 Tentative anatomy of a new generation of IT-architecture to support GSBPM-processes

  9. MRR + SWM Conceptual Diagram Statistical Workflow OtherServices Process Execution Engine RulesEngine MRR ID Service ResolutionService SchemaRepository BP InstanceRepository RulesRepository ServicesRegistry MetadataRegistry CentralisedDataRepository Access / UserManagement CentralisedMetadataRepository CorporateDirectory MetadataRepositories DataRepositories

  10. Challenges in reaching the future state • Must support needs of 100+ statistical business processes spanning all statistical subject-matter domains • How can we ensure the information models supported, and services provided, by the MRR will meet the future needs of each of these production processes? • The statistical business processes are necessarily heterogeneous in statistical frameworks, methodologies , required outputs • Which existing needs and methods will need to be supported in future? • Many existing needs and methods will be harmonised during transformation • It is not feasible to transform every single statistical business process and every single application from “As Is” to “To Be” at the same time • How to maintain consistency and integration during the period of transition where “legacy” processes and applications (with “legacy” information requirements) need to be supported along side processes and applications transformed to a “standardised and industrialised” basis? • Maintenance of business continuity (timely and quality assured delivery of agreed statistical outputs to the nation) cannot be risked during transition • Require • extensive analysis (eg thorough understanding of “As Is” and “To Be”) • testing (eg Proofs of Concept) • etc (stakeholder communication and engagement, co-ordinated planning and project management,…)

  11. GSIM Common Reference Model GSBPM SEMANTIC REFERENCE MODEL Conceptual 10/11 MRR Proof of Concept DDI SDMX Common Generic Industrialised Statistics Simulated Forms Builder MRR Simulated Registry Search SWM Practical Harmonised Methods Harmonised Technology

  12. Forms Forms design and approval Select sample Time Series Databases Sample Time series analyses Load sample to PIMS Label files NAB and FAS sign off Create common frame Common Frame Published Data Dispatch Time series to PPW Create survey frame QEWS Frame Data Collection and IFU Paradata (Collection Information) Significance Editing Clean Data MRR Proof of Concept 2010/11 Core case study was elements of statistical business process for Quarterly Business Indicators Survey (QBIS) Business Process Steps Business Output and Input Artefacts Derivation Processes

  13. Simplified End-to-end Process For QBIS Forms design and approval Forms Select sample Time Series Databases Sample Time series analyses Load sample to PIMS Label files NAB and FAS sign off Create common frame Common Frame Published Data Dispatch Time series to PPW Create survey frame QEWS Frame Data Collection and IFU Paradata (Collection Information) Significance Editing Clean Data Ultimately want the metadata in the MRR to drive reuse in the above processes in conjunction with rules and processes stored in the SWM Statistical Workflow Management Metadata Registry and Repository

  14. Proof of Concept : Supported object types* Study Unit (collection cycle) QBIS 2010 quarter 3 Resource Packages Categories ANZSIC 06 (industry classification) Codes Categories Universe (population/ scope) Codes Question scheme (modules/parts) Standard Question Wording Concepts Questions Variables Interviewer instructions Data sets Sequencing Process metrics Collection Instrument Object type was supported in MRR * Relationships (eg between objects) are also an object type in their own right Object type was simulated (not fully modelled)

  15. Metadata Census • Initially conceived 2010.Q2 as project to understand all existing Metadata Stores in ABS • Identify & analyse all Metadata Stores, • Classify types of metadata, • Identify what types of metadata are kept in which stores for which applications • Synthesise findings and provide empirical “bottom up” input to • MRR requirements and design • International “OCMIMF” collaboration project which is developing the Generic Statistical Information Model (GSIM) • Maintain currency of information gathered • reference when planning and managing transformation

  16. What was Found (1) • ABS has hundreds of systems/applications which: • Store data • Store metadata about data • Store metadata associated with data in other systems, • Run processes across data/metadata in systems • Duplicate data/metadata in other systems • Production of comprehensive, integrated findings from the Metadata Census was not feasible within the given time and resource allocation

  17. Example of primary “As Is” applications for one stream of production

  18. What was Found (2) • Inconsistent use of terminology within ABS • Inconsistent modelling (at conceptual, logical and physical levels) of some types of metadata • eg classifications • eg concepts -> QDT “Properties”, CPCF Mat “Properties”, CPCF Mat “Concepts”, DER/QDT “Concepts”, etc. • Inconsistent and insufficient support for versioning of metadata • Loss and redefinition of metadata throughout statistical process • One example • A variable in the ABS Input Data Warehouse has meaning from previously defined metadata around concepts, questions and qualifiers. • During processing, in particular moving data to ABS Output Data Warehouse, links to the earlier metadata isn’t carried through, and so is lost. • As these links are lost, the metadata is being redefined repeatedly throughout the statistical process.

  19. Second phase • Focus in depth on metadata for a specific statistical business process • QBIS used as example • Collate information in order to create an object model describing objects that would be registered in the MRR during the Proof of Concept • GSIM has not yet reached a level of detail and common agreement which would provide a “top down” path for describing these objects • Current target for GSIM to reach required level of agreed detail is December 2012 • The model to be used in the meantime is termed the ABS Transitional Model • Anticipate ABS Transitional Model will be fundamentally compatible with GSIM in most regards • Anticipate adjusting ABS model for alignment with GSIM where appropriate • Model for Proof of Concept was small in scope (the six primary object types) • ABS Transitional Model expected to grow to 20-40 object types by June 2012 • Design of ABS Transitional Model (and GSIM) recognises SDMX and DDI as valuable technical standards supporting implementation & interoperability • Seek to support “crosswalks” to information models underpinning SDMX and DDI where these are relevant and fit for purpose

  20. Example : Questions(Based on DDI Model)

  21. Only a few objects Already many relationships!

  22. Early work in progress on opportunities for rationalisation and reuse

  23. Metadata Census : Conclusions • Things can get very complicated very quickly • Comparing ‘To Be’ to what currently exists makes it even more so. • Especially as what currently exists is inconsistent • Practical analysis of statistical information (primarily metadata) flows throughout the statistical business process is invaluable input • GSBPM is a key reference point for processes • GSIM will be a key reference point for information • Practical analyses in the meantime help build a better GSIM!

More Related