InSPIRe

1. InSPIRe Australian initiatives for standardising statistical processes and metadata Simon Wall Australian Bureau of Statistics December 2012

2. Challenges facing NSIs Riding the big data wave New competitors & changing expectations Increasing cost & difficulty of acquiring data Competition for skilled resources Reducing budget Rapid changes in the environment

3. Challenges facing ABS • Reduce the cost and time of doing business • Grow the business through new statistical products and services • Deliver the first large scale digital Census (2016) on time, to budget and quality • while delivering Business As Usual.

4. How will we get there? • Through large scale Innovation across the whole ABS • We will: • radically transform the way we acquire, collate, use, reuse and disseminate statistical information • By: • industrialising, modernising and reengineering our business processes • reengineering our statistical infrastructure and the way we manage information • develop capability needed to meet future needs • Collaborating with other international NSIs

5. InSPIRe

6. What is InSPIRe? • Infrastructure for Statistical Process and Information Management Re-engineering • Building core infrastructure in the ABS for: • Business Process management • Information management • SWM • Statistical Workflow Management System • MRR • Metadata Registry and Repository

7. SWM • The Statistical Workflow Management System • An environment that will be used to develop and manage common business processes • These processes can drive and/or be used by tools and applications.

8. MRR • The Metadata Registry and Repository consists of two parts: • Repository • The centralised ‘bucket’ to store standards based metadata. • Registry • The catalogue that lets you find out what is in the repository.

9. MRR • Registers and stores a wide range of information, including: • Metadata (i.e. Classifications, Variables, etc) • Data – registered in MRR, stored in EDW • Process Definitions and Paradata – information to run a process • Process Metrics – information about a process that was run.

10. MRR • The MRR • Enforces registration of the information, ensuring consistent documentation • Reads information in a range of standard supported metadata formats • (initially DDI and SDMX) • Re-issues metadata in whatever standard format is required by a given process. • Regardless of the format in which the metadata was originally created in. • This information is stored for use and later re-use by the metadata driven processes in SWM • Enables the searching and discovery of metadata for re-use.

11. Interfaces • Processes and Systems will interact with the MRR through a web service interface that supports a controlled set of standards. • Currently limited to DDI 3.1 and SDMX 2.1

12. Users User Interface InSPIRe SWM Business Process Search Register Retrieve MRR

13. Benefits of InSPIRe • Basis for greater automation and reuse of processes and metadata • Faster to market • For individual collections • For building new collections • Cheaper to market (long term) • For individual collections • For building new collections

14. Progress so far… • Proof of Concept Project (2010/2011) • A group of simple use cases to prove the idea of the MRR and SWMs. • Demonstrating: • Benefits • Potential functionality • What is technically achievable • Building capability

15. Pathfinders • Pathfinder Integration (July 2012) • Expanded on PoC • ‘Pathfinder’ projects to produce further use cases for the MRR and SWMs. • Not end-to-end processes, but isolated ‘snap shot’ projects to demonstrate the capabilities of InSPIRe. • Four pathfinders chosen • Including web data capture (e-forms) and REEM (Remote Execution Environment for Microdata)

16. REEM and Web Data Capture Pathfinders • Proved integration with real ABS systems • Demonstrated the benefits of integration with InSPIRe, namely: • Re-use of processes (in SWM) • Standards based metadata (DDI in MRR) driving business tools • Automating business processes (in SWM)

17. Early Adopter Projects • Projects which are positioning to integrate with InSPIRe within the next 12 months. • Includes projects focusing on administrative data, e-forms and the 2016 Census. • Suitable candidates to prove the operation InSPIRe. • implementing processes in different phases of the GSBPM.

18. What we have achieved so far • MRR • Design and review, mappers, shredders, database, automated generation of metadata types, basic search, registry model • SWM • Environment setup, test cases with projects • Capability Development • Best practice guidelines for implementing DDI, metadata content guidelines, InSPIRe integration information packs

19. What we still have to do • MRR • Production version • SWM • Further process definition, integration with other projects • Metadata Authoring Environment • A mechanism for: • Taking existing metadata content and registering it in the MRR • Creating new metadata content via a content creation tool and registering it in the MRR

20. The future…An example

21. Survey Designer Data Collection e-form Transformation Environment Web form Content Creation Tool DDI DDI DDI Retrieve Store Retrieve Store SWM Register MRR EDW