ABS Statistical Databases

1. ABS Statistical Databases Session 6 Mark Viney Australian Bureau of Statistics 6 June 2007

2. "Stove Pipe" approach INPUT THRUPUT OUTPUT INPUT THRUPUT OUTPUT INPUT THRUPUT OUTPUT

3. "Clearing-House" Approach THRUPUT INPUT OUTPUT OUTPUT ABSIW INPUT IDW OUTPUT OUTPUT INPUT OUTPUT Standardised Standardised Standardised interface interface interface

4. e-Census

5. e-Census 2006 • Conducted 2006 Population Census with the option of electronic submission of responses • drop-off/ pick up • drop-off/mail back in 2011 • 10.2% of returns were electronic • no edits incorporated into electronic form • less visits to pick up paper forms • less paper forms • less scanning/repair

6. ABS Secure Deposit Box

7. Secure Deposit Box • An externally facing database to allow respondents to lodge their raw data electronically • Excel spreadsheet (essentailly replacing a paper form) • Administrative datasets

8. ABS Statistical Databases • ABS Input Data Warehouse (ABS IDW) • ABS Information Warehouse (ABSIW)

9. ABS Input Data Warehouse (ABS IDW)

10. Input Data Warehouse • Used as a repository for data as soon as it is entered into ABS computer systems • Initially used for data received electronically • Now used to load (and process) survey data

11. Input Data Warehouse • Structure • Star schema • 1 fact table and several dimension tables • each data cell is stored as 1 row in the fact table

12. Star Schema

13. ABS Input Data Warehouse - What it allows us to do • Keep a historical record of what each cell was at every point in the processing • Reason for the change • when it changed • who changed it • change in value • Ready access to both current and historical data

14. ABS Input Data Warehouse - What it allows us to do • A data store for use with :- • editing • imputation • winsorisation • estimation • Quick easy analysis and confrontation of data:- • across time • across dataitems • across data sources

15. ABS Input Data Warehouse - Flow of Information

16. What we hope to achieve from IDW • Reduced costs • Improved data quality • Tools to assist with management of data providers • Better understanding of Editing processes • Significance Editing • One single source of microdata • for all statistical collections • Well managed and secure data storage

17. ABS Information Warehouse (ABSIW)

18. ABS Information Warehouse • Need to make both data and metadata:- • Visible • Relatable • Accessible • Understandable • Reliable • Media Independent

19. ABS Information Warehouse • Visible • central known location • Relatable • across collections • Accessible • tools to allow extraction and manipulation

20. ABS Information Warehouse • Understandable • data fully described by metadata • Reliable • single source • high availability • Media Independent • single source for outputs • paper publications • electronic releases • ad - hoc requests

21. ABS Information Warehouse • Define and manage metadata • Load lightly aggregated data • Validate data as compliant with metadata • Manipulate data • Produce statistical outputs • Make data publicly available

22. ABS Information Warehouse - Flow of information Load info about collection Disseminate time series Load data to the ABSDB Closed DB Sign-off data to the ABSDB Open DB Data from a collection Load info on how to categorize data Disseminate output tables Derive ad-hoc client data requests Load info on what data items mean Processing System Information Warehouse PPW

23. ABS Information Warehouse - Define and Manage Metadata • Interfaces to manage metadata • load, amend, validate, extract • dataitems,classifications, collections,datasets,publications • Application Program Interfaces (API) to link with other systems/programs • increasingly using XML

24. ABS Information Warehouse - Loading data • Load data from major sources • Input Data Warehouse • SAS • FAME • SuperCROSS

25. ABS Information Warehouse - Generating New data Cubes • Passing data through one or more steps to derive a new table • aggregation • drop dataitems • calculate new items

26. ABS Information Warehouse - Other Manipulations • Seasonal Adjustment • SeasABS (X-11) • Chain Volume Measures • FAME (timeseries) • Supertables • Confidentialisation • Disclosure Avoidance Analysis System

27. ABS Information Warehouse - Data Delivery • Data combined with metadata • Output formats created tailored to specific use • spreadsheets • timeseries • supertables • paper publications • electronic release

28. ABS Information Warehouse - Public Release • Make data available on an internally accessible database at a predetermined time (usually 11:30 am Canberra time) • This data is then available to ABS Statistical Consultants to satisfy customer requests • Feed data to website • www.abs.gov.au

29. ABS Website www.abs.gov.au

31. National Data Network (NDN) www.nationaldatanetwork.org

32. Australian Bureau of Statistics We assist and encourage informed decision making, research and discussion within governments and the community, by providingleading a high quality, objective and responsive national statistical service

33. National Data Network • Website that raises visibility of statistical data • regardless of publishing agency A national platform for acquiring, sharing and integrating data relevant to policy and research in Australia

34. National Data Network • One central website • descriptions of data • quality statement • references to other data • Several websites (Nodes) owned and maintained by other agencies www.nationaldatanetwork.org

36. National Data Network

37. National Data Network • Current Focus • Publish / Search / Acquire • Future Focus • Design / Capture / Process • Analyse / Report

38. Australian Bureau of Statistics We assist and encourage informed decision making, research and discussion within governments and the community, by providingleading a high quality, objective and responsive national statistical service

39. Questions?