The National Data Network - what it is - what the ABS is doing

1. The National Data Network- what it is- what the ABS is doing October 2004

2. The National Data Network.. … a national platform for acquiring, sharing, and integrating data relevant to policy and research • Multi-jurisdicition and multi-sector in scope • Shared facilities and standard protocols which will be well understood by a large body of custodians and analysts/researchers • Supports a range of data access arrangements (with strong privacy/ confidentiality protection) • Services as well as data

3. A “distributed custodianship” approach… • Data Custodians will remain in full control of their data. • Major Data Custodians will operate “NDN Nodes” which will: • Contribute metadata to a central catalog of data and services • Manage access to data and services held/operated by the custodian

4. data National Data Network Custodian Node data Custodian Node Data Dictionary Service log Data Dictionary (Eg. Meteor) www.nationaldatanetwork.org Data Catalog Custodian Node Services (eg. access control) Search for Data Resources Access Data (having met all Custodian Conditions) data User

5. National Data Network The National Data Network will link and share data so that the best use can be made of our national information resources without compromising standards of privacy or intellectual property. Why Do We Need a National Data Network? Australia has a rich collection of population surveys, cohort studies, population databases and administrative records that are all held separately, so that their national potential is not being realised. These can be linked together to answer important questions about the wellbeing and life chances of Australia's children and young people. Prof. Fiona Stanley

6. ABS Corporate Plan Objective #1: An expanded and improved national statistical service

7. Data Network “Service Framework” Owner/ Custodian Services Design Data Capture Process Publish Search Acquire Analyse Report Link/ Integrate Researcher/ Analyst Services Network Administration Services Registration Audit Planning

8. Data Network Services • Auto coding • Assisted coding • Character recognition • Collection Control • Respondent Management • Confidentialise • Document • Expose • Define Access Policy • Archive • Acquire test data • Qualify for access • Extract • Subscribe • Document data source issues • Document Findings • Expose Findings Design Data Capture Process Publish Search Acquire Analyse Report Link/ Integrate • Supervised Analysis • Analyse • Graph • Tabulate • Data Item Definitions • Classifications • Standard Questions • Sample selection • Form Design • Edit • Seasonal Adjustment • Estimation • Imputation • Aggregate • Search • Request • Match • Link

9. Getting Started.. • Start with development of services at the interface between custodian and researcher: Publish Search Link/ Integrate • Develop “base” standards for documenting data sources and “access rules”

10. Catalog Register sources and services Data Network – resource registration Census BCPs ABS CURFs RADL Register sources and services ANZSIC Coder www.nationaldatanetwork.org Centrelink Data Definitions

11. 1.Search request 2.Search results Data Network - search Census BCPs ABS CURFs RADL ANZSIC Coder Catalog www.nationaldatanetwork.org Centrelink Researcher Data Definitions Data Sets

12. 2.Undertaking lodged 3. Undertaking accepted 1.submit undertaking 4. Receives access token Data Network – undertaking process Census BCPs ABS CURFs RADL ANZSIC Coder www.nationaldatanetwork.org Catalog Centrelink Researcher Data Definitions Data Sets

13. 1.request file 3. Notify that file was provided 2. Receives file log Data Network – Access – via download Census BCPs ABS CURFs RADL ANZSIC Coder Researcher www.nationaldatanetwork.org Catalog Centrelink Data Definitions Data Sets

14. NDN , Aust Spatial Data Infrastructure, and other networks - must be interoperable Spatial data Spatial data NDN Node ASDD Node NDN Node ASDD Node Search results Aust Spatial Data Infrastructure NDN ASDD Node NDN Node NDN Node Spatial data Eg. An NDN search will access the ASDD and return meta data for Spatial Datasets

15. The Core Software components • Maintain metadata on node • Search • Download/ Access Services NDN Client • Local metadata services • Log activity • Service Downloads NDN Node NDN CENTRAL • Central metadata store • Service Search requests • Log Activity • Authentication/ Access control

16. Demonstration

17. The ABS will collaborate with others to develop a “Demonstration NDN” in 2004/5 • National Data Network Website established • demonstration nodes established with “relevant” data • Services: • Search • Access services working (including Remote Analysis) • Coding

18. Getting Started..A “demonstration NDN” Work is progressing on three “fronts”: • governance, protocols, priorities, resources • infrastructure development • early adopter custodian enlistment

19. Progress – Governance, Plans • Interim Governing Board • Invitation sent to prospective members • First meeting scheduled for 9 December 2004. It will discuss governance arrangements, principles, priorities. • ABS project team assembled, project plan developed

20. Progress – Standards, Architecture • Draft “Specification” document almost complete (detailed technical description of the NDN) • We have been discussing our architecture with some other networks and projects - “Notional Architecture” for Australian Spatial Data Infrastructure, Information Qld project • A Metadata Workshop (convened by AGIMO) will be held on8 November.

21. Progress • “Exposure Draft” Overview produced and being distributed quite widely. • Newsletter sent to 130+ interested people

22. We have been out talking to many potential custodians.. • The response has generally been very positive • We have several agencies expressing interest in participating • Acceptance of the ABS role in the NDN and the need for something like the NDN is high

23. Issues raised.. • Concerns that have been expressed are mainly in these areas: • What will it cost me to participate and what are the benefits? • How does the NDN fit with other networks and initiatives such as Australian Spatial Infrastructure, Information Qld? • We want to author metadata only once/ have a “single point of truth” • What standards are you using and why?

24. Progress • NDN Website established

25. Progress • NDN prototype software developed and being demonstrated..

26. Next stage is to sign-up early adopters via a “statement of intent” The Statement of Intent will help clarify who is doing what and why, timeframes, liabilities, resources. • First candidate is AIHW (draft almost ready to send)

27. NDN Initial Implementation Candidates

28. ABS node – integrated with ABSIW data ABSIW ABS Node Services metadata RADL ANZSIC Coder NDN Central

29. ABS Roles / Responsibilities (Interim) Governing Board Chair – Dennis Trewin NSS Board - steering Geoff Lee & Jonathan Palmer – Project Owners Stats Coordination – Project Office (planning , communication, resourcing..) TSD – software development IMD – ABS Node establishment, Metadata standards development Client Node Managers – will be recruited from relevant SM areas (usually NSCs)

30. ABS Node Content • What we expose (when and under what conditions) is still being discussed. • Initially • population data, data relevant to other early adopters, curf access? • Services: coding (geocoding, RADL) • Later • Probably not publications • Broad range of statistical services

31. ?