Towards More Integrated Generalized Processing Systems – ABS Experience for Economic Surveys

1. Towards More Integrated Generalized Processing Systems – ABS Experience for Economic Surveys ICESIII, Canada, 2007 Presented by Eden Brinkley

2. Outline of presentation • Background • ABS end to end environment • Key elements in more detail • Progress to date • Challenges ahead • Key learnings

3. Background • Business Statistics Innovation Program (BSIP) • Impacted significantly on organizational arrangements, as well as on methodological and technological developments • Formally ran from 2002 to 2005, but aspects still ongoing • Established an overarching end-to-end (e2e) framework for business processes

4. E2E framework Major change program -

5. Key aim of the E2E framework

6. Key enablers for E2E framework • Shared unit record data warehouse • Facilitates common approaches (leading practice) and more rapid deployment • Shared provider environment • Allows "total approach management" for providers • Modular approach to methodologies and systems • Not one size fits all, but a reduced set of options • Focus on continuous process improvement • Quality aspects underpin each part of the process

7. E2E program

8. E2E program outcomes • Improved data quality • Improved provider relations • Significant operating efficiencies • Enhanced opportunities for staff • Stronger statistical leadership • Increased capacity to respond to emerging statistical demands

9. E2E program governance • Current governance arrangements • E2E Program Board (senior management) • Program co-ordination office (E2E Project Team) • 'Champions' for 18 key elements of the E2E framework

10. Annual Integrated Collection Use of ATO data Acquire data improvements Input Data Warehouse Output improvement Making quality visible Metadata management ABS Survey Facilities Editing Futures Total approach management Business Process Management System Account Compilation Confidentiality E2E framework - Integration, Issue management and gap analysis, Business process improvement Access and identity management Backcasting facility Multi-modal initiatives Key champions for E2E

11. E2E processing framework

12. E2E systems infrastructure

13. Metadata • Improved metadata management is a key goal • ABS strategic direction for many years now • Previous focus on output metadata • Now focusing much more on input and process metadata • Recent developments • Introduced a conceptual framework for consistently describing input metadata based on the ISO11179 standard (i.e. covering data elements, properties, qualifiers, object classes) • New metadata driven infrastructure covering ‘pipes’ between front end processes (i.e. data capture, data storage and input processing) • Reduction in use of hard-coded individual applications, and a greater re-use of shared common components

14. Input Data Warehouse (IDW) • In a nutshell: • Single repository for unit record data for businesses (Oracle database) • Contains administrative by-product and directly collected survey data • Replaces existing mainframe, spreadsheet, SAS, Unix/Oracle tables, and other unit data stores • Supports both transactional processing and post-processing analysis • Populated using a rigorous metadata driven approach

15. Outcomes sought from the IDW • Reduced processing costs • Reduced provider load • Improved data quality • Improved data integration, management and security • Faster development and implementation of best practice processes • Facilitates development of new analytical outputs and products • Facilitates National Statistical Service objectives

16. IDW technical details • Oracle database with a star schema design (dimensional model) • Includes services to Put/Get Data and Edit/Transform/Load • Interacts with Lotus Notes • As at June 2007: • Tax Cluster - 4.3 billion rows • Business Cluster - 66 million rows • Trade Cluster - 358 million rows • Transport Cluster - 18 million rows • Total production database size is 1,413 Gb

17. IDW clusters

18. Editing infrastructure • Editor's Toolkit • Notes database • ‘One stop shop’ for managing micro editing activity • Manages workflows and assignment of tasks • Launchpad for other systems components (e.g. Blaise) • Significance Editing Engine • SAS program • Places important anomalies in unit record data into a critical stream for clerical resolution • Blaise • Covers CADE and CATI activities • Supports micro editing, incl. automatic corrections • both batch and on-line

19. E2E implementation strategy • Involves ‘planned opportunism’ • To be achieved largely within existing $ allocations • Timing negotiated by individual business areas • Takes into account other business drivers (e.g. ANZSIC06 implementation) • Business continuity strong driver for all • Involves some non-negotiables • Adopt all relevant components at some point • Migration can be over several years • Need to begin before everything in place • Important to demonstrate some early wins

20. E2E progress to date • Organizational moves completed and initial efficiencies in pre-micro editing functions achieved • Mapping of current business processes largely completed • Some new methodologies developed and incorporated (e.g. Significance editing) • Best practice in most areas still evolving (analysis of current processes not complete) • Some key technologies not yet built (e.g. tools for Macro editing, Account Compilation, etc.)

21. E2E progress to date (cont’d) • IDW in place and operational • Version 1 (Tax cluster for admin by-product data from Australian Tax Office) from December 2003 • Version 2 (for survey and other admin data) from December 2004 • V2 similar to V1, but extended to incorporate Property and Qualifier dimensions, plus other dimensions that didn't apply to tax data (e.g. sampling)

22. Tension between support for the growing number of already migrated collections, and support for the new migration program E2E progress to date (cont’d)

23. Challenges ahead • Articulation of data model and data flows • Better tools and processes for metadata definition and creation • Better linking (re-use) of conceptual metadata used between front, middle and back end processes • Better focus on 'process' metadata used to drive processes • Getting best practice processes developed, documented and implemented across collections • Development of macro editing and account compilation tools/services • Improved knowledge management

24. Key learnings • Standard business processes, methodologies, systems and training, leveraging off each other, do lead to reduced costs and faster deployment • Up front planning of the re-engineering schedule is the key to a smooth transition • Understanding priorities across the program leads to a better balance between development and support effort • Important to evaluate successes and failures – but be prepared to dig deep! • Critical to have strong stakeholder engagement and cross communication

25. Questions?