National Archives of Australia Professional Development Forum 21 October 2008

1. National Archives of AustraliaProfessional Development Forum21 October 2008 Making metadata matter Outcomes from the Clever Recordkeeping Metadata Project Joanne Evans, Sue McKemmish and Barbara Reed on behalf of the CRKM Research team

2. Outline • Metadata • Recordkeeping metadata • The metadata challenge • Service Oriented Architectures (SOA) and recordkeeping • Clever Recordkeeping Metadata Project • Are we ready for SOA? • Building recordkeeping infrastructure

3. Maturing understanding of metadata in IT/IS From metadata as ‘structured or semi-structured data about data’ To metadata as data about: • Data, data sources, data collections, data attributes, data models • Systems, processes, components of processes • System environments • Software suites, programs, program fragments • Specifications • Events • People and their roles in IT systems • Organisations, departments and individuals

4. Metadata about the Resource Metadata Resource Metadata Resource … Metadata is • Structured data/information that describes an object in order to facilitate its understanding, management and use • Itself data – it is in purpose/use that it becomes metadata • Recursive – always metadata about the metadata • Intrinsic • Extrinsic • Dynamic – accrues and changes as information objects move through space and time • Complex And has multiple purposes across different metadata communities

5. Metadata applies at various layers of granularity, aggregation and abstraction For example, VERS encapsulates metadata about the VEO, Record Object, Documents that make up the Record Object and their Encodings.

6. Metadata is defined in schemas and standards • Metadata schema • The semantic and structural definition of a set of metadata, including the names of metadata elements, how they are structured, and their meaning • Also known as metadata set, specification or vocabulary • Metadata standard • A metadata standard is a metadata schema that has been ratified by some authority for usage by a particular community

7. Recordkeeping metadata is • Structured or semi-structured data about: • records at all levels of aggregation, their content, structure and context • related business and social functions, activities, processes, transactions, events • organisations, groups and individuals involved in records creation, management and use • recordkeeping functions, activities, processes transactions, events • mandates, including laws, standards, business rules • relationships • Standardised information which identifies, authenticates, describes, manages and makes accessible, through time and space, records created in the context of social and business activity

8. Cook’s “Electronic records, paper minds …” “We have 21st century digital technologies, but automated 19th and 20th century “industrial” systems” “We need to start a revolution in IM, focussing on: • Sharing and re-using information • Evidence based policy and decision-making • Building IM into our organisation’s DNA – integrated into all applications.” National IM Skills Summit, 15 October 2008, Parliament House, Brisbane

9. The metadata challenge: moving from the Wright Brothers model of metadata management

10. To a 21st century model

11. From • Paper paradigms • Paper standards • Automated paper systems • Unsustainable, unscalable, expensive and resource intensive manual metadata creation and use processes • Stand alone systems or hardwired applications that achieve a degree of interoperability in particular implementation environments only • Metadata standards and schemas that do not support interoperability

12. Towards • Digital paradigms • Digital standards • Digital recordkeeping processes and systems • Sustainable, scalable, automated, metadata creation, gathering, sharing and re-use processes • Integrated systems and federated digital repositories • Metadata interoperability

13. Clever Recordkeeping Metadata Project ARC Linkage Project mid 2003-2006 • Chief Investigator Professor Sue McKemmish, Monash University • Partner Investigators Professor Anne Gilliland-Swetland, UCLA, and Mr Adrian Cunningham, National Archives of Australia Industry Partners and Collaborators Project Links • InterPARES - the MADRAS Metadata Registry • ISO23081 and IT21/7 – Recordkeeping Metadata Standards • NAA Redevelopment of Commonwealth Recordkeeping Metadata Standard http://www.infotech.monash.edu.au/research/groups/rcrg/crkm

14. Objectives of CRKM • To demonstrate how to move away from stand-alone applications and resource intensive, manual metadata creation and use processes • Towards integrated, interoperable digital recordkeeping systems and processes • And the clever use of metadata: • created or gathered once • shared, used and re-used many times

15. Towards 21st century systems enabled by 21st century technologies

16. Web services and service oriented architectures (SOA) • Fits emerging IT paradigms • Enables integration into business systems • Decomposes (but also potentially radically changes) means of delivering recordkeeping • Suits the Web 2.0 world

17. Not that new Moving to new uses Components of larger programmes Products in their own right Constructed to undertake repeatable and reusable bits of a process Reusable across many different applications ‘loose coupling’ between application that uses service and service itself Can be public or private Bring issues: Granularity Packaging What will be useful? Web services

18. Service Descriptions Registry Points to description Locate service Describes service Finds service Consumer Service Services model Exchange messages Source: Based on diagram from http://www.softstar-inc.com/

19. Service Oriented Architecture: SOA is an approach to architecture whereby business services are the key organizing principles that drive the design of IT to be aligned with business needs A paradigm for organizing and utilizing distributed capabilities that may be under the control of different ownership domains. It provides a uniform means to offer, discover, interact with and use capabilities to produce desired effects consistent with measurable preconditions and expectations OASIS definition

20. Reference model • AGIMO Architecture Reference Model, June 2007 • Lightly customised version of US OMB FEA • Provides common language for agencies involved in delivery of cross agency services • Supports identification of duplicate, re-usable and sharable services • Enables more cost effective and timely delivery of ICT services through repository of standards, principles and templates

21. Gershon Review, October 2008 • Many private sector organisations have already embraced service oriented architecture (SOA), where ICT assets are aligned to business services in a standard, flexible and architected fashion, and also the benefits that come with it such as increased agility, re-use and reduced costs. Government organisations are slowly moving towards leveraging SOA in the development of their applications, but there is potential for the pace to be quickened and for further thinking to done at the whole-of-government level rather than agency level to maximise the potential benefits

22. SOA architecture diagram Source: SOA Meta Model, The Linthicum Group, 2007

23. Orchestration or Choreography

24. $5 million SA tech project canned – Australian IT • Mahesh Sharma | October 17, 2008 THE South Australian government has pulled the plug on its $5 million records management system project, ending a five-year saga plagued by repeated cost blowouts, delays and confusion. The project, commissioned by SA's Department of Families and Communities (DFC) in September 2003, was originally budgeted at $2 million but increased to $4 million after the South Australian Housing Trust (SAHT), which was funding the project, granted a request to double the spend. However, SAHT froze the project in mid-2007 when the department requested an additional $1 million for the project. Subsequently auditors KPMG was called in to evaluate whether any components of the project could be salvaged, but in May this year the firm recommended the records management system be scrapped altogether and that the department start from scratch. Details of the project's demise emerged during a government budget and finance committee hearing earlier this week. It remains unclear who is responsible for the failure of the project as there were two groups directly involved in the project - Housing South Australia and the department. Housing SA runs services on behalf of the Housing Trust and had a service agreement with the department. "There were a number of changes (to the project’s management)," Housing South Australia corporate services director Dennis Huxley said at the hearing. "At one stage it was in the Housing Trust, then it transferred to DFC, then we changed it back to Housing. "A number of changes occurred over that time. The project has been floundering for some time," Mr Huxley said

25. MAC report, 2007 • Future technology directions categorised as: • Platform (achieve RK while doing a lot of other things, eg ECM view) • Linked in (records in business systems linked to recordkeeping functionality in dedicated systems) • Embedded (eg Microsoft type approach)

26. Opportunities • Away from expensive and fragile hardwiring of integration • Services as a document-centric technology • Orchestrations for business processes, including recordkeeping • Possibility of transforming recordkeeping in the long term • Interim strategies available for more immediate uptake

27. Recordkeeping Services Service oriented architectures (SOA)

28. Metadata Broker Recordkeeping in SOA

29. CRKM Metadata Broker Validation service Target metadata Source metadata Translation service Crosswalk compilation service Registration Registry services Schema information Repository Machine processable representations of metadata schemas and crosswalks Registry Authoritative information on metadata schemas, metadata elements and crosswalks in human readable and machine processable forms Request for Schema

30. Request the AGEMS – RKMSCA crosswalk Translation Service Registry Service Web Service Layer Web Service Layer Request to translate AGEMS to RKMSCA AGEMS – RKMSCA Crosswalk Service AGEMS RKMSCA Web Service Layer Metadata Broker as cluster of web services Metadata Broker Client

31. Are we ready for service orientation? Design for recordkeeping Design for recordkeeping metadata Design for interoperability

32. To participate in SOA … • Overcome paper thinking and dominance of paper paradigm

33. To participate in SOA … • Move beyond static resource discovery metadata models SPIRT Conceptual Models incorporated into ISO 23081:1

34. … Dynamic metadata model Dynamic Metadata Model from ISO 23081:2

35. To participate in SOA … • Recordkeeping metadata standards for machine rather than human processing in order to realise interoperability • Standards development activities must:- • be aware of standardisation activities in the IT sector in general, understand the forces driving their development, and be compatible with these activities, • address ontological and machine processability issues, and • seek an optimal balance between practical imperfections and theoretical models.

36. Conceptual Model Recordkeeping Metadata Standards Metadata/Data Standard Metadata/Data Standard Metadata/Data Standard Abstract Version 1 Version 2 Version n Representation (e.g. XML Schema, RDFSchema, etc.) Encoding 1 Encoding 2 Encoding n Registry Objects Version 1 Version 2 Version n Transport and Exchange (e.g. HTTP Get, OAI-PMH, SOAP, etc. )

37. Standards and interoperability • Standards compliance does not guarantee interoperability • Recordkeeping metadata standards are just a part of an infrastructure for interoperability • Balance between standardisation activities for best current practice versus standardisation activities to deliver better next generation practices

38. Records Management Application Web Management Systems Archival Gateways Email and Desktop Applications Metadata Broker Subject Portals Community Archives Archival Management Application Business Information Systems Implementing recordkeeping metadata Schema Web Management System Crosswalk Archival Gateways Schema Email Records Management Application Archival Management Application Subject Portals Desktop Applications Crosswalk Crosswalk Schema Schema Schema Schema Community Archives Business Systems Crosswalk Schema Metadata Broker

39. Recordkeeping Services Innovate to service oriented future

40. For more information see the Clever Recordkeeping Metadata Project Website • http://www.infotech.monash.edu.au/research/groups/rcrg/crkm