Preservation of Interactive Multimedia Performances: Ontologies and OAIS

1. Preservation of Interactive Multimedia Performances: Ontologies and OAIS Tran Vu Pham & Kia Ng ICSRiM - University of Leeds, info@icsrim.org.uk, www.icsrim.org.uk

2. Outline • Interactive Multimedia Performances • Preservation of Interactive Multimedia Performances • OAIS – Open Archival Information System • Roles of Metadata and Ontologies • Extensions of CIDOC-CRM for interactive multimedia performance domain • Summary

3. Motion Analysis and Recognition 3D motion data Motion Capture and Processing Motions Motion-Multimedia Mapping Strategy Mapping Parameters GUI (For monitor & control) Multimedia output Multimedia Generation Interactive Multimedia Performance Production Process Interactive multimedia performance production process (based on Music via Motion (MvM) system, www.kcng.org/mvm/)

4. The Need for Preservation of Interactive Multimedia Performances • Reconstruction of a performance at a later time • Education and training: for study and analysis on various aspects of an interactive multimedia performance • Historical study/analysis in performing arts in the future • … • just to preserve • we do not really know what our future generations will want to know from us

5. Preservation of Interactive Multimedia Performances • Interactive multimedia performances rely heavily on digital media • “Digital information is more and more ubiquitous, more and more indispensable. Yet digital information is extremely fragile.” (CASPAR project) • Performances are abstract and cannot be physically stored or archived • To reconstruct an interactive performance the creation process needs to be preserved • Various digital components involved and their inter-relationships need to be preserved:

6. Interactive Multimedia Performances • Some examples • Music via Motion (MvM), video • Creative Interface for augmentation with traditional musical instruments, video • Gesture controllers • …

7. Elements of an Interactive Multimedia Performance – An Example • For an interactive multimedia performance using MvM system • The performance procedure • Motions of performers are captured and stored in 3D scene data files and video files • A MAX/MSP patch that contains the mapping strategy • MAX/MSP application to run the patch • The operating system on which the MAX/MSP application was running • The hardware system used in the performance: the PC, cameras and their setting, amplifier, mixer and speakers. • Multimedia contents produced during the performance

8. Relationships amongst Components

9. The Challenges • Preserve the digital components • Changes of software, hardware, data formats, operating systems, etc. • Interoperability with the future is challenging • Reconstruct a performance from preserved components • Changes of community knowledge • Knowledge of designated user community at some point is the future may be different from our knowledge • How to keep knowledge used to reconstruct a performance in the future? • DRM issues related to preserved contents

10. Reference Model for an Open Archival Information System (OAIS) • Originally a recommendation for space data system standards • An ISO standard for archival information systems • OAIS defines a standard set of terminologies, conceptual and functional models for an archival information system • OAIS can be used to promote the interoperability amongst digital libraries and archives that maintain digital information over the long term

11. OAIS Concepts – Information Object

12. Conceptual Model of Information Object

13. Reference Context Provenance Fixity OAIS Concepts – Information Package

14. OAIS External Interactions

15. OAIS Functional Model

16. Metadata and Ontology • Metadata • Data about data • Preservation metadata • Metadata designed specifically to support the digital preservation process • Ontology • “An ontology is a specification of a conceptualization” (Gruber 1993) • Practically, commonly agreed shared conceptualisations

17. Roles of Metadata and Ontology in Digital Preservation • Metadata and ontology can be used for: • Description about the data (e.g. OAIS Descriptive Information, OAIS Representation Information) • Logical relationships amongst data objects (e.g. OAIS Context Information) • History of data (e.g. OAIS Provenance Information) • Digital right associated with data objects (e.g. OAIS Access) • To allow “interoperability” amongst digital archival systems and automatic reasoning about relationships amongst digital objects in an archive

18. A Simple Example: Preserve a Pizza Using Pizza Ontology • The example ontology taken from: http://www.co-ode.org/ontologies/pizza/2007/02/12/

19. Using Ontology for Preservation of Interactive Multimedia Performances • Help the user to input all necessary data elements to the archive • Help to perform consistency check on data elements of a performance during the ingestion and after each update is made against individual elements in archives • Help designated users to retrieve required data elements from the archive and to locate necessary equipments to reconstruct a performance • Enable interoperability amongst digital archives involved in preservation of interactive multimedia performances

20. Extending CIDOC-CRM and FRBRoo for Interactive Multimedia Performances • CIDOC-CRM • The International Documentation Committee of the International Council of Museums Conceptual Reference Model • An ISO standard, initially designed for cultural heritage data • Being extended to promote interoperability amongst digital libraries • Provide general concepts for describing temporal and physical relationships amongst entities • FRBRoo • An object oriented version of Functional Requirements for Bibliographic Records (FRBR) of the International Federation of Library Associations and Institutions (IFLA) • For cataloguing bibliographic records • Being aligned to CIDOC-CRM

21. CIDOC-CRM extends FRBROO extends IMP Domain Ontology A Layering Approach CIDOC CRM + FRBRER = FRBR00… Martin Doerr and Patrick Le Boeuf Generic Domain Specific

22. Scope of Extended Concepts for Interactive Multimedia Performances • Concepts for describing generic procedures of an interactive performance • Execution of these general procedures is expected to produce a performance • Concepts for describing activities during an actual performance (based on temporal concepts defined in CIDOC-CRM) • Concepts for describing required digital data objects, applications, equipments and instruments, and the logical relationships amongst them

23. Performance Procedure

24. Describing a Performance

25. Describing a Performance (cont’)

26. Digital Object Hierarchy

27. Relationships amongst Digital Objects

28. Summary • Digital preservation of interactive multimedia performances is challenging • Preserving knowledge is vital for future reconstruction of interactive multimedia performances • Ontology plays an important role in preservation of knowledge within OAIS framework • Develop ontologies and framework for describing interactive multimedia performances, by extending CIDOC-CRM and FRBRoo • The ontology will be evaluated within CASPAR framework using the contemporary art testbed

29. Thank you! • Contact information: • CASPAR Project URL: • http://www.casparpreserves.eu/ • CASPAR team at the University of Leeds • Kia Ng: kia@computer.org, www.kcng.org • ICSRiM: • www.icsrim.org.uk