Using Several Ontologies for Describing Audio-Visual Documents: A Case Study in the Medical Domain

Using Several Ontologies for Describing Audio-Visual Documents:A Case Study in the Medical Domain Sunday 29th of May, 2005 Antoine Isaac1 & Raphaël Troncy2 Multimedia and the Semantic Web

Agenda • Context and Aims • Corpus • Ontological Resources • AV Ontology, Medical Ontology • Annotating the Videos • Querying and Reasoning • Performing SW-inspired Reasoning • Conclusion A. Isaac & R. Troncy - MSW'2005

Describe AV documents • Various uses / Different granularities • Identification, feature extraction, structural decomposition, semantic description • Description deep meaning cannot be accessed and processed by classical systems • Knowledge is often implicit: labels and comments in natural language • Formalisation for description syntax, not semantics • Formal semantics should be interesting • Reasoning with AV document descriptions • Interoperability with formal domain-specific ontologies, allowing to mix AV and domain-related reasoning A. Isaac & R. Troncy - MSW'2005

Objectives • Settle an mini-experiment to show the benefits of using semantic web technologies for annotating audiovisual content • Show that the use of: • formal ontologies, • annotation pattern, • inference capabilities … is highly desirable for a better access to AV content A. Isaac & R. Troncy - MSW'2005

Corpus • Medicine-related TV documentaries • 30 documents, about 30 hours • 50% deal with heart (surgery) theme • Description point of view: how AV features are used to popularize scientific notions • Describe both the form and the content • AV-oriented parts (documentary structure) • Theme-oriented parts (medicine notions) A. Isaac & R. Troncy - MSW'2005

Ontological Resources • Audio-Visual Core Ontology[Isaac & Troncy, 2004] • Characterization of programs and sequences (AV genre) • Decomposition of programs and sequences • Ability to introduce description from the point of view of the activities that constitute the context of AV documents • roles of people involved, way production and broadcast are achieved, etc. • Dual Legitimacy • Use: conceptualizationgrounded on observed purposes and domain initiatives, study of 30 years of documentary practices • Conception: articulation with an upper-level ontology: DOLCE [Gangemi, 2002] A. Isaac & R. Troncy - MSW'2005

Ontological Resources A. Isaac & R. Troncy - MSW'2005

Ontological Resources • Extension of AV core with specific application notions • Exemplification, demonstration, etc. • Re-use of Medical Ontologies • Menelas: domain of coronary pathologies • Concepts dealing with heart surgery • Alternative choices are possible • Galen(concepts dealing with surgical procedures) • Articulation between the ontologies • No use of automatic alignment methods or tools • State by hand OWL axioms (equivalentClass) A. Isaac & R. Troncy - MSW'2005

Description Process • Segmentation of the AV material • Selection of relevant documentary items • Knowledge-based AV annotation • Documentary structure characterization • Segment content description A. Isaac & R. Troncy - MSW'2005

Segmenting the Videos A. Isaac & R. Troncy - MSW'2005

Describing the Videos • Documentary patterns • Layered approach [Troncy, 2003] • AV description language [Troncy & Carrive, 2004] • Knowledge-based Annotation Process • The structure is described at the knowledge level • Concepts and relations from the AV ontology are manually introduced in the description • Content description • Link to external world themes and entities A. Isaac & R. Troncy - MSW'2005

Describing the Videos • Relational Indexing Pattern • Help for user: specify how concepts and relations have to be used (annotation ‘design pattern’) • Important for ontology conception and use (with reasoning knowledge) • Simple pattern that can lead to complex descriptions • Recursive relational structure A. Isaac & R. Troncy - MSW'2005

Describing the Videos A. Isaac & R. Troncy - MSW'2005

Querying and Reasoning • Example: « retrieve the programs that explain a disease and show one of its causes » • Need for the following inferences: • Subsumption • Composition A. Isaac & R. Troncy - MSW'2005

Querying and Reasoning A. Isaac & R. Troncy - MSW'2005

Performing reasoning • A layered complexity approach • RDFS: subsumption • OWL(DL): complex definitions + algebraic properties • Rules: horn clauses • Concrete implementation • RDFS: Sesame Architecture [Broekstra, 2002] • OWL DL: BOR Reasoner [Simov, 2002] • OWL-DLP [Grosof, 2003] + Rules : Sesame custom inference module A. Isaac & R. Troncy - MSW'2005

Examples • DL definitions • (Composition) rule A. Isaac & R. Troncy - MSW'2005

Summary A. Isaac & R. Troncy - MSW'2005

Conclusion • This experimentation: • Uses Semantic Web languages and tools for describing AV contents • Uses different ontologies to capture the structure and the content of the documents • Uses relational indexing pattern for the annotation • Future work: thorough evaluation of those techniques involving real users • A problem that cannot be generally solved: fixing a trade-off between expressivity and tractable computation • Ad hoc, according to the needs of the application targeted A. Isaac & R. Troncy - MSW'2005

Using Several Ontologies for Describing Audio-Visual Documents: A Case Study in the Medical Domain