Semantic Integration and Retrieval of Multimedia Metadata

1. Roberto García and Universitat de Lleida, Lleida, Spain Òscar Celma Universitat Pompeu Fabra, Barcelona, Spain Semantic Integration and Retrieval of Multimedia Metadata

2. Contents Introduction Ontological framework Integration framework Integration & Retrieval Scenario Conclusions Future Work

3. Contents • Introduction • Ontological framework • Integration framework • Integration & Retrieval Scenario • Conclusions • Future Work

4. Introduction Digital Media requires Multimedia Metadata. Main requirements: Great amounts of metadata Low-level to user-friendly metadata Interoperability Proposal: semantic metadata and an ontological framework for integration.

5. Contents • Introduction • Ontological framework • Integration framework • Integration & Retrieval Scenario • Conclusions • Future Work

6. Ontological Framework Multimedia ontology. Profit from non-semantic initiative MPEG-7. XMLSemantic Web: XSD2OWL: schema to ontology. XML2RDF: XML instance data to RDF instances. CompleteMPEG-7 OWL Ontology

7. Ontological Framework • MPEG-7: Multimedia Content Description Standard • content-based description for audiovisual information • content searching, filtering and identification • Descriptors: • low-level features • structure • semantic • models • collections • creation...

8. Ontological Framework • Other semantic multimedia metadata initiatives: • Hunter02: not complete, RDF+DAML. • Tsinaraki04: not complete, semantic part of MDS. • Troncy03: not complete, from an ontology to MPEG-7. • Direct benefits: XQuery vs. Semantic Query, example next…

9. Ontological Framework • SegmentType hierarchy. One XQuery for each type vs. One semantic query for supertype.

10. Ontological Framework XML Schema OWL element|attribute rdf:Property owl:DatatypeProperty owl:ObjectProperty element@substitutionGroup rdfs:subPropertyOf element@type rdfs:range complexType|group|attributeGroup owl:Class complexType//element owl:Restriction extension@base|restriction@base rdfs:subClassOf @maxOccurs @minOccurs owl:maxCardinality owl:minCardinality sequence choice owl:intersectionOf owl:unionOf • ReDeFer XSD2OWLMappings:

11. Contents • Introduction • Ontological framework • Integration framework • Integration & Retrieval Scenario • Conclusions • Future Work

12. Integration Framework • MPEG-7 OWL Ontology as integration base for: • Semantic metadata: align ontologies with MPEG-7 ontology. • MusicBrainz (http://musicbrainz.org) • Simac (http://www.semanticaudio.org) • Music Vocabulary (http://www.kanzaki.com/ns/music) • Etc. • XML2RDF:RDFfromMPEG-7 XML instance metadata.

13. Integration Framework. Architecture

14. Integration Framework MusicBrainz to MPEG-7 OWL mappings: musicbrainz:Artist  mpeg7:CreatorType musicbrainz:Album  mpeg7:CollectionType musicbrainz:Track  mpeg7:AudioSegmentType dc:author  mpeg7:Creator dc:title  mpeg7:Title musicbrainz:sortName  mpeg7:Name musicbrainz:duration ≡ mpeg7:MediaDuration

15. Integration Framework Simac to MPEG-7 OWL mappings: simac:Artist  mpeg7:CreatorType simac:name ≡ mpeg7:GivenName simac:Track  mpeg7:AudioSegmentType simac:title ≡ mpeg7:Title simac:duration ≡ mpeg7:MediaDuration simac:Descriptor ≡ mpeg7:AudioDSType simac:mode ≡ mpeg7:Scale simac:key ≡ mpeg7:Key simac:tempo ≡ mpeg7:Beat simac:meter ≡ mpeg7:Meter

16. Integration Framework Music Vocabulary ontology to MPEG-7 OWL mappings: music:Music_Unit  mpeg7:AudioSegmentType music:sections ≡ mpeg7:AudioSegment music:Artist  mpeg7:CreatorType music:key ≡ mpeg7:Key music:meter ≡ mpeg7:Meter

17. Integration Framework. Architecture

18. Integration Framework XML2RDF: XML tree  RDF graph. Deduce blank node types from XSD2OWL ontologies restrictions.

19. Integration Framework • XML2RDF example

20. Contents • Introduction • Ontological framework • Integration framework • Integration & Retrieval Scenario • Conclusions • Future Work

21. Integration & Retrieval Scenario Low-level descriptors from audio track signal: Track editorial metadata: <http://example.org/track#1> a simac:Track; simac:tempo “122”; musicbrainz:trmid “e3c41bc1-4fdc-4ccd-a471-243a0596518f”. <http://example.org/track#2> a musicbrainz:Track; dc:title “Blowin’ in the wind”; dc:author [musicbrainz:sortName “Bob Dylan”]; musicbrainz:trmid “e3c41bc1-4fdc-4ccd-a471-243a0596518f”.

22. Integration & Retrieval Scenario mpeg7:AudioType(track1)  mpeg7:AudioType(track2)  musicbrainz:trmid(track1, trm1)  musicbrainz:trmid(track2, trm2)  (trm1 = trm2)  owl:sameIndividualAs(track1, track2) • Generic MPEG-7 based integration rules. • Example: audio track with the same fingerprint are the same individual, their metadata is merged.

23. Integration Framework

24. Integration & Retrieval Scenario • Transparent retrieval, MPEG-7 based queries. • RDQL example: SELECT ?title WHERE (?track,<rdf:type>,mpg7:AudioSegmentType), (?track,<mpeg7:Title>,?title), (?track,<mpeg7:Creator>,?author), (?author,<mpeg7:Name>,“Bob Dylan”), (?track,<mpeg7:Beat>,?tempo) AND (?tempo >= 120) USING mpeg7 FOR <http://rhizomik.upf.edu/ontologies/2005/03/Mpeg7-2001.owl#>

25. Contents • Introduction • Ontological framework • Integration framework • Integration & Retrieval Scenario • Conclusions • Future Work

26. Integration Framework

27. Conclusions MPEG-7 OWL ontology, an upper-level multimedia ontology for: Semantic metadata integrationalign ontologies to MPEG-7 OWL. XML metadata integrationXML2RDF. Semantic queries for retrievalgeneric queries based on MPEG-7.

28. Contents • Introduction • Ontological framework • Integration framework • Integration & Retrieval Scenario • Conclusions • Future Work

29. Future Work mpeg7:AudioType(track)  mpeg7:Beat(track, t)  (t > 120)  mpeg7:Loudness(track, l)  (l > 0.9)  genres:Rock(track)  danceability(track, “high”) • Semantics-based recommendation systems • http://foafing-the-music.iua.upf.edu • http://rhizomik.net • Infer user-friendly metadata from low-level descriptors.

30. Contact: roberto@rhizomik.net ocelma@iua.upf.edu Thank you for your attention