A Stepwise Modeling Approach for Individual Media Semantics

1. A Stepwise Modeling Approach for Individual Media Semantics Annett Mitschick, Klaus Meißner TU Dresden, Department of Computer Science, Multimedia Technology Group „Applications of Semantic Technologies“ (AST' 06)

2. Outline • Introduction • Approach: • Semantics for Personal Media Management • Information Instantiation and Integration • Implemented System Architecture • Conclusion and Future Work AST 2006

3. Introduction • Media Management @ Home: • Growing amount of digital assets (text, images, video, audio) • complex, high-dimensional • User: • non-/semi-professional • individual conceptualizations • usually minimum efforts to annotate/organize • Tools mostly apply administrative (objective) metadata vs. user interpretation/conceptualization • Goal: automated modeling of essential semantic descriptions from existing metadata to ease annotation work AST 2006

4. Introduction • Solutions: • Automated feature extraction and indexing techniques to manage complexity of huge multimedia collections • low-level features  Similarity-based Search (Query-by-Example, etc.) • higher-level features  Pattern Recognition/Classification (objects, people, genres, etc.) requires prototype features (user context, user feedback)  “Semantic Gap” • Media descriptions based on common metadata standards to enable interoperability & comparability • MPEG-7 Multimedia Description Schema • RDF/OWL representation for semantic modeling • Semantic Desktop • Personal Information Management based on Semantic Web Technologies AST 2006

5. Approach :: Semantics for Personal Media Management • Knowledge representation • Allow description of background-knowledge (context information like events, activities, actors, locations, etc.) • Allow integration of domain models and individual extensions (e.g. integration of MPEG-7 MDS, FOAF for description of people, etc.) • Manageable, straightforward  for non-professionals • ABC [Hunter, et al.] • “facilitate interoperability between metadata ontologies“, “conceptual basis for automated mapping …”, “guidance to communities…” • Base classes to attach domain-specific entities  MPEG-7 MDS • Temporal semantics and related actualities (who, what, when, where) evolution/transition of objects AST 2006

6. Approach :: Semantics for Personal Media Management • Acquisition of as much information as possible from content + context • File information, administrative metadata, structural features, etc. • Information sources depend on application context • naming, availability/accessibility, semantics, etc. of features/attributese.g. “Date Time Original”/”Date Created”/…, “Creator”/”Author”/…, etc. • properties might be relevant, or redundant in some contexte.g. “EXIF:ExposureTime” and “EXIF:ShutterSpeedValue” AST 2006

7. Approach :: Information Instantiation and Integration • First step: • Apply sets of mapping rules to construct media-centric description (MDS) • Mapping rules expandable regarding supported metadata schemes( developer) Make=FUJIFILMModel=FinePix F601 [(urn:Exif:Make rdf:value ?j) (urn:Exif:Model rdf:value ?k) (urn:Filepath rdf:value ?l) hashCode(?l, ?t) makeTemp(?x) -> (?t mds:creationTool ?x) (?x mds:make ?j) (?x mds:model ?k) (?x rdf:type mds:Device)] <rdf:Description rdf:about="#ins61737146"> <mds:creationTool rdf:nodeID="A0"/> ... </rdf:Description> ... <rdf:Description rdf:nodeID="A0"> <mds:make>FUJIFILM</mds:make> <mds:model>FinePix F601</mds:model> <rdf:type rdf:resource="#Device"/> </rdf:Description> <rdf:Description rdf:about="urn:Exif:Make"> <rdf:value …>FUJIFILM</rdf:value> </rdf:Description> <rdf:Description rdf:about="urn:Exif:Model"> <rdf:value …>FinePix F601</rdf:value> </rdf:Description> AST 2006

8. Approach :: Information Instantiation and Integration • Next steps: • Apply derivation rules (schema level)to establish relationships to actualities and temporalities( developer, user) • Apply classification rules (instance level)to refine description model regardingknown pattern ( user) AST 2006

9. Approach :: Implemented System Architecture • Design rationale: • Automated import and indexing of media items triggered by file system actions • Registry of media type specific analyzing components  providing metadata and feature extraction • Processing of semantics and ontology model wrapped by generic model API  independent from RDF/OWL storage and querying solution (Jena, Sesame, …) • Extensible, flexible service architecture plug-in based setup, extension at run-time(OSGi service platform) AST 2006

10. Conclusion and Future Work • Our approach: • Part of the K-IMM (Knowledge through Intelligent Media Management) project  managing private media collections with the help of semantic technologies • Stepwise modeling of resource descriptions to separate distinct problems of construction of media semantics • Building objective media descriptions from various sources • Deriving subjective, user-oriented semantics • Future Work: • Strategies to extend and refine basic rule sets by means of user feedback, context capture • Distribution of modeling steps (Web Services/Agents) • Detailed, comprehensive evaluation AST 2006

11. Thank you for your attention! Annett Mitschick annett.mitschick@inf.tu-dresden.de Multimedia Technology Group, Department of Computer Science, TU Dresden http://www-mmt.inf.tu-dresden.de AST 2006