320 likes | 479 Views
The MPEG-7 Standard - A Brief Tutorial -. Ali Tabatabai Sony US Research Laboratories February 27, 2001. Objectives of the MPEG-7 Standard Main Elements of MPEG-7 Scope of MPEG-7 MPEG-7 Application Areas MPEG 7’s relation with other standards. Outline. Fast & Accurate Access.
E N D
The MPEG-7 Standard- A Brief Tutorial - Ali Tabatabai Sony US Research Laboratories February 27, 2001
Objectives of the MPEG-7 Standard Main Elements of MPEG-7 Scope of MPEG-7 MPEG-7 Application Areas MPEG 7’s relation with other standards Outline
Fast & Accurate Access Personalized Content Production and Consumption Content Management Automation • Why do we need MPEG-7 ? • Support for Advanced Query Visual Audio Sketch • + • N e e d
Established in 1988 • A Working Group of ISO/IEC in charge of the Development of Standards for • Coded Representation of Digital Audio and Video MPEG: A Brief History (1) • MPEG: Moving Picture Experts Group • ISO / IEC/JTC1/SC29/WG11
MPEG: A Brief History (2) • MPEG-1: Interactive CD and MP3 11 / 1992 • MPEG-2: DTV, STB, DVD 11 / 1994 • MPEG-4: Web and Mobility ver1: 09 /1998ver2: 11 /1999 • MPEG-7: ??? 08 / 2001 • MPEG-21: Multimedia Framework 11 / 2001
IS NOT a COMPRESSION Standard similar to MPEG-1/2/4 or their Extension IS NOT a STANDARD for FEATURE EXTRACTION/MATCHING Content Description of Various Audio Visual Information MPEG-7: What Is It ? THE MPEG 7 STANDARD Types of Audio Visual Information • Audio, Speech • Moving video, still pictures, graphics • Information on how objects are combined in scenes
MPEG-7: Application Areas • Storage and retrieval of audiovisual databases (image, film, radio archives) • Broadcast media selection (radio, TV programs) • Surveillance (traffic control, surface transportation, production chains) • E-commerce and Tele-shopping (searching for clothes / patterns) • Remote sensing (cartography, ecology, natural resources management) • Entertainment (searching for a game, for a karaoke) • Cultural services (museums, art galleries) • Journalism (searching for events, persons) • Personalized news service on Internet (push media filtering) • Intelligent multimedia presentations • Educational applications • Bio-medical applications
MPEG-7Description Scope for AV Content • Description Granularity • Low-level • High-level • Form • Access • Classification • Link • Context
MPEG-7: Main Elements • Descriptors (D) • syntax and semantics of each feature representation • Description Schemes (DS) • structure and semantics of the relationships between components • Description Definition Language (DDL) • creation of new DS’s • modification/extension of existing DS’s
MPEG-7: Major Functionalities • Systems (ISO / IEC 15938 - 1) • Description Definition Language (ISO / IEC 15938 - 2) • Visual (ISO / IEC 15938 - 3) • Audio (ISO / IEC 15938 - 4) • Multimedia Description Schemes (ISO / IEC 15938 - 5) • Reference Software (ISO / IEC 15938 - 6)
DDL • <Object> • <Label/> • <Definition/> • . • . • </Object> • Instantiation • DS2 • DS2 • Systems • D1 • D2 • D3 • 0001100 • MPEG-7: Main Elements (2) • DS1 • DS3
MPEG-7: Systems • It defines tools to: • provide for efficient storage and transport • synchronize between content and description • manage and protect intellectual property
MPEG-7: DDL and its Components • Description Definition Language: • Creation of the Ds and DS’s: XML Schema & MPEG-7 Extensions • Instantiation of XML • XML Schema: • Data types • Simple and Complex types • Elements, attributes • Inheritance, Abstract types • MPEG-7 extensions: • Array and Matrix data type
MPEG-7: Audio • Sound Effects • Music Instrument Timbre • Spoken Content • Melody Contour
MPEG-7: Visual (1) • Color • quantization, dominant, scalable, color-structure, layout, GoF/GoP • Texture • Shape • region-based, contour-based, 3D • Motion • camera motion, motion trajectory, parametric motion, motion activity
MPEG-7: Visual (2) • Localization • spatio temporal • Others • face recognition
MPEG-7: Basic Visual Structures • Grid Layout • 2D-3D Multiple View • Time Series • Spatial 2D Coordinates • Temporal Interpolation
Video segments Still regions Moving regions Audio segments Low level Audio Visual descriptors Color Shape Position Texture Color Camera motion Motion activity Mosaic Color Motion trajectory Parametric motion Spatio-temporal shape Spoken content Spectral characterization Music: timbre, melody
Basic elements Schema Datatype & Link & media Basic DSs tools structures localization MPEG-7: MMDS Basic Elements Root, Top-level elements, Packages Time, Duration, Medialocators Language Annotation,Person, Place
Creation & production Media Content Usage Content management Content description Structural Conceptual aspects aspects Basic elements Schema Datatype & Link & media Basic DSs tools structures localization MPEG 7: Content Management and Description Title, Creator, Creation location & date, Purpose, Classification, Genre, Review, Parental guidance, etc. (Author generated) Format, Coding, Instances, Identification, Transcoding Hint, etc. (Several instances) Rights holder, Access rights, Usage Record, Financial aspects, etc. (Evolution) Viewpoint of the structure: Segments • Spatial / temporal structure • Audio, video low-level Ds • Elementary semantic information. Viewpoint of conceptual notions • Events, objects, abstract concepts, and their relation
SR1: · Creation, Usage meta information · Media description · SR6: Textual annotation · · Color Histogram Color histogram, Texture · Textual annotation SR2: · Shape · Color Histogram · Textual annotation SR5: · Shape · Textual annotation SR3: · Shape SR4: · Color Histogram · Shape · Textual annotation · Color Histogram · Textual annotation Example of Segment trees Foreground Background
Time Semantic DS (Events) • Introduction • Summary • Program logo • Studio • Overview • News Presenter • News Items • International • Clinton Case • Pope in Cuba • National • Twins • Sports • Closing Axis Segment Tree Shot1 Shot2 Shot3 Segment 1 Sub-segment 1 Sub-segment 2 Sub-segment 3 Sub-segment 4 segment 2 Segment 3 Segment 4 Segment 5 Segment 6 Segment 7
Navigation & Creation & Access production Media Content Usage Summary Content management Content description Variation Structural Conceptual aspects aspects Basic elements Schema Datatype & Link & media Basic DSs tools structures localization MPEG 7: Navigation and Access Efficient support of:discovery,browsing, navigation, visualization Substitution of the original content Adaptation to terminal, network, or user preferences
Hierarchical Summary HighlightLevel HighlightLevel Highlight Highlight Highlight Highlight Highlight Highlight Highlight Segment Segment Segment Segment Segment Segment Segment A-V Data MPEG 7: Hierarchical summary
Frame Frame Frame Property Property Property Sequential Text Text Summary Property Property Sound Sound Sound Property Property Property A-V Data MPEG 7: Sequential summary
MPEG 7: Variation Universal Multimedia Access Adapt delivery to network and terminal characteristics (QoS)
Analytic Model Content organization Navigation & Creation & Access production Media Content Usage Summary Content management Content description Variation Structural Conceptual aspects aspects Basic elements Schema Datatype & Link & media Basic DSs tools structures localization MPEG-7: Content Organization Collection & Classification Description and organization of collection of documents
Analytic Model Content organization Navigation & Creation & Access production Media Content Usage User preferences Summary Content management Content description Variation Structural Conceptual aspects aspects Basic elements Schema Datatype & Link & media Basic DSs tools structures localization MPEG 7: User Interaction Collection & Classification User Interaction User identification and preferences: Filtering, search and browsing
MPEG-7 Its Relation with other standards • AHG on “Metadata harmonization”: • SMPTE: Metadata dictionary, KLV encoding • Dublin Core Metadata Initiative • European Broadcast Union • AHG on TV AnyTime Application • Large number of Liaisons: • SMPTE • Dublin Core • W3C (XML Schema) • etc.
MPEG-7: TimeLine - The Work Plan • Divergence Competition: • Individual work • Definition scope and r • Convergence 1996 1998 1999 2000 2001 Call for proposals Working draft Committee draft International standard Final committee draft Draft international standard
MPEG-7: AV content description for interoperable application Description Definition Language: XML Schema (flexibility) + Binary version (efficiency) Description Schemes: Library of description tools Covers a wide range of generic needs Conclusions on AV Content Description and MPEG-7