MPEG Video Coding II — MPEG-4, 7 and Beyond

1. MPEG Video Coding II— MPEG-4, 7 and Beyond

2. Outline Overview of MPEG-4 Object-based Visual Coding in MPEG-4 Synthetic Object Coding in MPEG-4 MPEG-7 and Further Exploration

3. Review: Object-based Visual Coding

4. Review: Synthetic Object Coding

5. MPEG Standards • MPEG-1: • Storage of moving picture and audio on storage media (CDROM) • MPEG-2: • Digital television • MPEG-4: • Coding of natural and synthetic media objects for multimedia applications • MPEG-7???

6. Problem Definition • How to find the desired content in a multimedia database? • Keywords • “Apple”, “IPAD” • Semantic • “IPAD was announced by Apple” • Query by example • “Find objects like these… ” • Hierarchical approach • Low-level features, e.g., texture, color, motion, etc. and domain specific high-level information • Try http://images.google.com

7. Query • Query by example

8. MPEG-7 • The main objective of MPEG-7 is to serve the need of audio-visual content-based retrieval (or audiovisual object retrieval) in applications • Growth of digital aduiovisual information • To find a video clip of Clinton’s speech on Internet • To find a motocycle like the terminator II • To record TV programs that a viewer like • “Multimedia Content Description Interface” to standardize the description of various types of multimedia content • Still pictures, graphics, 3D models, audio, speech, video, and composition information • Special cases: facial expression, personal characteristics

9. MPEG-7 (cont.) • To enable fast and efficient search and retrieval • From text-based search (e.g., keywords) to content-based search (e.g., color, motion) • MPEG-1/2/4 vs. MPEG-7 • MPEG-1/2/4: Representation of data • MPEG-7: Representation of “metadata” (information about data) • MPEG-7 may use the shape description in MPEG-4 or the motion vector field in MPEG-1/2

10. Scope of MPEG-7 ??? Feature extraction is outside MPEG-7 Search and query are outside MPEG-7

11. Scope of MPEG-7 (cont.) • “Standardize the minimum” • Analysis should not be standardized • Can keep improving • Room for competition • Search engine should not be standardized • Application dependent • Room for competition • Description for the same content may be different for different use domains and different applications

12. Concepts in MPEG-7 • MPEG-7 has developed Descriptors (D), Description Schemes (DS) and Description Definition Language (DDL). The following are some of the important terms: • Feature — characteristic of the data, e.g., color, motion • Descriptor — definition (syntax and semantics) of the feature, e.g. histogram, motion vectors • Description Scheme (DS)— specification of the structure and relationship between Ds and between DSs. • Description • An instantiation of a DS • Combination of Descriptors and DS’s • DDL —A language to define, modify, and combine DS’s • So, MPEG-7 will standardize a set of descriptors and DS’s, DDL, a scheme for coding the descriptions

13. Applications Supported by MPEG-7 • Fig. 1: Possible Applications using MPEG-7.

14. Description Definition extension Language Definition Tags DS1 < scene id=1> < time> .... D1 DS2 Instantiation < camera>.. < annotation </scene> D3 D2 DS4 D3 Structuring DS3 D6 D4 D5 Encoding & Description Schemes Delivery 10101 1 0 Example Relations Between D’s DS’s D2 D1 D7 D8 D6 D4 D10 D9 D5 Descriptors: (Syntax & semantic of feature representation)

15. Descriptors (D) • Localization • Region Locator • Spatio-Temporal • Locator • Color Descriptors Dominant Color Scalable Color Color Layout Color Structure • Texture Descriptors • Shape Descriptors Region Shape Contour Shape 3D Shape • Motion Descriptors for Video • Camera Motion • Motion Trajectory • Parametric Motion • Motion Activity • Other • Face Recognition

16. Camera Motion • Fig. 2: Camera motions: pan, tilt, roll, dolly, track, and boom.

17. Motion Trajectory • Used to define the localization, time, and space of an object • Consists of a set of key points along with a set of optional interpolating functions used to describe the path of the object between two key points

18. Description Scheme (DS)

19. Description Scheme (DS) • Basic elements • Basic datatypes: types provide a set of extended data types and mathematical structures such as vectors and matrices, which are needed by the DSs for describing • Schema tools: assist in the formation, packaging, and annotation of MPEG-7 descriptions • Content Management • Creation and production • Title, creator, creation locations, creation time, texture annotations • Subject, purpose, language, … • Media coding, storage and file formats • Format, compression, … • Content usage • Usage right, usage records, • Content Description • Structural Description. A Segment DS, for example, can be implemented as a class object. It can have five subclasses: Audiovisual segment DS, Audio segment DS, Still region DS, Moving region DS, and Video segment DS. The subclass DSs can recursively have their own subclasses.

20. MPEG-7 for Content Description

21. Fig. 3: MPEG-7 video segment.

22. Content structural aspects (Segment Relationship DS graph) Temporal segments

23. Content Organization • Collections: Collection Structure DS • Model: Model DSs provide tools for modeling the attributes and features of AV content

24. Description Scheme (DS) • Navigation and access • Summaries. • Partitions and Decompositions. • Describe different decompositions of the audio-visual signal in space, time and frequency • Describe different views of the audio-visual data, which is important for multi-resolution access and progressive retrieval • User Interaction • User Preference

25. Description Definition Language (DDL) • MPEG-7 adopted the XML Schema Language initially developed by the WWW Consortium (W3C) as its Description Definition Language (DDL).

26. How to Build and Deploy MPEG-7 Descriptions A description A Description Scheme (structure) . + in DDLanguage A set of Descriptor Values (instantiation of a Descriptor for a given data set) MPEG-7 Description Tools are a library of standardized Descriptions and Description Schemes Adopting the XML Schema as the basis for the MPEG-7 DDL and the resulting XML-compliant instances (Descriptions in MPEG-7 textual format)

27. Example Applications • Digital libraries • E.g., image/video catalog, musical dictionary • Multimedia directory services • e.g., yellow pages • Broadcast media selection • e.g., radio channels, TV channels • Multimedia authoring • e.g., personalized news services, digital photo/video albums

28. Example Uses (cont. ) • Movement • On a give set of objects, describe movements and relations between objects • Retrieve a list of animations fulfilling the described temporal and spatial relations • Scenarios • On a given content, describe actions and get a list of scenarios where similar actions happen • Voice • Using an excerpt of Pavarotti’s voice to retrieve a list of Pavarotti’s records or video clips

29. Further Exploration • Text books: • Multimedia Systems, Standards, and Networks by A. Puri and T. Chen • The MPEG-4 Book by F. Pereira and T. Ebrahimi • Introduction to MPEG-7: Multimedia Content Description Interface by B.S. Manjunath et al. • Web sites: • The MPEG home page • The MPEG FAQ page • Overviews, tutorials, and working documents of MPEG-4 • Tutorials on MPEG-4 Part 10/H.264 • Overviews of MPEG-7 and working documents for MPEG-21 • Documentation for XML schemas that form the basis of MPEG-7 DDL

30. IBM Image Retrieval System based on MPEG-7 http://mp7.watson.ibm.com/

31. Backup

32. Texture regularity (periodic to random) Coarseness (grain to coarse) Directionality (/300)