Timing Issues in Multimedia Formats: Review of the Principles and Comparison of Existing Formats

1. Timing Issues in Multimedia Formats: Review of the Principles and Comparison of Existing Formats Author’s- Boris Rogge Jeroen Bekaert Rik Van de Walle

2. About the paper • In recent years. a large number of new multimedia (MM) formats have been created. • It is widely recognized that the life cycle for a MM document consists of three areas. • First, a document model is needed to model the MM scenario. • Second, this document model is to be translated into a computer description. • Third, a set of synchronization primitives must be defined in order to be able to present the computer description to the end-user. • For each of the three areas a model is put forward. • From these models, a reference model is created consisting of ten rules and a document model. • This model is then applied to a number of real-world MM formats • Finally, a comparison is presented.

4. First, Document Model Document model is made up of two parts: • Spatial model. • Temporal model. These models determine whether information will be lost during the conversion from a MM scenario to a computer description.

5. First, Document Model Temporal model: • The temporal model describes the synchronization issues within individual media elements and the synchronization issues between different media elements. This implies that the temporal model must specify a number of temporal relationships.

6. First, Document Model The synchronization problem consists of two parts: 1) The description of the relations within a time scenario, based on a temporal model and 2) The handling of the defined specifications during the execution We deal with first problem • Firstly, the need for a fine-grained temporal • Secondly, the presence of interaction with the end-user.

7. First, Document Model Lets introduce timestamp and interval based temporal models, compare them and extract the info that will be used in the reference model. 1) Timestamp-Based Temporal Models: 2) Interval-Based Temporal Models: 3) Timestamp Versus Interval-Based Relationships: 4) Reference Model:

8. Second, Computer-Based MM Descriptions Computer based computer description should contain two types of information. 1) Structural information (raw data). 2) Presentation information (how raw data should be presented.) Requirements for a computer-based MM description • Basic requirements • Advanced requirements

9. Second,(Basic Requirements) • Temporal model • Spatial model • Interaction with the end-user.

10. Second, (Basic Requirements) Temporal Model: • During the generation of a computer based description, we need a temporal model that: 1) Allows us to express all desired temporal relationships and 2) Incorporates interaction with the end-user.

11. Second, (Basic Requirements) Spatial Model: • Three approaches of positioning the visual elements on the presentation medium can be distinguished: 1) Absolute positioning (based on a coordinate system) 2) Directional relations (relations like strong-north and weak-north) 3) Topological relations (relations like disjoint, meet, and overlap)

12. Second, (Basic Requirements) Interaction Model: The expressive power of a temporal model determines the amount of interaction that can be used. There are two possible ways of interaction with the end-user. 1) Navigation: the end-user can influence the further course of the MM document by interacting with the interface presented to him. 2) Presentation: the end-user can change the visual and audible layout of the MM document through interaction with the presented interface.

13. Second, (Advance Requirements) • Reuse • Adaptation • Presentation-Neutral Description

14. Second, (Advance Requirements) Reuse: • Reusing information is situated on three levels (granularity of reuse) • Reuse of media data • Reuse of a fragment • Reuse of a complete MM document as a part of another computer description.

15. Second, (Advance Requirements) Adaptation: • The presentation of MM documents should adapt to the end-user’s context. The targeted end-user system environment, available network bandwidth, and CPU power are resources that should be taken into account when presenting a MM document.

16. Second, (Advance Requirements) Presentation-Neutral Description: • Current networks are very heterogeneous, having different bandwidth capacities, different devices attached to them, and different operating systems running on these devices. It is therefore in the interest of future MM document models to be platform- and program-independent.

17. Third, Synchronization Mechanisms • The generated description is analyzed and the content is synchronized by the synchronization mechanisms. • This section is mainly based on – 1) First, we describe the different tasks for synchronization mechanisms. 2) Second, we focus on real-world effects encountered when processing computer- based descriptions. 3) Third, a system model is introduced which can be used with distributed systems. 4) Finally, a number of quality of service (QoS) parameters are derived starting from the proposed method.

18. Third, (Synchronization mechanism) Tasks for Synchronization Mechanisms: Whenever MM data are to be transported through a network or presented on an end-user system, resources are often inadequate. Variance on transmission times and variance on local processing power can disturb the correct playback of MM data. The mechanisms (hard- and software) trying to close the gap between the envisioned execution time and the limited possibilities of the end-user system are called synchronization mechanisms.

20. Third, (Synchronization mechanism) • Whenever the synchronization is done at the end-user system, not only must raw media data (video frames, audio samples, text, animation, etc.) be transmitted, but the synchronization information must also be sent. We describe three possible methods for transmitting this synchronization information. 1) Out-of-band information 2) Timestamps 3) Multiplexing

21. Third, (Synchronization mechanism) QoS Parameters • We have 5 QoS parameters 1) Number of images or samples per second 2) Number of bits per second 3) Variance on the delay: 4) Bit-error probability: 5) Data type and machine/hardware type:

22. Reference Model This reference model will allow the systematic comparison of existing MM models. When defining this reference model, two main goals were set: 1) The model must be able to compare existing MM formats in a quantitative manner 2) The model must take into account future needs for MM formats.

23. …Reference Model Ten Criteria's: 1) Criterion 1: Fine-Grained Temporal Model: 2) Criterion 2: 29 Interval Relationships: 3) Criterion 3: Timestamp Relationships Versus Original Temporal Information: 4) Criterion 4: Interaction: 5) Criterion 5: Extensibility: 6) Criterion 6: Reusability: 7) Criterion 7: Adaptability: 8) Criterion 8: Presentation-Neutral Representation: 9) Criterion 9: Real-Time Application Support: 10) Criterion 10: QoS Parameters:

24. …Reference Model Document Structure: • Document structure is proposed taking into account the ten criteria presented. • The document structure is shown in the following figure.

27. MM Format Comparison We identify three different types of media formats: 1) Compressed media formats (formats using compression algorithms) 2) Streaming media formats (formats optimized for streaming the data over a network) 3) Media delivery formats (formats describing the relationships between a number of other MM formats)

29. Reference Model Evaluation We will have a short look at two possible applications 1) Video-on-demand 2) Interactive TV

30. … Reference Model Evaluation Video-on-demand • Networking requirements • Navigation requirements • Resource discovery • Rights management Based on the 4 requirements we would say that either Real media or ASF are best fits.

31. … Reference Model Evaluation Interactive TV • Interactivity • Return channel • Navigation requirements For this example no single MM format contains the properties needed to create an interactive TV application, it is rather difficult to select one. Based on these arguments we dare state that the MPEG-4 standard looks very promising at this point in time.

32. Conclusion • A reference model was created for future MM formats based on state-of-the art technology in timing issues. • The goal of this reference model is providing a tool for comparing existing MM formats. • A document model were defined within this reference model. • Based on this reference model we performed a comparison of a number of existing MM formats.

33. Thank You