Enabling integrated cross-media entertainment services over heterogeneous networks

1. Enabling integrated cross-media entertainment services over heterogeneous networks Introduction: Reasoning, business aspects, involved processes Dr. Kostas Karpouzis Image, Video and Multimedia Systems Lab, National Technical University of Athens Institute of Communication and Computer Systems

2. The current scene… • Digital Content • A valuable asset nowadays • Associated with a variety of emerging and novel distributed multimedia services • Big revenue bringers: entertainment services • Convergence of media and technologies • From single, distinct ones to multi-disciplinary ones • Allowing advanced service provisioning and consumption

3. The current scene… • The plethora of terminal devices available for networking access lead to immense market opportunities for content and service providers • The emergence of advanced network infrastructures enables fast, efficient and reliable end-to-end transmission of huge multimedia content • Media providers faced with two choices: • select the most appropriate device for their contentand tailor  the consumer experience and business models to take advantage of this environment • invest in multiple devices, access networks and content formats • the current trend… • enables transparent access to services, thus more subscribers

4. Why entertainment..? • Entertainment and leisure market is an attractive commercial target for many business groups • Data providers, telecommunication companies, broadcasters, online publishers, device producers • It further allows increasing revenue streams • A never-ending consumer interest • Online entertainment and leisure market is a small percentage of the overall • it is expected to become the major boost of this business sector

5. The underlying concepts… • Cross-media world: what does it mean? • Publish/consume content to a variety of terminals • fixed or mobile, accessing heterogeneous networks • Actually refers to: • Create once, Publish everywhere (COPE paradigm) • At content/service providers, mediators, network operators • Universal access to content (UMA paradigm) • From content/service providers to end-users • Heterogeneous networks • Diverse core and access networks • Satellite, wireless, mobile, cable, ISDN, … • Different QoS management • User-oriented • Services driven by users’ real needs and preferences

6. So… • How can the same content be transmitted over different channels and consumed at various terminals without costly re-processing and re-production under specific QoS requirements dictated both by content/service providers and end-users?

7. The business implications • Content/service providers, network operators, content/service mediators • increase market competitiveness and advantage • establish new user-centered business opportunities • definition of new business models • increase subscribers • open up new revenue streams • be technologically advanced and thus competitive • Consumers • Access to any type of entertainment service wherever they are, using any type of terminal • Services tailored to their needs and preferences

8. The underlying value chain Content / Service Mediation & Management Content / Service Creation Content Distribution & Delivery Personalized Consumption Digital Rights Management

9. The technological requirements • Enable • Create once, publish everywhere • Universal multimedia access • Adapting technology to people & people being the centre • i.e., primarily • Context awareness, content adaptation • scalable coding, automatic transcoding • use of standardized metadata descriptions & profiling (MPEG-7, MPEG-21) • Personalization

10. The technological requirements • Other requirements involved • At the service provider side • Service management, discovery, integration • Smart, self-adaptive content authoring & aggregation • Context modeling • At the mediation/network layer • Monitoring • End-to-end QoS provisioning • At the end user layer • interactivity • mobility • transparent access to services • In all, digital rights management and protection

11. Service Management Registration Identification Discovery Integration Abstract end-to-end system design Digital Rights Management Content and Service Creation Service Providers Scalable coding Content Authoring Content description / representation - metadata Content Providers Content and Service Mediation Re-production Workflow Cross Media Provisioning Content Aggregation Content Adaptation Service/ Content Mediators Context Awareness Content Transcoding Context Modeling Content Distribution & Delivery QoS Management Transmission Personalization User/Terminal Profile Personalized Consumption Users PC PDA Mobile Digital TV

12. Enabling integrated cross-media entertainment services over heterogeneous networks The concepts…: Context Awareness, Content Adaptation

13. Context Awareness • Context • includes any information that characterizes an entity’s situation • Entity: person, place, object relevant to an interaction between a user and an application • is a function of time and environment • environment is a function of users, services, resources and other entities in environment • Context-aware system • when using contexts to provide relevant information or services to the user • Relevance depends on user’s task • Context-aware applications • must detect, interpret and respond to contexts • Contextualization • enables efficient content adaptable services and systems • important component of the ubiquitous/pervasive paradigm • aids in understanding the environment sufficiently

14. Context Awareness • Context-aware computing involves • collection of context information • dynamic program behavior dictated by knowledge environment • This is enabled by: • Use of profiles • Environment (network, terminal, service) • User (preferences, location, …) • represented in standardised format (MPEG-7, MPEG-21) • since a common representation format is required • to allow for interoperability in a distributed world, over heterogeneous networks and a variety of terminals • Monitoring mechanisms gathering context information • Decision engines that use context information as input to adapt content based on environment status • Usually processed at service mediation and adaptation layers, but gathered stored and interpreted at different parts of the system such as end-user terminal devices or access networks • So… • a necessary step before content adaptation

15. Context Awareness • A context profile should be • Structured • Aids in effectively filtering relevant information • Interchangeable • Among different system components • Mechanisms for transferring sub-trees (and not entire profiles) desirable • To compensate for big network delays • Composable/decomposable • To allow maintenance in a distributed way • To grasp changes in environment • Uniform • Eases interpretation • Extensible • To allow for future requirements/advances • Standardized • To allow interoperable exchange among different entities in a distributed environment

16. Context Awareness • Context-aware infrastructure • Physical and logical sensors (software entity, network components, software agents) used to collect context information related to presence, location, identity and profile of users and services/content • Typical context use • Locate services and users • Call up services according to user behavior • Provide information for service composition • Facilitate ad hoc communication mechanism between users • Adaptation of QoS to changes in environment as a result of user and service mobility • Passive • context gathering and representation • Active • smart context information delivery

17. Content Adaptation • Why adapt contents? • Most contents for viewing are for the larger screens • Creating multiple versions a burden • Even if you don’t mind, there are just too many possible devices • Different users want different things • Having one, original version is easier to manage • Content adaptation is about generating any content version from one single original version • Create once, Publish everywhere (COPE) • Universal Multimedia Access (UMA)

18. Content Adaptation • Pre-adaptation • Keeping just the original version (any other version is runtime-generated) could be slow • Pre-adaptation • to create all possible versions (at creation/production phase), and do static “selection” at runtime, or • to create just a few essential versions, and do dynamic adaptation – hence the “balance” • Dynamic adaptation • On-the-fly adaptation • Where it happens • Either at end systems (end-to-end service) • Or at intermediate network nodes (active service) • At specially designed mediators

19. Content Adaptation • Major processes involved • Access to and management of context information and content/service metadata • Decision engine to negotiate further actions • Adaptation by • Layered coding (scalable bitstreams) • Transcoding

20. Content Adaptation A Content Adaptation System Mediation Layer

21. Content Adaptation Use of context profile Information and content Instance metadata to decide upon transcoding strategies

22. Enabling integrated cross-media entertainment services over heterogeneous networks The concepts…: Scalable Coding/Transcoding

23. Transcoding introduced • Video transcoding: converting a previously compressed video signal into another one • Possibly different bit rate, frame rate, frame size or compression standard • Initially proposed for compressed video bit rate scaling • Sun et al., 1996

24. Transcoding introduced (2) • Also used for: • Video frame size conversion • spatial resolution • Video frame rate conversion • temporal resolution • Bit rate adaptation • Multipoint video combining • Error resilience

25. Transcoding for heterogeneous nets • Diversity of channel capacities • Channel capacity of the outgoing channel may be less than that of the incoming channel • Or may change over time • Distribution to users with different connections • target transmission channel conditions are generally unknown during encoding

26. Major transcoder architectures • Cascaded Pixel Domain Transcoder • Keesman et al., 1996 • Close-loop transcoder connects a standard decoder and a standard encoder together • Most straightforward method

27. Major transcoder architectures (2) • Open-Loop Transcoder • Sun et al., 1996 • Input video bitstream is first partially decoded to the DCT coefficient level • Bit rate is scaled down by cutting higher frequency coefficients or by requantizing all coefficients with a larger quantization step size

28. Major transcoder architectures (3) • DCT domain transcoder (DDT) • Zhu et al., 1999 • requantization of the DCT coefficients • error is stored in a buffer and is fed back to the requantizer to correct the requantization error introduced in previous frames • Simplifies architecture of OLT by reusing motion vectors • Merges two motion compensation loops in the CPDT into one

29. Video rate control • Accurate and dynamic control of the output bit rate according to the channel bandwidth

30. Video rate control (2) • Two different steps: • Frame-Layer Rate Control: allocate target bits for each frame according to image complexities, buffer occupancy, or a given channel bit rate • Macroblock-Layer Bit Allocation: derive the actual quantization parameter for each picture Macroblock and make the number of produced bits meet the bit target

31. Frame-Layer Rate Control • Bit budget for every frame has to be determined considering the channel bit rate and buffer occupancy • Assumption for constant bit rate (CBR) • Internet transmission cannot provide a guaranteed constant bit rate channel • wireless channels: high bit error rate and variable effective channel bit rate

32. Frame-Layer Rate Control (2) • Theoretically, compression and error protection can be performed separately and sequentially • Shannon’s separation theorem • In practical video communication systems source coding and channel coding are combined together

33. Wireless video communication • Wire line channels: signal strength is relatively constant • reception errors mainly due to additive noise • Wireless channels: reception errors mostly due to time-varying signal strength • multi-path propagation from local scatters • error bursts

34. Wireless video transmission • Delay constraints • Channel errors hurt perceptual quality of video at decoder • Evident in standard coders • variable length coding • predictive coding compression • e.g., MPEG or H.263

35. Challenge transmission error • At the encoder side: • layered coding • multiple description coding • error resilience entropy coding • Increase robustness of video stream against channel errors

36. Challenge transmission error (2) • At the decoder side: • Error concealment techniques recover lost information without relying on information from the encoder • Channel coding techniques • FEC codes may cause unnecessary overhead and bandwidth waste when channel is in good state • ARQ error control requires retransmission only during periods of poor channel conditions

37. Abstract architecture Delay = Processing Delay + Transmission Delay + Buffer Delay

38. Adaptive transcoding • Frame layer rate control based on video content • frame types • video source coding rate • scene changes

39. Frame types and source video • MPEG-1, -2 and H.26x use motion compensation • Motion compensation reduces temporal redundancy between successive frames

40. MPEG-1 and MPEG-2 • In MPEG, a GOP (Group of Pictures) contains one I-frame (intra) and several P- or B-frames (inter) in a certain pattern • I-frame: no motion compensation • motion compensation of following frames depends on I-frame quality • P-frames use the previous I- • B-frames use both the previous and successive I- or P-frames as references for motion compensation

41. H.263 • Focus on low bit rate • More popular for wireless applications • Fixed quantization parameter • Near constant visual objective quality • Varying bit rate • scene content • different frame types • Constant rate  quality degradation

42. H.263 (2) • Source bit rate can be recorded during encoding and saved as side information • unified quantization parameter in same type of frames  generated frame size indicates scene variations and motion activities • Thus, video content can be measured by the source bit rate • Used to calculate bit budget for every frame

43. Scene changes • Scene changes represent distinctive differences between adjacent video frames • Rapid motion of moving objects • Changes to different visual content • Information obtained from previously coded frames is no longer useful

44. Scene changes (2) • When scene changes happen, the first frame after scene change should be transcoded in high quality to prevent quality degradation after scene change • Scene change detection • Most MBs in the anchor frame will be encoded as INTRA blocks • INTRA mode MBs percentage can be used to detect scene changes

45. Adaptive frame layer rate control • Relaxed requirement for end-to-end delay • Allowed initial startup delay • decoder buffer smoothes delay jitters • H.263 determines frame budget by buffer occupancy and channel bandwidth • nearly constant bit budget  similar end-to-end delay for each frame

46. Adaptive frame layer rate control (2) • Adaptive methods consider scene changes, frame types and source coding rate • e.g., I-frames are transcoded as I-frames, with unified quantization parameter used for every MB, while the bit budget for B-frames will be half of that for P-frames • Scene change detected  anchor frame is transcoded as an I-frame

47. A practical scenario • Offline encoding • Initial presentation is prepared according to pre-defined scenarios • Uses pre- created static content • Online transcoding • In charge of transmission of initial presentation • Generation, encoding and transmission of real-time interactions

48. Enabling integrated cross-media entertainment services over heterogeneous networks The technologies…: Metadata & Profiling (environment, user, terminal)

49. The need for standardised metadata (MPEG-7) • Today people cannot easily create, find, edit, share, and reuse media • Computers don’t understand media content • Media is opaque and data rich • We lack structured representations • Without content representation (metadata), manipulating digital media will remain like word-processing with bitmaps • Need standardized metadata framework • Designed for video and rich media data • Human and machine readable and writable • Standardized and scalable • Integrated into media capture, archiving, editing, distribution, and reuse

50. MPEG-7 • MPEG-7 • Describing the multimedia content data that supports some degree of interpretation of the information’s meaning, which can be passed onto, or accessed by, a device or a computer code • Motivation • Create standardized multimedia description framework • Enable content-based access to and processing of multimedia information on the basis of descriptions of multimedia content and structure (metadata) • Support range of abstraction levels for metadata from low-level signal characteristics to high-level semantic information