CMU - March 1, 2001

1. CMU - March 1, 2001 Video over the Internet

2. Internet video Basic types & feasibility Delivery impairments & remedies Avoiding congestion RTP – Real-time Transport Protocol Sample systems from AT&T Labs - Research AT&T VideoTalks™ SALSA: System for Audiovisual Live Services & Applications Internet TV Outline

3. Video as a part of information-on-demand Streaming Start the playback before receiving the entire file Delayed start to buffer some data VCR type controls (Pause, FF, Rewind, Single Step, etc.) Offline coding Live Video (Program distribution: Internet TV) Multicasting is possible One stream for multiple users Requires real-time coding Interactive Video (Internet Video Phone) Delay sensitive (no streaming) Requires Quality-of-Service (QoS) Basic types of the Internet Video

4. Fundamental Questions • Can the Internet become the global delivery vehicle for all video applications? • What, if any, functionality need to be added to the Internet in order to have an affirmative answer to the first question?

5. Delivery Impairments & Remedies • What bad things may happen to a packet • delivered over the Internet? • Lost • Delayed • Delivered out-of-order • Replicated • May have bit errors Handled by transport protocols

6. Video transport characteristics loss tolerance Video Audio Data delay tolerance + error propagation

7. Packet loss • Inevitable under “best effort” delivery • 1 – 5% packet loss rates are common • 30% or more is possible • time varying • difficult to characterize • difficult to measure • Loss concealment is a must • Can do much better if we incorporate network properties in codec design

8. Application Layer Framing (ALF) - Packetization consistent with the bitstream structure. No chop & ship! Contain the extent of a packet loss to the content of the lost packet Handle fragmentation intelligently Make each packet self decodable Include the necessary info for decoding in the packet Include info about previous packets if needed multiple description coding previous packet priority information Packet Loss Remedies (I)

9. Proper use of the “intra” information Use key frames to break long prediction chains Be careful about the increase in the instantaneous bit rate Use sub image level intra updates based on accumulated probability of loss thresholds Use multiple prediction threads (Video Redundancy Coding – H.263+) Packet Loss Remedies (II) prediction chains I I Frame number: 1 2 3 4 5 6 7 8 9 10 11 12 13 14

10. Split the encoded bitstream into two or more parts based on: Blur level –– Frequency domain Resolution –– Spatial Quantization level –– SNR Frame rate –– Temporal Layered Coding

11. Packet Loss Remedies (III) - Layered coding • Layered coding can be used to identify the high priority (HP) parts of an encoded bitstream. • HP data may or may not be used to generate video by itself • Example: Fine Granular Scalability (FGS in MPEG4) • HP data can be transmitted with priority Enhancement layers Predictive coding layer

12. Prioritized Transport • No QoS guarantees • Forward Error Correction(FEC) • the simplest form is redundant transmission • bandwidth increase problem • Automatic Repeat reQuest (ARQ) • based on acknowledgements (ACK/NACK) • hard to use with multicast (reliable multicast) • delay problem (roundtrip & loss of ACK/NACK) • Guaranteed QoS • Differentiated services (DiffServ) • Integrated services • more expensive than best effort

13. Bandwidth remains to be a limited resource needed by many… A value structure for the QoS is needed Global implementations require laws and international agreements! Intranet applications are already available while large scale deployment is yet to be seen QoS Considerations

14. Delay • Two components: • Propagation (fixed) • Queuing (variable,delay jitter) • Large delays are equivalent to loss • For smaller delays, use a Play-out buffer • Initial size determination (determines the initial waiting time) • Adaptation • display rate modifications

15. Two protectors of the current Internet Low-bandwidth access Built-in TCP (flow control) Future Adaptive applications Avoiding Congestion quality rate bottleneck bandwidth

16. Rate adaptation • Bottleneck bandwidth: • bursty, hard to measure • not the same for everyone in a multicast • Alternatives: • use mean opinion • receiver driven layered multicast for • heterogeneous reception bandwidth • varying computing power destinations experiencing different bottleneck bandwidths / varying computing powers source layered video

17. RTP Real-time Transport Protocol - RTP Application Presentation Session Transport Network Data Link Physical Application Transport (UDP, TCP) Internet (IP) Network Interface Hardware O.S. boundary high-level protocol boundary ISO’s 7-layer reference model TCP/IP Internet layering model

18. IP Header 20 bytes RTP: An implementation of ALF • Basic services provided by RTP • Payload type identification • Sequence numbering • Time stamping • Delivery monitoring ( through associated protocol: RTCP) UDP Header RTP Header Payload 8 bytes 20 bytes ~3% overhead for ethernet, (can be reduced by header compression) • Appropriate loss resilience techniques are implemented using specific “payload formats”

19. Non-interactive applications are feasible now through intelligent implementations Interactive applications require QoS support but not for their entire video bandwidth The Internet’s rich protocol set and extendible architecture facilitates rapid application development and deployment The Internet as a Global Video Network

20. AT&T Research comprehensive conferencing system VT Interactive - high-quality multimedia interaction VT Desktop - multimedia presentations on desk-top VTJukebox - multimedia presentation recorder/player VTonDemand - VoD with indexing VTLite - low-bandwidth access to VideoTalks System Example: AT&T VideoTalks™

21. A VideoTalks Conference Room sending receiving

22. Echo Canceller Voice Lift Speakers Program Speakers Viewgraph Codec Presenter Codec Audience Codec Desktop Codec Conference room equipment set up Presenter Microphone Presenter Display Presenter Camera Audience Microphones Mixer CD Player DAT Player Audio Video Network VCR Audience Display Audience Cameras Video Src Select Ethernet Document Camera Video Src Select Viewgraph Display PC VGA / NTSC Slide Camera Pointer

23. NetVG Ethernet Network Interface Video Capture VGA Interface Document Camera Projector Mouse Interface Pointer PC Mouse Display

24. iVToD - Indexed presentations on the desktop • Viewgraphs based indices for on demand playback of a presentation • derived automatically • displayed on a web browser • presentation starts from the selected viewgraph

25. iVToD Demo

26. System Example: SALSA Enabling two way interactive video services using the enhanced upstream bandwidth provided by the multiplexed-fiber/passive-coax technology of CATV plus guaranteed QoS over IP: System for Audio/Visual Live Services and Applications

27. Videophone for family and friends video dating, chat Personal TV studio at home Virtual gatherings, virtual town hall Video messaging Interactive shopping: personal tailoring/fashion home design/interior decorator customer service auctions/garage sales Monitoring/surveillance: babysitter camera elderly citizens’ watch remote medical monitoring and diagnostics Distance learning, remote tutoring Small business monitoring, video commuting, remote interviewing Remote video archiving, video clip-art library Interactive participation in TV game shows home video contests Services that can be enabled include:

28. Modifications on the head-end software and hardware are difficult and should be avoided if at all possible. Set top boxes (STBs) that will be used in the near future do not have enough resources for an effective video codec implementation and transport over the Internet. Enabling upstream video on CATV design alternatives • Additional hardware to be inserted into the STB to help with video coding: no standards available, immature technology • An external encoded video source connected to the STB through USB or IEEE 1394 (Firewire): expensive & limited equipment, packetization implementation on the STB is non-trivial • An external PC connected to the STB: hard to use, unstable • An STB adjunct box that implements the video codec functionalities: a “component” solution

29. Architectural Elements: Based on an add-on component for a set-top box (STB) Software-only codecs field upgradable packet loss resilient coding for the Internet transport layered coding for economical use of QoS (optimal split between guaranteed QoS and best effort bandwidth) User Interface (UI) is implemented on the STB All signaling and communications is IP based Can communicate with a PC based solution Multipoint ready Advantages: Easy to use Stable Low cost High quality audio and video SALSA

30. SALSALayered Coding: Temporal with Reference Picture Selection (H.263++) Best effort ~300 kb/s P B P P B P … … I P P QoS ~100 kb/s Frame: k k+1 k+2 … k+m-1 k+m k+m+1 … k+2m …

31. IP Network supporting Guaranteed QoS SALSATransport QoS Frames | | | SALSA Layered Encoder ||||||||||| ||||||||||| Best Effort Frames QoS Frames | | SALSA Layered Encoder CMTS ||||||| ||||||||||| Docsis 1.1 Best Effort Frames QoS Frames | | SALSA Layered Encoder || |||||||||||| ||||||| Best Effort Frames QoS frames from different encoders should not align

32. STB Main Screen Featuring VideoPhone

33. Call set up

34. SALSAprototype on the TV set

35. SALSADemo Packet Loss Simulator Camera / Microphone USB Ethernet A/V 10baseT Hub A/V

36. SALSADemo

37. MovieMail Webmail server Moviemail server Video jukebox Layered A/V stream • Stream media to jukebox while recording. • Store media with URL reference. • Attach URL to e-mail message. • Stream media to recipient upon request. salsa dct5000 • Browser-based client • Distributed IP Architecture • Cross platform

38. MovieMail Demo

39. Micro-channels Channels for special interests Everyone can have their own studio SBTV: Searchable, Browsable TV What information you want, when you want it Agents for program selection ... TV of Tomorrow

40. SBTV