IPTV

1. IPTV Grae Phillip Kevin Miguel Steve

2. Primary Objectives • Develop a tool that facilitates switching bitstreams in real-time given a switching time table • Research and develop algorithms that facilitate switching resolutions • Implement a GUI for IPTV and then use that GUI to facilitate running PiP and multichannel preview.

3. Outline • H264 • Format • Streams • MP4 vs. H264 • MP4 • Format • Atom Parsley • Box Details • Multi-Track MP4 • Audio and Video • Tracks at Different Resolutions • RTSP • Need • Structure • Example

4. H264 Format: H264 can encode each frame differently. P frames simply encode the differences from the previous frame. (P =Previous frame) B frames can encode the data based off of previous and future frames. (B=Bi directional) I frames, our friends, re-sync the video, basically stopping any drift, and creating random access points in the video.

5. H264 Streams: RAW 264 Data NAL units Elementary Stream vs. Transport Stream and Program Stream

6. H264 MP4 vs. H264: MP4 is a generic container file that holds H264 or other data. MP4s hold separate streams for audio and video. Quick Time includes previews, posters, and various other extras in their MP4s.

7. Outline • H264 • Format • Streams • MP4 vs. H264 • MP4 • Format • Atom Parsley • Box Details • Multi-Track MP4 • Audio and Video • Tracks at Different Resolutions • RTSP • Need • Structure • Example

8. MP4 Format: Boxes (or Atoms) Hierarchy

9. MP4 Atom Parsley

10. MP4 Box Details • Size • First 32 bits or 4 byte integer • FourCC • Next four bytes as ASCII characters • Other Data • Size, Name, then Data or Inner Boxes

11. MP4

12. Outline • H264 • Format • Streams • MP4 vs. H264 • MP4 • Format • Atom Parsley • Box Details • Multi-Track MP4 • Audio and Video • Tracks at Different Resolutions • RTSP • Need • Structure • Example

13. Multi-Track MP4 • Audio and Video • Tracks at Different Resolutions • Strengths: • Demuxer can enable or disable tracks as we wish. • Allows switched with I frame break. • Less complex system – Only one mp4 file • Weaknesses • Channel switching requires more than one file anyway.

14. Outline • H264 • Format • Streams • MP4 vs. H264 • MP4 • Format • Atom Parsley • Box Details • Multi-Track MP4 • Audio and Video • Tracks at Different Resolutions • RTSP • Need • Structure • Example

15. RTSP Streaming: • Need a way for the client to request the lower or higher bit stream. Structure: • Similar to HTML • PLAY rtsp://<server>/sound.mp4 RTSP\1.0

16. RTSP Testing Programs: • rstp-lite • Perl program • openRTSP • Telnet • Most useful so far.

17. RTSPExample Request: OPTIONS * RTSP\1.0 Cseq:1 Response: RTSP/1.0 200 OK Server: DSS/5.5.5 (Build/489.16; Platform/Linux; Release/Darwin; state/beta; ) Cseq: 1 Public: DESCRIBE, SETUP, TEARDOWN, PLAY, PAUSE, OPTIONS, ANNOUNCE, RECORD

18. RTSPExample DESCRIBE rtsp://128.206.20.146/sample_100kbit.mp4 RTSP\1.0 Cseq:2 Accept: application/sdp, application/rtsl, application/mheg RTSP/1.0 200 OK Server: DSS/5.5.5 (Build/489.16; Platform/Linux; Release/Darwin; state/beta; ) Cseq: 2 Last-Modified: Mon, 09 Jun 2008 18:59:39 GMT Cache-Control: must-revalidate Content-length: 1208 Date: Fri, 27 Jun 2008 20:34:06 GMT Expires: Fri, 27 Jun 2008 20:34:06 GMT Content-Type: application/sdp x-Accept-Retransmit: our-retransmit x-Accept-Dynamic-Rate: 1 Content-Base: rtsp://128.206.20.146/sample_100kbit.mp4/

19. RTSPExample v=0 o=StreamingServer 3423587645 1213037979000 IN IP4 128.206.20.146 s=/sample_100kbit.mp4 u=http:/// e=admin@ c=IN IP4 0.0.0.0 a=control:* a=mpeg4-iod:"data:application/mpeg4-iod;base64,AoJrAE///w/z/wOBdgABQNhkYXRhOmFwcGxpY2F0aW9uL21wZWc0LW9kLWF1O2Jhc2U2NCxBWUVDQVV3Rkh3TklBTWtnQUdVRUx5QVJBRzNkQUFFaytBQUJKUGdGSUFBQUFiRHpBQUFCdFE3Z1FNRFBBQUFCQUFBQUFTQUFoRUQ2S0RBZzhxSWZCaEFBUkFBQUFsZ0FBQUFBSUFBQUFBQURBVElDbndNdUFHVUFCSUNBZ0JSQUZRQVlBQUFBVGlBQUFFNGdCWUNBZ0FJVmtBWVFBRVFBQUI5QUFBQWZRQ0FnQUFBQUF3PT0EDQEFAADIAAAAAAAAAAAGCQEAAAAAAAAAAANpAAJARmRhdGE6YXBwbGljYXRpb24vbXBlZzQtYmlmcy1hdTtiYXNlNjQsd0JBU2daTUNvRmNtRUVIOEFBQUIvQUFBQkVLQ0tDbjQEEgINAABkAAAAAAAAAAAFAwAAYAYJAQAAAAAAAAAA" a=isma-compliance:1,1.0,1 a=range:npt=0- 70.00000 m=video 0 RTP/AVP 96 m=audio 0 RTP/AVP 97

21. mp4_track_t structure

22. MP4 Demux function

23. Current problems Problem: The current way of bitstream switching does not compensate for resolution change Possible Solution: It may be as simple as reassigning the width and height parameters of the elementary stream structure when we switch tracks Problem: VLC’s h264 packetizer does not work for streaming Solution: Research the 264 syntax and fix up the current packetizer so that it DOES work Problem: The added code is not generic enough Solution: Make it more generic and malleable (like all of VLC)

24. Interpolation Algorithms • Bicubic • Lanczos • Catmull-Rom spline • New Edge-Directed Interpolation (NEDI) • Spline (mostly towards cubic spline)

25. Catmull-Rom Spline

26. New Edge-Directed Interpolation • Defines non-edge and edge areas separately. • Processes the non-edge areas with a bilinear algorithm. • Edge areas get processed with a covariance based adaptive algorithm.

27. What I have done • Week 1 • Researched the various projects • Week 2 • Researched ipTV • Week 3 • Researched VLC and streaming video • Researched interpolation algorithms • Week 4 • Found documentation on the JM encoder • Examined the code for JM encoder • Looked for where the encoding process begins so that an interpolation algorithm can be added to allow in-loop interpolation of video files • Week 5 • Continued examination of JM encoder

28. Plans • Week 6 and beyond • I plan to continue examining the JM encoder • Once a good location for an interpolation algorithm is found I will add various interpolation algorithms to determine which will yield the best video quality

29. The JM Encoder JM so far in my examination I have basically just followed the program through a test encoding process. In my examination I have kept track of all the function calls and what *.c files the functions are in along with the line numbers they are on. I have actually just gotten through the setup steps

30. JM functions JM has very complicated code even though I have spent 2 weeks looking at it I understand very little about what it is actually doing. All I can conclude about it is that it has all been setup functions so far This includes many repetitive calls to memory allocation functions in memalloc.c that use multi-dimensional arrays

31. JM Functions Each function takes the variables from the lower levels of the array and ignores the highest dimension until it has 2 dimensions then it does its calculations and goes to the 3rd dimension and repeats with 2 and 3 then the 4th dimension uses 3 and 4 and so on.

32. More JM Functions • parset.c calls functions in vlc.c that write to the bit stream. • The functions ue_v(...), se_1(...), u_1, and u_v(...) are called many many times and all call the same function that write to the bit stream.

33. VLC GUI for PiP and Multi-channel preview Steven Steffen IPTV group REU 2008

34. Project so far • Currently I have looked into VLC’s web interface for PiP and multi-channel preview. • Looked at the command line executions for PiP and Multi-channel preview • Looked into XML and the VLC Skin Editor. • Began building a C++ forms program to achieve the desired PiP, multi-channel, and any other user accessible function needed.

35. VLC Web Interface VLC’s web interface is essentially a GUI for VLC that is based from an internet browser. To get to this interface, first go to preferences -> interfaces and turn on the http interface option. Then go to an internet browser and type in the URL http://localhost:8080/ The use of this interface is nearly identical to the normal GUI for VLC, with one major difference, it has the Mosaic Wizard.

36. Mosaic Wizard The mosaic wizard is the only method besides a long stream of command line text to get PiP, and even then, it has a few flaws. One thing is, the pictures can only be blocked in an edge to edge square, they cant be spread out, or even just diagonal to each other. Therefore, one cant get for instance, one picture in each corner of the larger one, because they aren’t directly adjacent to one another.

37. The Problem Therefore, another means of achieving this is needed, as well as an environment for the rest of the group to implement their discoveries if needed. • The problem is that although VLC supplies the means to achieve PiP and multi-channel preview, it is very sluggish, and not very user friendly.

38. The current solution Some little problems The main problem I’ve run into is getting the system() function to work along with the VC++ textbox to button coding. Another thing I noticed before I got too far along was that this algorithm will open a separate instance of VLC, which is fine if the program is run by itself, but if I add is as a button to an already made skin, I will need to look into making it play in the same instance. • My current solutions, and what I’m currently working on, is a C++ forms program that essentially is another wizard with predefined formats for the placement of the separate images. Yes it wont be quite as customizable, but it will be much more user friendly

39. The Goals The main goal Further Goals As time permits, I would also like to achieve the following Make the program appear in an already made VLC skin as an added button Make the button play it on the same VLC instance Edit the program for added customization of the format. • The immediate goal is to just get a basic interface that implements the desired PiP action. • This will at first, probably just involve a separate program that calls VLC with the desired attributes.