VIC Development

1. VIC Development National Center of High-performance Computing Barz Hsu (barz@nchc.org.tw) 15 January 2007

2. Start Point • Intend to develop high visual quality video tool • Based on AccessGrid vic-2.8ucl1.1.3-3 • Initial release of NCHC-VIC with • MPEG4 codec • Up to NTSC/PAL resolution • High-efficient render • Fullscreen mode • Linux version only

3. Timeline • Develop NCHC-VIC from January 2004 to August 2005 • Develop GStreamer-based media tools from June to August 2006 • Merge NCHC-VIC to UCL code base from September to December 2006

4. Outlines • Review NCHC-VIC • Porting Issues • Demos • GStreamer based new media tools

5. NCHC-VIC Features • Codec • MPEG4 • H.264 • Render • Xvideo • Swscale • Fullscreen • Postprocess • Deinterlace • Deblocking • Colorspace conversion

6. MPEG4 • FFmpeg encoder/decoder • ME (motion estimation) • EPZS algorithm • QPEL • CBR (constant bit rate) • GOP (group of picture): 15 • Low latency • No B-frame • rtp_callback • Old_latency = encoding_latency + tramsmission_latency • New_latency =~ encoding_latency/5 + tramsmission_latency • Low CPU usage

7. MPEG4 (cont’d) • IETF RFC 3640 • RTP payload type 45 • Rearrangement RTP packet buffer for only one frame • Error resilience/concealment • Compatible with VP • Handy to use other codec such as • MPEG2, H.261, H.263, H.263+, DV, MJPEG, Theora

8. H.264 • X264 encoder • Turn off CABAC • save computation up to 30%, increasing coding bit rate around 15% • Inter-ME X264_ANALYSE_PSUB16x16 • RC_METHOD X264_RC_ABR • i_direct_mv_pred X264_DIRECT_PRED_NONE • Deblocking filter • No B-frame • GOP is 20~50 frames

9. H.264 (cont’d) • FFmpeg encoder • FFmpeg decoder • Each RTP packet is a collection of NAL units • IETF RFC 3894? • Compatibility? • Quicktime、VLC、inSORS • Contributed by Champ Yeh and Barz Hsu from NCHC

10. Advanced UI • Contributed by Morris Lin from NCHC

16. Accelerated Render • XVideo+Shared memory • Software scaling • CPU capability runtime detection • Win32/Win64, Linux, MacOSX • Colorspace Conversion

17. SWSCALE: Software Scaling • Libswscale • Shipping with FFmpeg and MPlayer • Arbitrary resizing • Bilinear interpolation • Colorspace conversion • YUV420 to RGB32/RGB24/RGB16 for render • YUYV to YUV420 for grabber • CPU multimedia instructions • Intel: MMX/MMX2/SSE/SSE2 • AMD: 3D Now/3D Now2 • PowerPC: Altivec • Runtime CPU capability detection • Speed up to 10X than pure C code

18. Xvideo Extension: Graphic Accelerated Render • XFree86 X11 Extension • Supported by Nvidia and ATI graphic card • Direct YUV420 render • No colorspace conversion needed • Perform image scaling and colorspace conversion by graphics • Most efficient way • Up to 40~100 videos at same time • Automatically use swscale if xvideo no longer available

19. Fullscreen • Boardless • Light weight If graphics accelerated • Dual head is fine • Only for Windows and Linux

20. Fullscreen

21. Postprocess • Provide better visual quality • Remove coding artifact by specific encoders • Deinterlace • Deblocking • Dering • Automatic level • Use higher level of postprocessing if CPU usage is not full

22. Linux Port • Nothing special! • File size of statically linking binary is around 10MB • Most robust and efficient version • V4l grabber bugfix

23. Windows Port • Visual Studio 2003 • Inline assembly syntax different from GCC • AT&T v.s. INTEL • Create DLLs and LIBs (import library) under Mingw/Cygwin • ffmpeg, x264, libswscale, libpostproc • Is it possbile to use sttatical linking? • S-video, Composite input support • correct framerate for win32 DS grabber

24. MacOSX Port • Merge AG Aqua branch • http://www.vislab.uq.edu.au/research/accessgrid/software/aquavic/ • Aqua and Quicktime framework • Without X11 • Use Aqua Tcl/Tk 8.4 shipped with MacOSX • Native look • Grabber-macosx • iSight, QuickCam support • More robust than AG Aqua branch

25. MacOSX Port (cont’d)

26. Compilation Steps (Linux) • Check out source code • svn co https://frostie.cs.ucl.ac.uk/repos/mmedia/vic/branches/mpeg4 • Configuration • ./configure • Check out FFmpeg with revision 7110 (Dec, 2007) • svn co -r 7110 svn://svn.mplayerhq.hu/ffmpeg/trunk ffmpeg • ./configure --enable-gpl --enable-swscaler --enable-pp • Check out X264 with revision 600 (Dec, 2007) • svn co -r 600 svn://svn.videolan.org/x264/trunk x264 • Options • --with-ffmpeginc • --with-ffmpeglib • --with-x264inc • --with-x264lib • Build • make

27. Compilation Steps (MacOSX) • Install TclTkAquaBI for Tcl/Tk 8.4.10 • http://tcltkaqua.sourceforge.net/ • Configuration and Building Steps are same as Linux platform

28. Compilation Steps (Windows) • Configuration • ./configure • Check out FFmpeg with revision 7110 (Dec, 2007) • svn co -r 7110 svn://svn.mplayerhq.hu/ffmpeg/trunk ffmpeg • ./configure --enable-mingw32 --enable-memalign-hack --enable-swscaler --enable-gpl --enable-shared --disable-static • Check out X264 with revision 600 (Dec, 2007) • svn co -r 600 svn://svn.videolan.org/x264/trunk x264 • ./configure --enable-shared • Patch Makefile for generating DLL • add the 3-6 lines in SONAME targe • $(SONAME): .depend $(OBJS) $(OBJASM) • $(CC) -shared -o $@ $(OBJS) $(OBJASM) -Wl,-soname,$(SONAME) $(LDFLAGS) • ifeq ($(SYS),MINGW) • $(CC) -shared -Wl,--output-def,libx264.def -o libx264.dll $(OBJS) $(OBJASM) lib /machine:i386 /def:libx264.def • endif

29. Compilation Steps (Windows) (cont’d) • Copy LIBs for VS2003 • avcodec-51.lib avutil-49.lib libx264.lib postproc-51.lib swscale-0.lib • Copy DLLs to the path of excuteable • avcodec-51.dll avutil-49.dll libx264.dll postproc-51.dll swscale-0.dll

30. Tcl/Tk 8.4 • Multi-threading • Tcl_init.tcl and tk_init.tcl files required • Further work needed to embed the init scripts to binary • It’s critical to call Tk_FindExecuteable() to locate init scripts for Windows platform

31. Open Issues?