Subrahmanya M V (Under the guidance of Dr. Rao and Dr.Jin-soo Kim)

1. Subrahmanya M V (Under the guidance of Dr. Rao and Dr.Jin-soo Kim)

2. Contents • Traditional Video coding • Wyner-Ziv video coding • Applications • Wyner-Ziv encoder • Key frame encoding • WZ frame encoding • Wyner-Ziv decoder • Key frame decoding • WZ frame decoding

3. Contents( contd.) • Side Information Generation • Inter-prediction • Intra-prediction • Motion estimation (ME) • Full-pixel • Half-pixel • Quarter-pixel • Motion compensation (MC) • Luma • Chroma • Bi-directional interpolation • Intra-prediction

4. Contents( contd.) • Implementation • Testing • Results • SI prediction schemes • ME block sizes • ME search ranges • Key frame distances • Bit rates • Conclusions • References

5. Traditional Video coding • Source coding • Encoder removes temporal and spatial redundancies • Intra frame encoding • Inter frame encoding: Computationally intensive • Complex Encoder + Simple decoder • [Dr. Jin-soo Kim]

6. Wyner-Ziv video coding • Source coding + Channel coding • Decoding with side-information(SI) • Slepian-Wolf coding: Lossless • Wyner-Ziv coding: Lossy • Simple encoder + Complex decoder • [Dr. Jin-soo Kim]

7. Applications • Light encoder and light decoder [20] • Wireless low power video surveillance • Sensor network • Multi-view image acquisition • Networked camcoders • [Dr. Jin-soo Kim]

8. Wyner-Ziv Encoder • Input YUV 420 • Output: Bitstream • Encoding Key frame + WZ frame

9. Key frame encoding • H.264 Intra frame coding • Intra prediction • Integer Transform • Quantization • Entropy Coding • [M. Lee, A. Moore]

10. Intra prediction • Predicted from top, left, top left and top right neighbors • 4x4 intra prediction • Nine directional modes [T. Wiegand and G.J Sullivan] [J. S. Park and H. J. Song]

11. Key frame encoding • Integer Transform • Quantization • Element by element Multiplication • Entropy Coding using • Exp-Golomb codes

12. WZ frame encoding • Residual signal Generation • [Dr. Jin-soo Kim] Residual Signal

13. WZ frame encoding(contd.) • Residual bit plane extraction

14. WZ frame encoding(contd.) • Low Density Parity Check(LDPC) encoding • Obtain parity bits for the message bits • Message bits(90%) + Parity bits(10%) • Only parity bits are sent to achieve compression

15. Wyner-Ziv Decoder • Decoding Key frame + WZ frame Key Frame WZ Frame [E. Peixoto, R. L. de Queiroz, and D. Mukherjee]

16. Wyner-Ziv Decoder( Contd. ) • Key frame decoding • Intra prediction • Entropy decoding • Inverse Quantization • Inverse Integer transform

17. Key frame decoding • Inverse Integer Transform • Inverse Quantization • Element by Element Multiplication • Entropy decoding • Exp-Golomb codes are decoded using prefix tables

18. Wyner-Ziv Decoder( Contd. ) • WZ frame decoding • SI generation: Prediction to obtain the message bits • LDPC decoding: Correct prediction errors using parity bits • Message contains a pair of values [qij(0), qij(1)] or [rij(0), rij(1)] – Signify the amount of belief that yiis a ”0” or a ”1”. • Initial belief is based on message bits predicted and parity bits sent • Belief is iteratively updated and new message bits are obtained • Iteration proceeds till fixed iteration count or message bits remain unchanged in an iteration

19. Side Information Generation • Motion estimation • Estimate motion of a block between two key frames to obtain motion vectors( MVx, MVy ) for each block • Interpolate motion vectors to obtain motion between key frame and WZ frame • Motion Compensation • Interpolate key frame based on motion vectors to obtain MC prediction for WZ frame • Intra prediction • Prediction for blocks which are absent in key frames

20. Motion Estimation • Forward ME between key frames – MVF • Backward ME between key frames - MVB • Derive MVs for WZ frame using Key frame MVS [E. Peixoto, R. L. de Queiroz, and D. Mukherjee 2008]

21. Higher key frame distance • For larger key frame distance hierarchical structure can be used [A. Aaron, E. Setton and B. Girod 2003]

22. SAD based ME • Obtain absolute difference between each pixel in the block to be searched and the reference block • Best match is a block with least sum of absolute differences (W+2SR) X (H+2SR)

23. Sub-pixel ME • Motion vector is estimated up to quarter-pixel units • Interpolation of reference is done to obtain half-pixel and quarter-pixel units • Half-pixel positions are obtained using five tap filter • Quarter pixels are obtained using bilinear interpolation

24. Half-pixel motion estimation • H33 = [F13 + -5 * F23 + 20 * F33 + 20 * F43 + -5 * F53 + F63 + 15] >> 5 • G33 = [F31 + -5 * F32 + 20 * F33 + 20 * F34 + -5 * F35 + F36 + 15] >> 5 • D33 = [H31 + -5 * H32 + 20 * H33 + 20 * H34 + -5 * H35 + H36 + 15] >> 5

25. quarter-pixel motion estimation q1 = ( F33 + G33 + 1 ) >> 1 q2 = ( G33 + F34 + 1 ) >> 1 q3 = ( F33 + H33 + 1 ) >> 1 q4 = ( H33 + G33 + 1 ) >> 1 q5 = (G33 + D33 + 1 ) >> 1 q6 = ( G33 + H34 + 1 ) >> 1 q7 = ( H33 + D33 + 1 ) >> 1 q8 = ( D33 + H34 + 1 ) >> 1 q9 = ( H33 + F43 + 1 ) >> 1 q10 = ( H33 + G43 + 1 ) >> 1 q11 = ( D33 + G43 + 1 ) >> 1 q12 = ( G43 + H34 + 1 ) >> 1

26. Motion Compensation (Luma) • Obtain a block from key frame with offset specified by motion vector

27. Motion Compensation (Chroma) • Chroma MV = Luma MV / 2 • Obtain A, B, C, D based on integer part of MV • Floor(MV/8) • Predict based on fractional part of MV • (MV & 7)

28. Bi-directional interpolation SI frame = (Forward prediction Frame + Backward Prediction frame + 1)/2

29. Intra prediction • 4x4 prediction • Prediction mode of co-located block from previous key frame

30. Implementation • Encoder • Key frame encoding is done using JM encoder • WZ frame decoding is implemented in ‘C’ • Residual generation • Quantization • Bit plane extraction • LDPC encoder • Decoder • Key frame is decoded using JM decoder • WZ frame decoding is implemented in ‘C’ • SI generation • LDPC decoder • De-quantization • Reconstruction • LDPC encoder and decoder are implemented by Dr. Jin-soo Kim [4]

31. Testing • Comparison with traditional coding • H.264 is used for comparison • Generate H.264 reconstructed sequence • Encode using H.264 encoder • Decode using H.264 decoder to obtain reconstructed YUV sequence • Generate Wyner-Ziv reconstructed sequence • Encode using Wyner-Ziv encoder • Decode using Wyner-Ziv decoder to obtain reconstructed YUV sequence • Compare two sequences using PSNR and SSIM • Significance of different parameters in Wyner-Ziv coding • SI Prediction method • Key frame distance • ME search range • ME block size

32. SI prediction schemes(Sample reconstructed images) • bus( CIF) [Average, Full pixel ME, Half-pixel ME, Quarter-pixel ME, Bi-prediction, Intra-prediction combined]

33. SI prediction schemes( Plots)

34. ME block sizes(Sample reconstructed images) • coastgaurd (CIF) [16x16, 8x8, 4x4]

35. ME block sizes( Plots)

36. ME search ranges (Sample reconstructed images) • stefan (CIF) [16, 8, 4]

37. ME search ranges( Plots)

38. Key frame distances (Sample reconstructed images) • garden (CIF) [1, 2, 3, 4, 5]

39. Key frame distances( Plots)

40. Comparison with H.264 (Sample reconstructed images)

41. Comparison with H.264( Plots)

42. Conclusions • H.264 based SI generation based on quarter-pixel ME performs better compared to previously done H.263which is based on half-pixel ME[1] . • ME block size 16x16 is ideal • Higher ME search range produces better results • At higher key frame distance quality drops sharply • Traditional video coding has better R-D performance

43. Future work • Wyner-Ziv to H.264 transcoder • Light encoder + light decoder • Inter and intra prediction done in Wyner-Ziv decoder can be reused for the H.264 encoder

44. Accepted conference paper SubrahmanyaVenkatrav and Dr. K. R. Rao, “Side information generation in Wyner-Ziv decoder”, The 7th International Joint Conference on Computer Science and Software Engineering May 13-14, 2010 Bangkok, Thailand

45. References • E. Peixoto, R. L. de Queiroz, and D. Mukherjee, “Mobile video communications using a Wyner-Ziv transcoder,” Proc. SPIE 6822, VCIP, 68220R Jan. 2008. • A. Aaron, E. Setton and B. Girod, "Towards practical Wyner-Ziv coding of video," Proceedings. IEEE International Conference on Image Processing, 2003. ICIP 2003., vol.3, pp. III-869-872, 14-17 Sept. 2003. • K. R. Rao and J. J. Hwang, Techniques and standards for image, video, and audio coding, Prentice Hall PTR, 1996. • Jin-Soo Kim, "Brief overview of Wyner-Ziv CODEC" (Private Communication) • A. Aaron, D. Varodayan, and B. Girod, “Wyner-Ziv residual coding of video,” Proc. International Picture Coding Symposium, Beijing, P. R. China , April 2006. • T. Wiegand and G.J Sullivan, “The H.264/AVC video coding standard”, IEEE SP Magazine, vol. 24, pp. 148-153, March 2007. • G. J. Sullivan, P. Topiwala, and A. Luthra, "The H.264/AVC advanced video coding standard: Overview and introduction to the fidelity range extensions", SPIE Conf. on applications of digital image processing XXVII, vol. 5558, pp. 53-74, Aug. 2004. • S.K. Kwon, A. Tamhankar, and K.R. Rao “Overview of H.264/MPEG-4 Part 10” J. VCIR, Vol. 17, pp. 186-216, April 2006, Special Issue on “Emerging H.264/AVC Video Coding Standard,”.

46. References(Contd) • A. Wyner and J. Ziv, "The rate-distortion function for source coding with side information at the decoder,"  IEEE Trans., Information Theory, vol.22, pp. 1-10, Jan 1976. • D. Slepian and J. Wolf, “Noiseless coding of correlated information sources,” IEEE Trans. on Information Theory Vol. 19, pp. 471–480, July 1973. • D. Varodayan, A. Aaron and B. Girod, "Rate-adaptive distributed source coding using low-density parity-check codes,"  Conference Record of the Thirty-Ninth Asilomar Conference on Signals, Systems and Computers, pp. 1203-1207, Oct. 28 – Nov. 1, 2005. • Z. Li and E.J. Delp, "Wyner-Ziv video side estimator: conventional motion search methods revisited," IEEE International Conference on Image Processing, 2005. ICIP 2005, vol.1, pp. I-825-828, 11-14 Sept. 2005. • L. Liu and E. J. Delp, "Wyner-Ziv video coding using LDPC codes," Proceedings of the 7th Nordic Signal Processing Symposium, 2006. NORSIG 2006. •  D. Kubasov, K. Lajnef and C. Guillemot, "A hybrid encoder/decoder rate control for Wyner-Ziv video coding with a feedback channel," IEEE 9th Workshop on Multimedia Signal Processing, 2007. MMSP 2007., pp.251-254, 1-3 Oct. 2007. • C. Brites and F. Pereira, "Encoder rate control for transform domain Wyner-Ziv video coding," IEEE International Conference on Image Processing, 2007. ICIP 2007., vol.2, pp.II -5-II -8, 16-19 Sept. 2007 • A. Roca, et al, "Rate control algorithm for pixel-domain Wyner-Ziv video coding ," Proc. SPIE, vol. 6822, 68221T (2008).

47. References(Contd) • D. Mukherjee, “Optimal parameter choice for Wyner-Ziv coding of Laplacian sources with decoder side information,” HP Labs Technical Report HPL-2007-34, 2007. • Z. Wang, et al, "Image quality assessment: From error visibility to structural similarity," IEEE Trans., Image Processing, vol.13, pp. 600-612, April 2004 • I. Richardson, “The H.264 advanced video compression standard,” Hoboken, NJ: Wiley, 2010. • D. Rebollo-Monedero, S. Rane, A. Aaron and B. Girod, "High-rate quantization and transform coding with side information at the decoder," EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding. • S.-K Kwon, A. Tamhankar and K.R. Rao, ‘Overview of H.264 / MPEG-4 Part 10” ISME, Hong Kong, Oct. 2004. • JVT documents ftp://standards.polycom.com http://ftp3.itu.ch/av-arch/jvt-site/ • M. Lee and A. Moore, “H.264 Encoder Design“, Group 3, May 17, 2006 • J. S. Park and H. J. Song, “Selective Intra Prediction Mode Decision for H.264 /AVC Encoders”, World Academy of Science, Engineering and Technology 13 2006. • B. M.J. Leiner, “LDPC Codes – a brief Tutorial”, April 8, 2005

48. References(Contd) • G. Cote, B. Erol, M. Gallant, and F. Kossentini, “H.263+: Video coding at low bit rates,” IEEE Trans. Circuits and Systems for Video Technology ,Vol.8, pp. 849–866, Nov. 1998 • (2008, August) H.264/avc JM reference software. Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG. [Online]. Available: http://iphome.hhi.de/suehring/tml/