Block Loss Recovery Techniques for Image and Video Communications

1. Block Loss Recovery Techniques for Image and Video Communications Jiho Park The Computational Intelligence Applications (CIA) Lab. Department of Electrical Engineering University of Washington May 21, 2002

2. Outline • Introduction • Background for Block Recovery • Spatial Block Loss Recovery Technique • Temporal Block Loss Recovery Technique • Interpolation based Coding and Data Priority Setting • Conclusion & Future Work

3. Image and Video Coding Standards • Adopt block based coding (e.g. JPEG, MPEG, H.261/3+) • Lossy coding techniques • Images are segmented into blocks • Predictive Coding, Discrete Cosine Transform and Quantization are adopted

4. Background –Block Coding & Transmission Error • Block based coding technique Discrete Cosine Transform N x N block Q RLC Packet • image/video transmission system. Encoding Packetizing Network Depacketizing Decoding Image/ Video Source

5. Transmission & Error • Packet error during transmission causes all data loss in a corresponding block. • During transmission, some packets are lost due to bit error, congestion of network, noise burst, or other reasons. Packet

6. Background – Error Correction • Automatic Retransmission Request(ARQ) – Decoder sends retransmission requests to encoder when error happens. pro : error-free. cons : additional delays, unacceptable for real-time. example : TCP/IP – unreal-time internet file transfer protocol • Forward Error Correction(FEC) – Encoder inserts additional codes so that decoder can correct errors. pro : error resilient. con : additional bits example : 18-bit FEC code in H.261(video-conferencing standard) • Error Concealment/Block Loss Recovery – Post-processing to restore damaged blocks by decoder pro : no additional bits. con : complex decoder

7. Background –Error Concealment / Block Loss Recovery • Spatial EC : Exploiting spatially neighboring image data of a missing block. • Temporal EC : Exploiting temporally adjacent frame of video. t-1 t

8. Spatial Block Loss Recovery • Spatial block loss recovery is suitable for image coding data, (e,g, JPEG, Image based coding frame of MPEG/H.261/263+)

9. Previous Research (Spatial EC) -Edge-based Average Interpolation (Ancis ’99) • Averaging the values of pixels in the same position of surrounding blocks. where wk = 0, 1, according to edges in surrounding blocks

10. Previous Research (Spatial EC) - Interpolation Exploiting Interblock Correlation (Hemami ’95) • To get weight, W, the values of boundary pixels in a missing block are set to the value of adjacent known pixels. • Weight, W, are computed by solving the equations using Linear Least Squares Problem. (1~64) (1~30)

11. Previous Research (Spatial EC) – DCT-based interpolation (Ziad, 2000) • Set Y2 to zero =>

12. Previous Research – Other Algorithms • Maximally Smooth Image Recovery (Wang, ’93) - minimizing errors between a missing block and surrounding pixels • DCT Coefficients Block Recovery (Park, ’97) - minimizing errors between boundary pixels of a missing block and surrounding pixels • Block Recovery using POCS (Sun, ’95)

13. Previous Research – Problem Formulation • Conventional Algorithms - 1) Not adaptive to local image structure (Ziad, Hemami,…) 2) ill-utilizing surrounding image structure (hemami,…) 3) no ability of recover certain frequency bands (Sun, Ziad, Wang, Park…)

14. Block Loss Recovery – Research Goal • To design a spatial block loss recovery Techninque • More Reliable • Adaptive to Image Structure • No Retransmission • No Additional Bits for error correction

15. Projections based Block Recovery – Motivation • Conventional Algorithms use information of all surrounding area. • Using only highly correlated area

16. Projections onto Convex Sets –Alternating Projections • Alternating Projections is projecting between two or more convex sets iteratively. Converging to a common point

17. Signal Restoration using Alternating Projections – Papoulis-Gerchberg Algorithm Original Signal FT Band Limiting. Convex Set, C1 Corrupted Signal FT IFT C2 Identical Middle. Convex Set, C2

18. Projections based Block Recovery – Algorithm • 2 Steps • Pre Process : 1) Edge orientation detection 2) Surrounding vector extraction 3) Recovery vector extraction • Projections : 1) Projection operator P1 2) Projection operator P2 3) Projection operator P3

19. Pre Process 1 –Edge Orientation Detection • Edge orientation in the surrounding area(S) of a missing block(M). In order to extend the geometric structure to the missing block. • Simple line masks at every i, j coordinate in surrounding area(S) of the missing block(M) for edge detection. Horizontal Line Mask Vertical Line Mask

20. Pre Process 1 – Edge Orientation Detection • Responses of the line masks at window W : • Total magnitude of responses : • Th > Tv ; Horizontal line dominating area Th < Tv ; Vertical line dominating area

21. Pre Process 2 – Surrounding Vectors • Surrounding Vectors, sk, are extracted from surrounding area of a missing block by N x N window. • Each vector has its own spatial and spectral characteristic. • The number of surrounding vectors, sk, is 8N.

22. Pre Process 3 – Recovery Vector • Recovery vectors are extracted to restore missing pixels. • Two positions of recovery vectors are possible according to the edge orientation. • Recovery vectors consist of known pixels(white color) and missing pixels(gray color). • The number of recovery vectors, rk, is 2. Vertical line dominating area Horizontal line dominating area

23. Projections based Block Recovery –Projection operator P1 • Recovery vectors, ri, for i = 1, 2 • Surrounding vectors, sj, for j = 1 ~ 8N • Surrounding vectors, S, form a convex hull in N2-dimensional space • Recovery vectors, R, are orthogonally projected onto the line defined by the closest surrounding vector, si, j : Projection Operator P1.

24. Projections based Block Recovery –Projection operator P1 • Projection operator P1 : Convex hull (formed by surrounding vectors, containing information of local image structure)

25. Projections based Block Recovery –Projection operator P1 • Surrounding vectors, sj, for j = 1 ~ 8N • Recovery vectors, ri, for i = 1, 2 • The closest vertex, sdi, from a recovery vector, ri. or equivalently in DCT domain, • P1 :

26. Projections based Block Recovery –Projection operator P2 • Convex set C2 acts as an “identical middle”. • Projection operator P2 :

27. Projections based Block Recovery – Projection operator P3 • Convex set C3 acts as a convex constraint between missing pixels and adjacent known pixels, (fN-1 fN). where, and is a N x N recovery vector in column vector form. fN-1 fN

28. Projections based Block Recovery – Projection operator P3 • Projection operator P3 :

29. Projections based Block Recovery –Iterative Algorithm • Missing pixels in recovery vectors are restored by iterative algorithm of alternating projections : • N x N windows moving : Vertical line dominating area Horizontal line dominating area

30. Projections based Block Recovery - Summary Edge Orientation Detection Surrounding Vector Extraction Recovery Vector Extraction Projection Operator P1 Projection Operator P2 Projection Operator P3 Iteration=I? All pixels?

31. Simulation Results – Test Data and Error • 512 x 512 “Lena”, “Masquerade”, “Peppers”, “Boat”, “Elaine”, “Couple” • 176 x 144 “Foreman” • 352 x 240 “Flower Garden” • 8 x 8 pixel block loss • 16 x 16 pixel block loss • 8 x 8 consecutive block losses • Peak Signal – Noise – Ratio

32. Simulation Results –Lena, 8 x 8 block loss Original Image Test Image

33. Simulation Results –Lena, 8 x 8 block loss Ancis, PSNR = 28.68 dB Hemami, PSNR = 31.86 dB

34. Simulation Results –Lena, 8 x 8 block loss Ziad, PSNR = 31.57 dB Proposed, PSNR = 34.65 dB

35. Simulation Results –Lena, 8 x 8 block loss Ancis PSNR = 28.68 dB Hemami PSNR = 31.86 dB Ziad PSNR = 31.57 dB Proposed PSNR = 34.65 dB

36. Simulation Results – Each Step Lena 8 x 8 block loss (a) (b) (c)

37. Simulation Results –Peppers, 8 x 8 block loss Original Image Test Image

38. Simulation Results – Peppers, 8 x 8 block loss Ancis, PSNR = 27.92 dB Hemami, PSNR = 31.83 dB

39. Simulation Results – Peppers, 8 x 8 block loss Ziad, PSNR = 32.76 dB Proposed, PSNR = 34.20 dB

40. Simulation Results – PSNR (8 x 8)

41. Simulation Results –Lena, 8 x one row block loss Original Image Test Image

42. Simulation Results –Lena, 8 x one row block loss Hemami, PSNR = 26.86 dB Proposed, PSNR = 30.18 dB

43. Simulation Results –Masquerade, 8 x one row block loss Original Image Test Image

44. Simulation Results –Masquerade, 8 x one row block loss Hemami, PSNR = 23.10 dB Proposed, PSNR = 25.09 dB

45. Simulation Results –Lena, 16 x 16 block loss Original Image Test Image

46. Simulation Results –Lena, 16 x 16 block loss Ziad, PSNR = 28.75 dB Proposed, PSNR = 32.70 dB

47. Simulation Results –Foreman, 16 x 16 block loss Original Image Test Image Ziad, PSNR = 25.65 dB Proposed, PSNR = 30.34 dB

48. Simulation Results –Flower Garden, 16 x 16 block loss Original Image Test Image Ziad, PSNR = 20.40 dB Proposed, PSNR = 22.62 dB

49. Simulation Results – PSNR (Row, 16 x 16)

50. Simulation Results –Mean & Variance of Error (8 x 8)