Visual Artifact Reduction By Post-Processing

1. Presented at Carnegie Mellon University Dept. ECE, November 1, 2001 Visual Artifact Reduction By Post-Processing Yu Hen Hu (hu@engr.wisc.edu) University of Wisconsin -- Madison Dept. Electrical and Computer Engineering in collaboration with Dr. Seungjoon Yang

2. Outline • Visual artifacts in digital image and video • Post-processing Method • Enhancement • Restoration • Blocking Artifact Reduction using GenLOT-embedded Inverse DCT • Ringing Artifact Reduction Using Robust ML filters

3. Visual Artifacts in Digital Visual Materials • Digital Images and Videos • Obtained by digitizing analog visual materials or direct capture. • Visual quality remain unchanged after copying. • Visual Artifacts in digital images and videos • Artifacts due to inadequate acquisition • Out of focus, smearing, dust, scratch, blotches • Artifacts due to inadequate processing • Lossy compression – coding artifact • Watermarking • Artifacts due to transmission error • Lost blocks, frames, discoloring

4. Spatial artifacts Content independent Blotches Scratches Additive noise Content dependent Blocking Ringing Smearing Temporal artifacts Frame jittering Successive image frames at slightly different spatial positions Line jittering Individual raster line mis-aligned, causing vertical straight line jitters Rapid, uneven camera motion Uneven panning, tracking, etc. Discontinued motion of objects Due to frame skipping Types of Visual Artifacts

5. Coding Artifacts • Depending on the type of the transform (a) Blocking Artifact (b) Ringing Artifact Non-Overlapping Transforms Overlapping Transforms

6. Visual post-processing • Image and video processing procedure applied after normal processing (including compression, decompression), prior to final presentation. • Purpose – to enhance visual quality of images and videos by removing unsightly visual artifacts. • Artifact reduction is accomplished in two phases: • Artifact detection • Artifact removal

7. Artifact Detection • A statistical signal detection problem. • Exploit unique feature of different types of artifacts • Difficult for content dependent artifacts • Temporal artifact detection must be performed by examining consecutive frames (spatial-temporal analysis) • Must exploit human visual system (HVS) characteristics • Visual masking effect

8. Artifact Reduction • Two steps: • Visual quality enhancement – reducing unsightly artifact • Restoring original content by reversing the degradation process • Need to estimate lost information • Artifact may cover original contents – e.g. blotches • Artifact may be due to lost of original content – e.g. coding artifact

9. Transformed Image Coding Artifact Reduction:Problem Statements

10. Transformed Image Coding f F Fq Transform Quantizatoin Lossless Coding Bit Allocation (a) Encoder Fq g Lossless Decoding Dequantization Inverse Transform (b) Decoder

11. Blocking artifact Cause Lossy quantization of frequency coefficients Symptom Spurious edges at block boundary (known position) Issues: preserving true edges that cross or overlap with block boundary Common approaches: spatial domain filtering DCT coefficients adjustment Ringing Artifact Cause Lossy quantization of frequency coefficients Symptom Spurious edges along major edges Issues: difficult to separate ringing artifact with true edge of texture Common approaches: Restoration while preserving major edges Medium filtering Coding Artifacts

12. Blocking Artifact • The edge map is enhanced to show spurious edges along block boundaries. • Visual masking effect: Blocking artifact is more visible at relatively flat region of an image.

13. Blocking Artifact • With block-based transform coding, pixels acrossing the block boundary are encoded with different set of basis functions. • Reconstructed Block (g: pixel value, Fq: freq. Coef., : basis) • Discontinuity at Block Boundary where • If the basis function overlaps between adjacent blocks, the blocking effect may be alleviated.

14. Ringing Artifact • Ringing Artifact: Oscillation (spurious edges) at the Vicinity of major (high contrast, large scale) edges • Visual masking effect: prominent in areas with relatively smooth background Ringing artifact Enhanced edge map

15. Cause of Ringing Artifact • Gibbs phenomenon -- Truncation of frequency domain coefficients corresponds to convolving the time domain sequence with a sinc function. The side lobes manifest themselves as oscillations in spatial domain around step discontinuities. • The cause of the ringing artifact is similar to that the of Gibbs phenomenon when long basis functions are cut short due to heavy lossy quantization Truncation of DCT coefficients from 128 down to 32 leads to ringing artifact: Original is a step function. With only 32 low frequency coefficients left, ringing occurs.

16. Blocking Artifact Reduction using GenLOT-embedded IDCT Embedding DCT/IDCT in a more general lapped orthogonal filter bank transformation. Modifying the GenLOT coefficients to reduce blocking artifact. Performed only at decoding end without altering encoding process. Maximum Likelihood estimation of ringing artifact free image Use a flat-surface image model to distinguish true edges from ringing edges that have smaller magnitudes. Use ML method to estimate true image value within a sliding window. Implemented as a data-adaptive, nonlinear robust filter New Approaches

17. Blocking Artifact Reduction using Embedded IDCT S. Yang, S. Kittitornkun, Y. H. Hu, T. Q. Nguyen, and D. L. Tull, “Generalized Lapped Biorthogonal Transform embedded inverse discrete cosine transform,” IEEE Trans. Image Processing, pp. submitted, 2000.

18. Existing Blocking Artifact Reduction Methods • MAP based restoration Approach [O’Rourke & Stevenson, 95] • POCS (Projection onto convex set): • Constraints can be imposed to ensure the quantized bit stream of smoothed image is close to that of decoded image. E.g. POCS [Yang, Galatsanos, Katsaggelos, 95] • Location-specific smoothing (filtering) • Replacing step edges along block boundary by smoothed pixel values. Nonlinear smoothing may be used (H.263, annex J) • Can be applied in spatial as well as frequency domain • Optimization Approach [Minami & Zakhor, 95]

19. MAP Estimation† • Given g, find f such that : Feasible Image Set : Prior Knowledge on Estimate †“Improved Image Decompression for Reduced Transform Coding Artifacts”, T.P. O'Rourke and R.L. Stevenson

20. MAP Estimation • Gibbs Image Prior Distribution • With Gibbs Image Prior Distribution c : clique  : potential function

21. MAP Estimation with Line Process • Take away the option for discontinuity at block boundaries.  : Line Process At block boundaries

22. POCS† • Find f such that • Actual Implementation †“Projection-Based Spatially Adaptive Reconstruction of Block-Transform Compressed Images”, Y. Yang and N.P. Galatsanos and A.K. Katsaggelos

23. Nonlinear Filtering† • Deblocking option in H.263+ Annex J where d1,d2,clip() are designed for appropriate smoothing †“Annex J of H.263+”,

24. Blocking Artifact Reduction using Overlapped Transformation • Blocking artifact is due to block-based transform of images. • At low bit-rate compression, many frequency coefficients are quantized to zero • Overlapped Orthogonal Transform (LOT) compute frequency coefficients of the same block size using longer bases that overlap adjacent blocks. As a result, blocking effect is less prominent using LOT in place of DCT • However, LOT is not standard-compliant! • Our approach: • Realize blocking artifact reduction in overlapped transform coefficient domain • Maintain standard compliance by embedding DCT/IDCT pair within an overlapped transformation.

25. Overlapped and Non-overlapped Transforms • 1D linear transform = projection of 1D signal onto basis function of the transform. • Base on how the basis functions are interlaid, can be classified into overlapped and non-overlapped transform. • Let • Non-Overlapping Transforms (e.g. blocked DCT) (b) Overlapping Transforms, e.g. LOT

26. ¯M Z-1 ¯M Z-1 Z-1 ¯M Generalized Lapped Biorthogonal Transform (GLBT)† Transformed coefficients from adjacent blocks T G0(z) GK-1(z) GK-1-1 Forward Transform Inverse Transform †“The Generalized Lapped Biorthogonal Transform”, T.D. Tran and T.Q. Nguyen

27. DCT is Part of GLBT DCT GLBT’s Frist Stage T

28. ¯M ­M Z-1 Z-1 ¯M ­M Z-1 Z-1 Z-1 Z-1 ¯M ­M GLBT Embedded IDCT (ge-IDCT) Standard non-compliant Forward Transform Inverse Transform Tdct G G-1 Tdct-1 Standard compliant DCT ge-IDCT

29. ¯M ­M Z-1 Z-1 ¯M ­M Z-1 Z-1 Z-1 Z-1 ¯M ­M ge-IDCT based Transform • ge-IDCT is standarad compliant! Tdct Quantization G G-1 Tdct-1 Same as standard DCT based Encoder Ge-IDCT based Decoder

30. ¯M ­M Z-1 Z-1 ¯M ­M Z-1 Z-1 Z-1 Z-1 ¯M ­M Blocking Artifact Reduction in Embedded Lapped Transform Domain Perform blocking artifact reduction in GLBT coefficient domain. Tdct Quantization G Post-Processing G-1 Tdct-1 Standard Encoder Improved Decoder

31. = Fe1 + + + Fe2 FO2 FO1 + + + + FO3 Fe4 Fe3 FO4 Deblocking • Blocking Artifact: Discontinuity between 2 DCT blocks • Empirically, odd-symm. frequency coefficients contribute more to blocking effects.

32. Frequency Weighting • Solution • reduce excessive energy in odd-sym. coef.’s. • Method • After lapped transform G of DCT coefficients, apply weights on two lowest odd-symm. coefficients • Features • Selective Smoothing • Preservation of Major Structure: Robustness

33. New Inverse Transforms (a) ge-IDCT (b) le-IDCT

34. sr sc sl PSNR & MSDS† • Peak Signal to Noise Ratio • Mean Square Difference of Slope †“An Optimization Approach for Removing Blocking Effects in Transform Coding”, S. Minami and A. Zakhor

35. Evaluation: PSNR & MSDS PSNR improvement MSDS reduction (a) Airplane (b) Lena (c) Peppers

36. Evaluation: Subjective Quality (a) JPEG (b) ge-IDCT At quality factor 15

37. Comparative Study • Existing Methods • Maximum a posteriori (MAP) Estimation (MAP) • Projection onto Convex Set (POCS) • Nonlinear Filtering (NF) • We also implement • MAP Estimation with Line Process (MAP-L)

38. Comparison: PSNR & MSDS (a) PSNR (b) MSDS

39. Comparison: Less Over-smoothing MAP ge-IDCT MAP-L

40. Comparison: Less Under-smoothing POCS le-IDCT NF

41. Comparison: Simplicity • Unitary transformation in ge-IDCT and le-IDCT can be implemented with Lattice Structures and CORDIC. Iterative Algorithms Non-Iterative Algorithms MAP, MAP-L, POCS NF, ge-IDCT, le-IDCT

42. Summary • New Inverse Transforms • (DCT, ge-IDCT) replaces (DCT, IDCT) • Processing in Embedded Lapped Transform Domain • Effective and Efficient Removal of Blocking Artifact

43. Ringing Artifact ReductionUsingMaximum Likelihood Robust Filtering S. Yang, Y. H. Hu, D. L. Tull, and T. Q. Nguyen, “Maximum likelihood parameter estimation for image ringing artifact removal,” IEEE Trans. Circuits and Systems for Video Technology, vol. 11, 2001, (to appear).

44. MAP Estimation • Wavelet & Bit plane Coding Based Codecs. • MAP Estimation • Redesending Functions

45. Feasible Image Set • For Bit Plane Coding • Feasible Image Set

46. Approximated MAP Estimation† • Enforcement of the Feasible Image Set • Replacement † “Artifact Removal in Low Bit Rate Wavelet Coding with Robust Nonlinear Filtering”, M. Shen and C.C.J. Kuo

47. Morphological Filter† • Detection Phase: • Removal Phase: † “Image coding ringing artifact reduction using morphological post-filtering, part I: the algorithm”, S.H.Oguz, Y.H.Hu, and T.Q.Nguyen

48. Flat Surface Model • “Replace ripped surface with flat surface.” • Flat Surface Model: Images are montage of flat surfaces. • Each surface consists of a cluster of pixel intensity values. K: the number of surfaces : averaged grayscale value of each surface z: surface information (membership of individual pixel)

49. Examples

50. ML Parameter Estimation • Estimate model parameter from the samples. • Maximum Likelihood Parameter Estimation A pair of (G,z) is regarded as complete data G: pixel values Z: membership of each pixel to a particular surface (cluster)