Image deblocking using local segmentation

1. Image deblocking using local segmentation By Mirsad Makalic Supervisor: Dr. Peter Tischer

2. Presentation Outline • An introduction to JPEG • Lossy image compression • Discrete Cosine Transform (DCT) and quantization • Local segmentation and prior research • Measuring image quality • Deblocking Filter • Super-Resolution Filter • Conclusion

3. Lossy image compression • JPEG is the most common lossy image compression format • Easy to implement • Good quality with high compression ratios • Block based transform approach • Divide image into 8x8 blocks • Perform discrete cosine transform (DCT) on each 8x8 block • Quantize the DCT coefficients

4. Discrete Cosine Transform DC coefficient • Transform 8x8 block of pixels into a set of weighted basis functions (DCT coefficients) AC coefficients

5. DCT - Subbands • An image consisting of just one coefficient from each 8x8 block is called the sub-band image • An image which is divided into 8x8 blocks has 64 possible sub-band images An 8x8 image split into 2x2 blocks has 4 subbands

6. Quantization • Exploit visual redundancy • Quantization is a many-to-one mapping • Divide each DCT coefficient by a value and round to the nearest integer • The decoder makes a guess from a range of values (pick midpoint by default) 5 6 7 8 9 10 11 12 13 14 Quantizer =10 = 1

7. Quantization • Coarse quantization introduces artifacts into reconstructed image • DCT coefficients are reconstructed inaccurately • Most visually distracting artifact is blockiness JPEG compressed image at quality 10 PSNR: 30.41 dB

8. Deblocking Techniques • Three approaches in the literature: • Filter the reconstructed pixel values • Attempt to reconstruct DCT coefficients more accurately • A hybrid approach

9. Local Segmentation • Most deblocking filters introduce excessive blurring • Destroys the structure of the image • Edges lose their sharpness • Local segmentation takes into account the structure of the image

10. Local Segmentation • Divide a mask of pixels into N segments and filter each segment independently Average of whole mask = 43.22, average of yellow segment = 20.8 • Two questions: • How many segments do we use? • How do we segment a mask of pixels?

11. Prior Research • Lukasz Kizewski, BSE (hons) 2004 • DC subband approach • Filter using a mask of DC subbands • How do we segment a mask of pixels? • Segment the pixel mask using thresholding • How many segments do we use? • Do-No-Harm heuristic

12. Prior Research • Do-No-Harm heuristic • Try a 1-segment model (average of the whole mask) • If filtered value is implausible reject and try a 2-segment model • A plausible value is one which falls inside the quantization range: midpoint +/- ½ Quantum • Try a 2-segment model • If still implausible then don’t filter

13. Prior Research • Results (a) Unfiltered image (a) Filtered image

14. Room for improvement • No objective measure used to test the effectiveness of the filter • Difficult to make comparisons • Difficult to rate changes in filter • Works only on DC subband • AC subbands contain edge and texture information • A very simple local segmentation method

15. Measuring image quality • Peak-signal-to-noise-ratio • Most commonly used metric • Does not necessarily reflect the subjective visual quality • Generalized Block-Edge Impairment Metric (GBIM) – H.R. Wu, M. Yuen

16. Measuring image quality - GBIM • Measures the quality of DCT encoded images • Assume that what happens inside a block is the same as what happens across blocks • Take absolute mean difference of pixels inside a block (vertical/horizontal) • Take absolute mean difference of pixels across blocks (vertical/horizontal) • Compare them, if the two differ greatly than it is a sign of blockiness 2x2 block example with vertical blockiness

17. A new deblocking filter • Filter all coefficients • Treat each 8x8 block as a 64 element vector where each value in the vector is one of the DCT coefficients • Local segmentation no longer as simple (need to segment masks of vectors) • K-Means or K-Nearest Neighbours • DNH needs to work on vectors

18. A new deblocking filter • Basic structure of filter is same: • Create an NxN mask where each item is a 64 element DCT coefficient vector • Segment mask using K-Means or K-Nearest Neighbours segmentation • Check if segmentation produces valid result using DNH, if not, try different segment • If no segmentation produces valid result, leave alone

19. K-Means • Start with one segment (average of mask) • If segmentation is invalid, increase number of segments by one until a maximum number of segments is reached • Try largest change first

20. K-Nearest Neighbours • Set the number of nearest neighbours to find as the number of items in the entire mask • Keep decreasing by segment size by one until a valid segment is found

21. Vector DNH • Center DNH • Compare the filtered vector against only the center vector in the mask • Strict Segment DNH • Compare the filtered vector against all the vectors in the segment the center vector is in • Lesser Strict Segment DNH • Same as strict segment DNH with some error tolerance

22. Results • Best found parameters for the filter: • 3x3 vector mask (covers 24x24 pixels) • K-Nearest Neighbour segmentation • Lesser strict segment DNH with 1% error tolerance • Tested: • 5x5 mask, K-Means, Center DNH, Strict DNH etc. • Many variations of the filter parameters

23. Results (a) JPEG compressed image (b) Filtered image • An improvement of 0.07 dB in PSNR and a reduction of 0.62 in GBIM

24. Results (a) JPEG compressed image (b) Filtered image • An improvement of 0.07 dB in PSNR and a reduction of 0.55 in GBIM

25. Other uses for local segmentation • Super-resolution • Combine multiple slightly different images to form one higher quality image • Can we extract more information out of a single image? • Neighbouring pixels are similar and share information • Use local segmentation

26. Super-resolution filter • Very similar to the deblocking filter • Instead of using a 64 element vector for each DCT coefficient, use one element vector containing each pixel value in the mask • How to test if it works? • Convert 8 bit image to 4 bits and attempt to reconstruct back an 8 bit image

27. Results • Best found parameters for filter: • 3x3 mask • K-Means segmentation with up to three segments • Center DNH • More than one iteration of the filter can further improve PSNR

28. Results (a) An image rounded to 4 bits per pixel (a) Filtered image (8 bits per pixel) • An improvement of 1.38 dB in PSNR

29. Results (a) An image rounded to 4 bits per pixel (a) Filtered image (8 bits per pixel) • An improvement of 1.32 dB in PSNR

30. Conclusion • Filtering AC subbands is difficult because most have been quantized to zero or have very large quantization ranges • Most improvements in image quality are from the stricter DNH and better local segmentation techniques • The increase in computational complexity may not be worth the increase in image quality

31. Conclusion • Super-resolution filter shows a lot of promise • Large increase in PSNR and image quality • A different DNH heuristic may work better with the super-resolution filter

32. Future Research • Allow more variation in pixels for the strict DNH • Assume local mask is linear and not constant • Try different segmentation techniques • Region growing • Further investigate iterative filtering

33. The End • Questions?