Transform-based Non-local Methods for Image Restoration

1. Transform-based Non-local Methods for Image Restoration IT530, Lecture Notes

2. Non-local Techniques Natural images have a great deal of redundancy: patches from different regions can be very similar NL-Means: a non-local pixel-based method (Buadeset al, 2005) • Awate and Whitaker (PAMI 2007) • Popat and Picard (TIP 1998) • De-Bonet (MIT Tech report 1998) • Wang et al (IEEE SPL 2003) Difference between patches

3. Transform-based Techniques DCT coefficients of a “typical” natural image Observed property of natural images: Sparse coefficients in Fourier/DCT/ Wavelet domain

4. Transform-based Techniques Manipulate coefficients of noisy patch coefficients of filtered patch DCT Coefficients (absolute value) Get filtered patch by inversion of the transform from filtered coefficients Noisy Patch Transform basis Patch matrix Repeat for all patches in sliding window fashion: final filtered image obtained by averaging overlapping filtered patches. Project Patch onto Transform Basis (DCT/Wavelet etc.) Transform Coefficients Clean Patch

5. Non-local transform based techniques • Machine learning approach. • Make use of non-local self-similarity at the patch level to learn good transform bases for denoising.

6. Non-local PCA • PCA is the most basic transform to learn! • Assume Gaussian noise (mean zero, known variance). • Given a patch (called “reference patch”) in a noisy image, search for similar patches elsewhere in the image. • Given such a collection of patches, compute their PCA bases. • Compute the eigen-coefficients of the reference patch.

7. Non-local PCA • Manipulate these eigen-coefficients (by hard-thresholding or Wiener filter). • Reconstruct the reference patch. • Repeat the procedure for every patch in the noisy image in sliding window fashion. • Average the multiple hypotheses that appear at a pixel. Ref: Muresan and Parks, “Adaptive principal components for image denoising”, ICIP 2003.

8. Criterion: Patch Similarity Two patches: identical modulo noise Cumulative of chi-squared is incomplete gamma function. A patch P1 is considered similar to patch Pref if their squared difference is less than or equal to

9. Non-local PCA Noisy patch The l-th coefficient of yi – computed by projection onto the l-th local PCA bases (obtained from similar patches) We are modelling as zero-mean Gaussian random variable with variance

10. Two-stage non-local PCA Ref: Zhang et al, Two stage image denoising by principal components analysis with local pixel grouping

13. KSVD BM3D PCA

14. KSVD BM3D PCA

15. KSVD BM3D PCA

16. Time complexity • The patches of size p x p are treated as vectors of size p2 x 1. • Hence the covariance matrices are of size p2 x p2. • Eigen-analysis will have a time complexity of O(p6) per patch (not counting the time to search for similar patches). • This is quite expensive.

17. Non-local collaborative filtering: Block Matching in 3D (BM3D) • The state of the art method today. • Based on the idea of non-local similarity at the patch-level. • Given a reference patch in the noisy image, this method again collects similar patches. • But this time, the similar patches and the reference patch are arranged in the form of a 3D stack (of say some K patches in all).

18. Non-local collaborative filtering: Block Matching in 3D (BM3D) • The stack is projected onto 3D transform bases (typically 3D DCT, or tensor product of 2D DCT and 1D Haar wavelets). • The 3D transform coefficients are manipulated – usually by hard thresholding using the universal threshold . • All the patches in the entire stack are reconstructed using an inverse 3D transform. • This is repeated for every patch in the image. The multiple answers appearing at any pixel are averaged.

19. BM3D – second stage • The preceding steps form the first stage of BM3D. • The output image of the first step is used to compute patch similarities (this will be more robust than computing the similarities in the noisy image). • Patches from the first-stage image are then appropriately assembled into a stack. • Corresponding patches from the noisy image are assembled into a second stack.

20. BM3D – second stage • 3D transform coefficients of both the stacks are computed. • The second stack is denoised using Wiener filtering as follows: • This is again repeated in sliding-window fashion with averaging.

21. Note: BM3D does allow individual patches from the patch to be independently filtered. It filters the similar patches from the stack collectively, assuming a dependence between them! This is the reason for its superior performance.

22. Results • State of the art denoising method. • Impressive results even at high noise levels. • Residual images produced are very noisy. • Preserves textures and fine details/edges very well.

24. Noisy PSNR: 22.13 NL-SVD: PSNR 30.88 SSIM 0.882 BM3D1: PSNR 31.02 SSIM 0.884 BM3D2: PSNR 31.66 SSIM 0.903 HOSVD: PSNR 31.53 SSIM 0.897 NLMeans: PSNR 29.42 SSIM 0.821 KSVD: PSNR 30.762 SSIM 0.877

25. NL-SVD BM3D1 BM3D2 HOSVD KSVD

26. Noisy PSNR: 22.13 NL-SVD: PSNR 30.187 SSIM 0.801 BM3D1: PSNR 30.395 SSIM 0.809 BM3D2: PSNR 30.8 SSIM 0.824 HOSVD: PSNR 30.491 SSIM 0.814 NLMeans: PSNR 28.91 SSIM 0.753 KSVD: PSNR 30.36 SSIM 0.803

27. BM3D2 HOSVD KSVD NL-SVD BM3D1

28. Gaussian Noise sigma = 15