A High-Quality Video Denoising Algorithm based on Reliable Motion Estimation

1. A High-Quality Video Denoising Algorithm based on Reliable Motion Estimation Ce Liu (MSR) William T. Freeman (MSR, MIT)

2. Initial Introduction • To remove real, structured noise introduced by low-end camcorders and digital cameras. • Structured noise: The noise in real cameras can have strong spatial correlations. • Sparsity in Image: For any image patch, there will be a similar ones in other locations of the image. This property is used in NLM method used for denoising. • Sparsity in video: A new frame can be well predicted from previous frames. (Temporally consistent or temporal coherence)

4. Chicken and Egg Problem… • Motion Estimation techniques (e.g. Block matching) are mainly based on temporal coherence across the frames. • Structured noise can mislead the search for similar patches and then breaks the temporal coherence criterion in video de-noising. • Motion should be estimated from the underlying signal after denoising, and denoising relies on the temporal correspondence from motion estimation. • Solution: Use of robust optical flow with spatial regularization to establish reliable temporal correspondence despite noise.

5. A temporally Coherent Video Denoising Framework • For every patch in a video, find a set of supporting patches from this frame and temporal adjacent frames that are similar to this patch. • Find spatially neighboring and temporally corresponding pixels. Algorithms Approximate KNN Optical flow

6. Approximate K-nearest Neighbors (AKNN) for a single frame • {I1, I2 , … , IT} – input noisy image. • z = (x,y,t) – index to a space time volume. • P(z) or P(x,y,t) – a patch at location z. • ; a set of approximate K-nearest neighbors for q = (x,y). • vi = qi – q ; offset of the found patch from the current patch. • Take a priority queue and store these K-nearest neighbors in it such that: Where D(·,·) is the sum of square distance (SSD) over two patches, defined as:

7. Three phases of AKNN • Initialization: The K-nearest neighbors are initialized by randomization where ni is a 2d standard normal random variable, and controls the radius (=w/3 initially). • Propagation: • The idea is to improve the approximate K-nearest neighbor set based on the fact that neighboring pixels tend to have similar AKNN structures (offsets). • In the scanline order, we attempt to improve AKNN {vi(x,y)} using neighbor {vi(x−1,y) and {vi(x,y,−1)}. In the reverse scanline order, we attempt to improve {vi(x,y)} using neighbor {vi(x+1,y)} and {vi(x,y+1)}. • Random Search: After the propagation step, we allow every patch to randomly match other patches in the image for M times using the following mechanism vi = σsαini , i = 1,··· ,M where ni is a standard 2d normal random variable, α = 1 and M = min(log2σs, K).

9. Non-local means with temporal coherence • Temporal coherence is vital for denoising. • Algorithm that produces more temporal coherent results is preferred. • Since, for real sequences, it can be difficult to distinguish high-intensity structured noise from image signal. Therefore, it is important to establish temporal correspondence between adjacent frames and require corresponding pixels to be similar. • For this optical flow algorithm is used. • Since, optical flow is not invertible, we estimate forward flow wf (z) = [vx, vy, 1] from frame It to It+1, and backward flow wb(z)=[vx,vy,−1] from frame It to It−1, in order to establish bidirectional correspondence.

10. Optical flow • Method to estimate the motion between two image frames taken at time t and t+Δt at every pixel position. • Let a pixel I(x,y,t) is moved in next frame to I(x+Δx,y+Δy,t+Δt), then I(x,y,t) = I(x+Δx,y+Δy,t+Δt) • By Taylor Series expansion which gives: Ixu+Iyv+It=0 • Horn-Schunck algorithm considers a global constraint of smoothness for aperture problem and gives an energy functional:

11. Optical flow (cont.) • This functional can be minimized by Euler-Lagrange equations as: • Which gives – where Δ is laplacian operator, Δu(x,y) = u(x,y) – u(x,y). • The above equations are linear in u and v, which can be iteratively solved as:

12. Non-local means with temporal coherence (cont.) • So, pixel z corresponds to z + wf (z) in next frame and to z + wb(z) in previous frame across |H| frames. • Modified AKNN’s , which forms the supporting patches for P(z) is Ni = {P(zij)}, j=1,…,K. for ith frame. where, zij = (xij, yij,,i)

13. Non-local means with temporal coherence (cont.) • The non local means estimate for pixel z can be written as: where Z is the normalization factor: and Dw (·,·) is a weighted SSD function, summed over spatial, but not temporal, offsets: where σp = s/2 , and Z’ is a normalization constant, γ = 0.9 to control temporal decay and σt is related to the noise level in the video sequence.

14. Non-local means with temporal coherence (cont.) • For a fixed number of iterations, the complexity of this denoising algorithm for a frame is O(NHKlogK), where N is the number of pixels per frame, H is the temporal window size, and K is the number of approximate K-nearest neighbors. • This is a significant reduction compared to O(N2) of the original NLM algorithm, since K << N (typically K = 10 and N = 640×480).

15. Noise estimation for adaptive noise removal • Set σt small when noise level is low to avoid over smoothing, and when the noise level is high, set σt large to smooth out noise. • Proposed noise model based on optical flow: • Ideally, the difference between the warped frame and It should be the difference of independent noise, but due to unreliable motion estimation an outlier in noise estimation is used: here, nz is a pixel-wise Gaussian random variable: E(nz)=0, E(nz2)= σn, and uz ∼ U[−1,1] is a pixel-wise uniform random variable. These two random variables are balanced by weight αz. Let Jt(z) = It(z) − It+1(z + wf (z)).

16. Noise estimation for adaptive noise removal (cont.) • We use an expectation-maximization (EM) algorithm to estimate parameters: • Initialize σn = 20. Loop between step 2 and 3 until convergence. • (E-Step) Evaluate • (M-step) Estimate • This estimation is performed for each of R, G and B channels independently. • The relationship between the noise level σn and scaling parameter σt in Eqn. (5) depends on K and H. • Empirically, it has been found that when K = 11 and H = 5, σt = σn generates visually pleasing results.

17. Experimental Results (given by Authors) • Implementation Details: • Used Patch size – 7X7 • K = 11 • H = 5 • 4 Iterations of random K-nearest neighbor matching for each frame. • The EM algorithm for noise estimation converges in about 10 iterations. • Tennis sequence: • added artificially generated AWGN (σ = 20). • Compared with BM3D which gave PSNR 30.22db in first run and 31.20db in second run. • This algorithm gave PSNR 30.21db. • Real video sequence room captured by a Canon S90. • Denoising quality depends on the quality of motion estimation.

18. Experimental Results (given by Authors) • Evaluation of motion estimation: • Applied block matching and optical flow methods on the video. • Block matching failed due to structured noise, but optical flow gave good smooth results. • For room sequence, two parameters are used σ = 20 and σ = 40. The adaptive noise filtering method outperforms BM3D.

19. Experimental Results (given by Authors) • Temporal coherence as a quality measure: • Used the human-assisted motion annotation tool [27] to annotate the ground-truth motion of the room sequence.

20. Experimental Results (given by Authors) The average standard deviation along motion paths is measured for different algorithms at different RGB channels. Proposed system has overall the least temporal fluctuation.