Noise Estimation from a Single Image

1. Noise Estimation from a Single Image Ce Liu William T. Freeman Richard Szeliski Sing Bing Kang

2. Parameter Tweaking in Computer Vision • Computer vision algorithms suffer from hand tuning parameters for particular images or image sequences • We want vision algorithms that behave properly under varying lighting conditions, blur levels and noise levels • Our work is one step in that direction • Given an image, estimate the noise level • Modify vision algorithms to be independent of noise

3. Image Noise Is Important in Vision • In image denoising the noise is assumed to be known as Additive Gaussian White Noise (AWGN) • However, in real applications the noise is unknown and non-additive • Many other computer vision algorithms also explicitly or implicitly assume the type and level of image noise • Hard to make vision algorithms fully automatic without knowing noise

4. s I Noise Level Function (NLF) • The standard deviation of noise s is a function of image brightness I • Measurable by fixing the camera and taking multiple shots of a static scene • For each pixel: • Mean: I • Standard deviation: s • NLF depends on camera, ISO, shutter speed, aperture • Our goal is to estimate NLF from a single image • How to estimate noise without separating noise and signal?

5. An Example Image

6. = + Signal Residual s s s Red Green Blue 0.2 0.2 0.2 Standard deviation 0.1 0.1 0.1 I I I 0 0 0 0 0.5 1 0 0.5 1 0 0.5 1 Brightness Piecewise Smooth Image Prior Affine model Patch Standard deviation s Brightness mean I For each RGB channel:

7. = + Signal Residual s s s Red Green Blue 0.2 0.2 0.2 Standard deviation 0.1 0.1 0.1 I I I 0 0 0 0 0.5 1 0 0.5 1 0 0.5 1 Brightness Piecewise Smooth Image Prior Patch

8. = + Signal Residual s s s Red Green Blue 0.2 0.2 0.2 Standard deviation 0.1 0.1 0.1 I I I 0 0 0 0 0.5 1 0 0.5 1 0 0.5 1 Brightness Piecewise Smooth Image Prior Patch

9. Segmentation-based Approach Observed image

10. Segmentation-based Approach Over-segmentation

11. Segmentation-based Approach Signal

12. Segmentation-based Approach Residual= noise + unmodelled image variation

13. s s s Residual std. dev. I I I Brightness Estimate NLFs • Assume brightness mean I is accurate estimate • Standard deviation s is an over-estimate: (may contain signal) • The lower envelope is the upper bound of NLF

14. s s s Residual std. dev. I I I Brightness Issues • Should the curve be strictly and tightly below the points?

15. s s s Residual std. dev. I I I Brightness Issues • Should the curve be strictly and tightly below the points? • How to handle the missing data?

16. s s s Residual std. dev. I I I Brightness Issues • Should the curve be strictly and tightly below the points? • How to handle the missing data? • Correlation between RGB channels?

17. s s s Residual std. dev. I I I Brightness Solutions • Formulate the inference problem in a probabilistic framework • Learn the prior of noise level functions

18. Outline • Over-segmentation and per-segment variance analysis • Learning the priors of noise level functions (NLF) • Synthesize CCD noise • Sample noise level functions • Learn the prior of noise level functions • Inference: estimate the upper bound of NLF • Bayesian MAP to estimate NLFs for RGB channels • Applications • Adaptive bilateral filtering • Canny edge detection

19. Dependent noise: Independent noise: Camera Noise • Noise model • Camera response function (CRF) f: download from Columbia camera response function database (used 196 typical CRFs) Shot Dark Current Camera Noise Noise Irradiance Scene Lens / L Radiance Atmospheric CCD Imaging / Fixed Pattern geometric Attenuation Bayer Pattern Noise Distortion Quantization Thermal Noise Noise Digital Image I Interpolation / A / D Gamma White t Demosaic Converter Correction Balancing Tsin et. al. Statistical calibration of CCD image process. ICCV, 2001

20. Camera response function: f Dependent noise: Independent noise: Estimate NLF Synthesize CCD Noise I

21. 0.02 0.06 0.04 0.02 0.18 Dependent noise: Independent noise: 0.02 0.18 Sample NLFs by Varying the Parameters Camera response function (CRF) f

22. The Prior of NLFs

23. I Likelihood Function • The estimated standard deviation should be probabilistically bigger than and close to the true value • Bayesian MAP inference

24. Validation (1): Synthetic Noise • Add synthetic CCD noise, estimate, compare to the ground truth —ground truth estimated — — —

25. Validation (2): Measure NLF of a Real Camera • 29 images were taken under the same settings (the camera is not in the database for training) • The real NLF is obtained by computing mean and variance per pixel

26. Validation (3): Robustness Test • Verify that different images from the same camera give the same estimated NLF (camera not in the database for training)

27. Application (1): Adaptive Bilateral Filtering • Bilateral filter is an edge-preserving low-pass filter • Spatial sigma and range sigma • Adaptive bilateral filter • Down-weigh RGB values by signal and noise covariance matrices • The range sigma is set to be a function of the estimated standard deviation of the noise Input noisy image Smoothing kernel Denoised image From Durand and Dorsey, SIGGRAPH 02

28. Red Red Green Green Blue Blue low noise high noise Test on Low and High Noise

29. Results—Adaptive Bilateral Filtering Standard bilateral filtering Adaptive bilateral filtering low noise high noise

30. Results—Adaptive Bilateral Filtering Standard bilateral filtering Adaptive bilateral filtering Zoom in high noise

31. Red Red Green Green Blue Blue Application (2): Canny Edge Detection low noise high noise

32. Results—Canny Edge Detection Parameters adapted in MATLAB Parameters adapted by estimated noise low noise high noise

33. Conclusion • Piecewise-smooth image prior model to estimate the upper bound of noise level function (NLF) • Estimate the space of NLF by simulating CCD camera on the existing CRF database • Upper bounds are verified by both synthetic and real experiments • An important step to automate vision algorithms independent of noise

34. Thank you! Noise Estimation from a Single Image Ce Liu William T. Freeman CSAIL MIT Rick Szeliski Sing Bing Kang Microsoft Research