Processing of I mages B ased on B lind E valuation of N oise T ype and C haracteristics

1. Processing of Images Based on Blind Evaluationof Noise Type and Characteristics Vladimir V. Lukina, Sergey K. Abramova, Nikolay N. Ponomarenkoa, Mikhail L. Ussa,b, Benoit Vozelb, Kacem Chehdib, Jaakko Astolac a National Aerospace University, 61070, Kharkov, Ukraine; b University of Rennes I, 22 305 Lannion cedex, BP 80518, France; c Tampere University of Technology, Institute of Signal Processing, P.O. Box-553, FIN-33101, Tampere, Finland Processing of Images Based on Blind Evaluationof Noise Type and Characteristics Vladimir Lukin

2. Contents • Contents • Introduction • A priori Knowledge on Noise Statistics, Its Use in Image Processing • Examples of False Preliminary Assumptions on Noise Type • General Aspects of Image Processing • Methods for Blind Determination of Noise Type • Methods for Blind Determination of Noise Statistical Characteristics • Edge Detection Example • Requirements to Methods of Blind Determination of Noise Statistical Characteristics • Image Filtering Examples • Noisy Image Compression • Automatic Procedure for Lossy Compression • Accuracy of Blind Estimation Methods • Conclusions Vladimir Lukin

3. Introduction Applications: Systems for single- or multi-channel remote sensing (RS) data processing Goal: Carrying out analysis of mixed noise influence for three particular operations of image processing: edge detection, filtering and compression of noisy images RS image Image processing Blind noise type determination → false decisions may occur Additive noise Multiplicative noise Impulse noise Reasons: insufficient observation interval, speckle effects Blind noise parameters evaluation → inaccurate estimates may occur Reasons: transmitting and encoding/decoding errors Reason: circuit and atmospheric noise Influence on: edge detection, filtering and compression Vladimir Lukin

4. A priori Knowledge on Noise Statistics, Its Use in Image Processing Characteristics of noise: • Noise type (additive, signal-dependent, multiplicative, impulse, mixed, etc.); • PDF, variance, relative variance, probability of impulse noise, etc.; • Is noise i.i.d. or spatially correlated; 2D ACF or spectrum in the latter case. Peculiarities: There are no universal methods for coping well with any type and level of noise. Specialized methods that exploit available information on noise type and its characteristics in full extent as a rule provide the most efficient processing. Examples: • Noise PDF is needed to design optimal edge detectors and to set thresholds for most known edge detectors; • Noise statistics is used in image filtering (wavelets, DCT, sigma, Lee, Frost, Kuan) and reconstruction (Katsaggellos et al); • Lossy compression of noisy images. Vladimir Lukin

5. A priori Knowledge on Noise Statistics, Its Use in Image Processing Statement: Information on noise type and statistics is to be available or it should be retrieved for data (image) at hand. Otherwise, a used method has to exhibit certain robustness, i.e. ability to perform well enough in conditions of some unknown parameters and characteristics of noise. Fortunately, some methods for blind noise model identification and estimation of its parameters exist and they are rapidly developing*. Goal: Carrying out analysis of these methods for three particular operations of image processing, namely, edge detection, filtering and compression of noisy images. * B. Vozel, S. Abramov, K. Chehdi, V. Lukin, N. Ponomarenko, M. Uss, Blind methods for noise evaluation in multi-component images, book chapter in “Multivariate image processing”, ISTE Ltd, France, 2009 (in press). Vladimir Lukin

6. Examples of False Preliminary Assumptions on Noise Type: Optical Images The noisy (dominant Poisson) test image (left) and this image after Gamma-correction (γ = 0.6) (right) Scatter-plots of local estimates of variance for original noisy (left) and corrected images (right) Vladimir Lukin

7. Examples of False Preliminary Assumptions on Noise Type: Radar Images Scatter-plot of local estimation pairs (variance and squared mean) with two approximation lines fitted by expert (green line) and automatically (red line) A real life K-band SLAR image Other examples:a model of fully developed speckle, noise in ultrasound images, noise in sub-band images of hyperspectral remote sensing data. Vladimir Lukin

8. General Aspects of Image Processing • Noise is a phenomenon inherent for any imaging and RS system; • Noise presence is often a dominant factor that prevents solving final tasks for which obtained data are intended for; • These tasks could be: object detection and localization, sensed terrain classification, segmentation, edge detection, image filtering, segmentation, compression, classification, etc. • There are some methods of image processing that do not exploit information on noise type and characteristics (like standard median filter) but methods that use a priori information on noise type and characteristics are able to perform better; • There is a lot of different types of noise and their combinations. Vladimir Lukin

9. Methods for Blind Determination of Noise Type An automatic system to identify the nature of the degradation affecting an image and select the appropriate algorithm of its processing has been designed*.   The system is intended for identifying four sources of degradation made of three different noises: additive, multiplicative and impulse noise and a defocusing blur. Thirteen observation models have been deduced from these four sources: each noise type separately and different possible combinations. The system has been tested for different simulated data and provided appropriately high probability of correct identification of noise/distortion type. * Carton-Vandecandelaere, M.-P., Vozel, B., Klaine, L., Chehdi, K., Application to Multispectral Images of a Blind Identification System for Blur, Additive, Multiplicative and Impulse Noises. Proceedings of EUSIPCO, III, 2002, pp. 283-286. Vladimir Lukin

10. Methods for Blind Determination of Noise Statistical Characteristics There are methods for blind estimation of: • additive noise variance*; • mixed noise parameters***. • multiplicative noise variance**; They are based on different principles: • Curve fitting to scatter-plots; • Analysis of local estimate histograms; • Data analysis in spectral domain; • Bit-plane analysis, etc. * V. Lukin, S. Abramov, B. Vozel, K. Chehdi, J. Astola, Segmentation-based method for blind evaluation of noise variance in images, SPIE Journal on Applied Remote Sensing, Vol. 2, Aug. 2008, 15 p. ** Lukin V.V., Abramov S.K., Ponomarenko N.N., Vozel B., Chehdi K., Methods for blind evaluation of noise variance in multichannel optical and radar images, Telecommunications and Radio Engineering, Vol. 65 (6), 2006, pp. 509-537. *** A. Foi, Pointwise Shape-Adaptive DCT Image Filtering and Signal-Dependent Noise Estimation: Thesis for the degree of Doctor of Technology, Tampere University of Technology, Tampere, Finland, Dec. 2007. Vladimir Lukin

11. Edge Detection Example Principle: For most of edge detectors, a local parameter is calculated for each pixel using image values in a given pixel and its neighbors. Then this local parameter is compared to a preset threshold that takes into account noise type and statistics. Example: а bcd Noise free (a) and noisy (mixed multiplicative and impulse noise, ) (b) test images; edge maps obtained by Sobel (c) and normalized quasirange (d) edge detectors Vladimir Lukin

12. Edge Detection Requirement to accuracy: It is desirable to provide within the limits from till . Signal-dependent noise case: If a dependence of local variance on local mean is known in advance or pre-estimated, then a threshold for an edge detector should be set locally as or Procedure: • to estimate local mean; • to calculate ; • to determine the corresponding local threshold; • to obtain a detector output; • to compare it to the local threshold; • to undertake a decision. Vladimir Lukin

13. Requirements to Methods of Blind Determination of Noise Statistical Characteristics To produce unbiased estimates; To provide appropriately accurate estimates (characterized by small variance of noise parameter estimation); To be applicable to images of different content and to noise of different level and with different spatial characteristics (a wide range of noise parameters’ variation can be observed in practice); To be fast enough. Note: if an image is multichannel, processing is component-wise and it can be parallel. A question is what are appropriately accurate estimates? Vladimir Lukin

14. Image Filtering Examples:Test Images and Quantitative Criteria Goal: Consider how filtering efficiency depends upon accuracy of additive noise variance estimation. Test images: Quantitative criteria: • An error of additive noise variance estimation: • Variation of filtering efficiency Test01 Test02 Where is MSE provided under condition that is set as the parameter of a used filter. Vladimir Lukin

15. Image Filtering Examples:Modified Sigma Filter The plots vs for the test images for 100 and 225 Preliminary conclusion: For the image Test01, smaller output MSE can be reached if when better edge/detail/texture preservation is provided. The opposite situation is observed for the image Test02. If , smaller output MSE is produced. Thus, does not necessarily lead to the best efficiency of noise suppression (according to output MSE criterion).  If is within the limits from 0.8 to 1.2, then it is almost guaranteed that one is in the neighborhood of minimal output MSE. This is observed for other filters as well as for multiplicative noise. Vladimir Lukin

16. Image Filtering Examples:Blind Noise Type and Variance Retrieval Real-life image: Identification result: Image is identified as corrupted by mixed additive and impulse noise. Impulse noise is specific, spatially correlated, where several neighboring image pixels in rows are corrupted. a b Estimation result: The noise variance was estimated in automatic manner robust to the presence of impulses. The resulting estimate is 24.93. c The original NOAA image (a), filtered image (b), and the obtained impulse noise map (c) Vladimir Lukin

17. Image Filtering Examples:Blind Noise ACF Retrieval If fluctuative noise type is determined correctly and its variance is estimated with high enough accuracy, it becomes possible to blindly evaluate spatial correlation properties of noise as well*. These estimates can be further exploited in DCT based filtering. Taking into account noise spatial spectrum leads to PSNR increasing by 1...3 dB. Original SAR image (left) and the obtained filtered image (right) * V. Lukin, N. Ponomarenko, K. Egiazarian, J. Astola, Adaptive DCT-based filtering of images corrupted by spatially correlated noise, Proc. SPIE Conference Image Processing: Algorithms and Systems VI, Vol. 6812, 12 p., 2008. Vladimir Lukin

18. Image Filtering Examples:Fully Automatic Procedure Stages: Blind determination of noise type, additive or multiplicative (if there is no a priori information); Estimation of its statistical characteristics (variance); Estimation of spatial correlation properties of noise; Image filtering with taking into account the obtained estimates of noise characteristics. Features: This procedure is also applicable if both additive and multiplicative noise components have been identified in a given image. Note that DCT based filters can be easily modified to mixed additive and multiplicative or other signal-dependent noise.  Similar modifications are not a problem for the family of sigma filters and hard-switching locally adaptive filters. Vladimir Lukin

19. Image Filtering Examples:3D Processing of Multichannel Images Preliminary conclusion: Automatic procedures based on blind estimation described above can be used for both component-wise and 3D processing of multichannel images. Improvement of PSNR due to 3D processing is 0.6 dB in comparison to component-wise DCT based processing. a b (PSNR= 22.87 dB) The noise-free test image Goldhill (a), noisy image ( ) (b), intermediate output image(c), resulting output image(d) c (PSNR= 25.17 dB) d (PSNR= 30.01 dB) Vladimir Lukin

20. Noisy Image Compression:Quantitative Criteria The standard measures to characterize a compressed image quality - , where is the decompressed image; - - for 8 bits image representation. Alternative measures to characterize a compressed image quality - , where is the noise free image; - . It is more reasonable to characterize a compressed image quality by quantitative measures calculated with respect to the corresponding noise-free image (MSEnf, PSNRnf) rather than to the original noisy one (MSEor, PSNRor). Vladimir Lukin

21. Noisy Image Compression:Optimal Operation Point Optimal operation point (OOP): The argument of the curves MSEnf(CR), MSEnf(bpp) or MSEnf(QS) for which these curves reach theirs minima have been called optimal operation point (OOP): CROOP , bppOOP or QSOOP.. σ2n = 400 σ2n = 100 σ2n = 50 OOP is observed and commonly occurs to be more “obvious” for less complex content images and/or for rather intensive noise. Main idea: It is worth compressing a noisy image in the neighborhood of OOP. Main problem: In practice, noise-free image is not at disposal. Dependences MSEnf (QSn) for the noisy test gray-scale image Lena for different additive noise levels Vladimir Lukin

22. Noisy Image Compression:Peculiarities Why lossy (not lossless) compression? • Lossy compression is able to provide considerably larger CRs (compared to lossless coding) without degrading image resolution and introducing disturbing artefacts; • A positive effect of image filtering can be observed due to lossy compression if introduced losses mainly relate to noise removal and useful image content is preserved. The RS (Helsinki region) image corrupted by additive Gaussian noise with σ2n= 100 The decoded lossy compressed image (bpp = 0.75) Vladimir Lukin

23. Noisy Image Compression:Real-life Test Images The automatic procedure for attaining OOP have been also designed for pure multiplicative and Poisson noise. The corresponding homomorphic transform is to be applied (for making noise additive) before compression. Inverse homomorphic transform is to be carried out after decompression. The noisy real-life test image Frisco The compressed image (HBC strategy, AGU coder with QSA=35) Vladimir Lukin

24. Automatic Procedure for Lossy Compression Stages: Blind determination of noise type: additive, Poisson or multiplicative; Estimation of noise statistical characteristics; Carrying out proper forward homomorphic transform if noise is Poisson or multiplicative; calculation or estimation of variance after homomorphic transform (if needed); Carrying out lossy compression with providing OOP by either setting QS according to above recommendations or by finding proper bpp (for JPEG2000 or SPIHT coders, see details in the paper*). Note that are to be within the limits from till . . * N. Ponomarenko, V. Lukin, M. Zriakhov, K. Egiazarian, J. Astola, Lossy compression of images with additive noise, Proceedings of International Conference on Advanced Concepts for Intelligent Vision Systems, Antwerpen, Belgium, September, 2005, pp. 381-386. Vladimir Lukin

25. Accuracy of Blind Estimation Methods: TID2008 (Tampere Image Database) Test noise-free color images of TID2008 (http://www.ponomarenko.info/tid2008.htm) Vladimir Lukin

26. Accuracy of Blind Estimation Methods:Spatial Domain Noise variance estimates obtained by the methods* (left) and** (right) with nonoverlapping 5x5 pixels block size for red, green and blue components for the test image set corrupted by spatially uncorrelated additive noise with =65 * Lukin V.V., Abramov S.K., Vozel B., Chehdi K. Improved minimal inter-quantile distance method for blind estimation of noise variance in images, Proceedings of SPIE/EUROPTO Symposium on Satellite Remote Sensing, Florence, Italy, Sept 2007, SPIE Vol. 6748, 12 p. ** V. Lukin, S. Abramov, B. Vozel, K. Chehdi, J. Astola, Segmentation-based method for blind evaluation of noise variance in images, SPIE Journal on Applied Remote Sensing, Vol. 2, Aug. 2008, 15 p. (open access paper) Vladimir Lukin

27. Accuracy of Blind Estimation Methods:Wavelet Domain Noise variance estimates obtained by the method* for red, green and blue components for the test image set corrupted by spatially uncorrelated (left) and spatially correlated (right) additive noise with = 65 Be careful with spatially correlated noise !!! * L. Sendur, I.W. Selesnick, Bivariate shrinkage with local variance estimation, IEEE Signal Processing Letters, Vol. 9, No 12, 2002, pp. 438-441 (see also http://taco.polv.edu/WaveletSoftware/index.html) Vladimir Lukin

28. Accuracy of Blind Estimation Methods:Spatial Domain & Spatially Correletad Noise Noise variance estimates obtained by the method* with nonoverlapping 5x5 (left) and 7x7 (right) pixels block size for red, green and blue components for the test image set corrupted by spatially correlated additive noise with =65 Summary: It is better using 7x7 blocks to have less problems with (possibly) spatiall correlated noise. * Lukin V.V., Abramov S.K., Vozel B., Chehdi K. Improved minimal inter-quantile distance method for blind estimation of noise variance in images, Proceedings of SPIE/EUROPTO Symposium on Satellite Remote Sensing, Florence, Italy, Sept 2007, SPIE Vol. 6748, 12 p. Vladimir Lukin

29. Conclusions It is possible to design multi-stage automatic procedures of image processing based on blind determination of noise type and estimation of its characteristics. The accuracy of such estimates is commonly enough for carrying out efficient edge detection, filtering, and lossy compression in automatic manner. Currently only quite simple situations like identified pure additive or pure multiplicative noise are studied well enough; design and performance analysis for other noise environments are far from completeness. Special attention should be paid to multichannel and hyperspectral images for which, on one hand, necessity to make processing automatic is the most actual and, on the other hand, the situation with such design is the most complex. Vladimir Lukin