Signal- und Bildverarbeitung, 323.014 Image Analysis and Processing Arjan Kuijper 23.11.2006

1. Signal- und Bildverarbeitung, 323.014Image Analysis and ProcessingArjan Kuijper23.11.2006 Johann Radon Institute for Computational and Applied Mathematics (RICAM) Austrian Academy of Sciences Altenbergerstraße 56A-4040 Linz, Austria arjan.kuijper@oeaw.ac.at

2. Last week • The diffusion can be made locally adaptive to image structure. Three mathematical approaches are discussed: • PDE-based nonlinear diffusion, where the luminance function evolves as the divergence of some flow. • Evolution of the isophotes as an example of curve-evolution; • Variational methods, minimizing an energy functional defined on the image. • The nonlinear PDE's involve local image derivatives, and cannot be solved analytically. • Adaptive smoothing requires geometric reasoning to define the influence on the diffusivity coefficient. • The simplest equation is the equation proposed by Perona & Malik, where the variable conduction is a function of the local edge strength. Strong gradient magnitudes prevent the blurring locally, the effect is edge preserving smoothing. • The Perona & Malik equation leads to deblurring (enhancing edges) for edges larger than the turnover point k, and blurs smaller edges.

3. Today • Total Variation • Rudin – Osher – Fatemi (ROF) Model • Denoising • Edge preserving • Energy minimizing • Bounded variationTaken from:

4. Let the observed intensity function u0(x, y) denote the pixel values of a noisy image for x, yW. Let u(x, y) denote the desired clean image, sowith n additive white (0,)noise. • The constrained minimization problem is: Min(H|I1,I2) = Min (H - l1I1 - l2I2)

5. We arrive at the Euler-Lagrange equations 0= dH - l1dI1 - l2dI2 • Integrating the expression over W gives l1 = 0. So the average intensity is kept.

6. The solution procedure uses a parabolic equation with time as an evolution parameter, or equivalently, the gradient descent method. This means that we solve

7. We must compute l(t). We merely multiply the equation in W by (u - u0) and integrate by parts over W. We then have

8. The numerical method in two spatial dimensions is as follows:

9. The numerical approximation is

10. and ln is defined discreetly via • A step size restriction is imposed for stability:

11. Results

13. Original • Noisy • Wiener • TV

14. ¡ + + ¡ ¡ ¢ ¢ 1 1 ¡ ¡ k k k k ( ) ( ) ± ¸ ¸ L L r C D r L D r L L D L L L L L + ¡ + ¡ ¢ ¢ = = 0 0 t i j i j i j ; ; ; • Do it slightly sloppy: • demo

15. R d A B A B ­ < > ¢ = ­ ; ± ± H I < > 2 ( ( ) ) ( ) @ @ ¸ ± ± ¸ ± I H H I I 0 0 · ¡ ¡ ¡ < > ; u u u u u = = = = = t t t t t ± ± I I ; < > ; • Write • Then I(u) is a constant of motion: I(u0)=I(ut): • Then the pde converges to a minimum on the manifold given by the constraints:

16. Some extensions R ( j j ) Á d H r L ­ = ­ • A blurred noisy image:u0 = (Au)(x,y) + n(x,y)where A is a compact operator on L2. • Multiplicative noise & blurringu0 = [(Au)(x,y) ] n(x,y)u0 = (Au)(x,y) + [u(x,y) n(x,y)] • Smarter functional:

17. Recent Developments in Total Variation Image Restoration • T. Chan, S. Esedoglu, F. Park, A. YipHandbook of Mathematical Models in Computer VisionSince their introduction in a classic paper by Rudin, Osher and Fatemi, total variation minimizing models have become one of the most popular and successful methodology for image restoration. More recently, there has been a resurgence of interest and exciting new developments, some extending the applicabilities to inpainting, blind deconvolution and vector-valued images, while others offer improvements in better preservation of contrast, geometry and textures, in ameliorating the stair casing effect, and in exploiting the multi-scale nature of the models. In addition, new computational methods have been proposed with improved computational speed and robustness.

18. Properties • Total variation based image restoration models were first introduced by Rudin, Osher, and Fatemi (ROF) in their pioneering work on edge preserving image denoising. • It is one of the earliest and best known examples of PDE based edge preserving denoising. • It was designed with the explicit goal of preserving sharp discontinuities (edges) in images while removing noise and other unwanted fine scale detail. • Being convex, the ROF model is one of the simplest variational models having this most desirable property. • The revolutionary aspect of this model is its regularization term that allows for discontinuities but at the same time disfavors oscillations.

19. Properties • The constraint of the optimization forces the minimization to take place over images that are consistent with this known noise level. • The objective functional itself is called the total variation (TV) of the function u(x); for smooth images it is equivalent to the L1 norm of the derivative, and hence is some measure of the amount of oscillation found in the function u(x).

20. Remark • The step fromtois not trivial!

21. BV • The space of functions with bounded variation (BV) is an ideal choice for minimizers to the ROF model since BV provides regularity of solutions but also allows sharp discontinuities (edges). Many other spaces like the Sobolev space W1,1 do not allow edges. • Before defining the space BV, we formally state the definition of TV as:where and is a bounded open set. • We can now define the space BV as • Thus, BV functions amount to L1 functions with bounded TV semi-norm.

22. TV & Contours @ f f ! ¡ v v · = = @ f t w • Why does this work?Ignoring the constraints we get • The TV norm of f can be obtained by integrating along all contours of f = cfor all values of c. • Thus, one can view TV as controlling both the size of the jumps in an image and the geometry of the isophotes (level sets).

23. Caveats • While using TV-norm as regularization can reduce oscillations and regularize the geometry of level sets without penalizing discontinuities, it possesses some properties which may be undesirable under some circumstances. • Loss of contrast:The total variation of a function, defined on a bounded domain, is decreased if we re-scale it around its mean value in such a way that the difference between the maximum and minimum value (contrast) is reduced. • Loss of geometry:In addition to loss of contrast, the TV of a function may be decreased by reducing the length of each level set. • Staircasing:This refers to the phenomenon that the denoised image may look blocky (piecewise constant). • Loss of Texture:Although highly effective for denoising, the TV norm cannot preserve delicate small scale features like texture.

24. Summary • Total variation minimizing models have become one of the most popular and successful methodology for image restoration. • ROF is one of the earliest and best known examples of PDE based edge preserving denoising. • It was designed with the explicit goal of preserving sharp discontinuities (edges) in images while removing noise and other unwanted fine scale detail. • However, it has some drawbacks as shown in the previous slides

25. Next week • Non-linear diffusion:Mean curvature motion • Curve evolution • Denoising • Edge preserving • Implementation • Isophote vs. image implementation