Graph Spectral Image Smoothing

1. GBR 2007 Graph Spectral Image Smoothing Fan Zhang, Edwin R. Hancock Computer Vision and Pattern Recognition Group Department of Computer Science University of York, UK

2. Overview • Literature and Motivation • Method • Graph representation of an image • Diffusion on a graph • Numerical implementation • Relationship to other methods (e.g. anisotropic diffusion, low-pass filtering, normalised cut) • Experiments • Conclusion

3. Literature • General diffusion-based partial differential equation (PDE): • If D=1, isotropic linear diffusion (Gaussian filter). • If D is a scalar function, inhomogeneous diffusion, e.g. Perona-Malik diffusion: • If D is a diffusion tensor, anisotropic diffusion, e.g. (Weickert, etc).

4. Motivation: Why Graph Diffusion? • Assumption for continuous PDEs • An image is a continuous function on R2 • Consider discretisation for numerical implementation. • Weakness: • Noisy images may not be sufficiently smooth to give reliable derivatives. • Fast, accurate, and stable implementation is difficult to achieve. • Solution: diffusion on graphs • An image is a smooth function on a graph. • Purely combinatorial operators that require no discretisation are used.

5. Aim in this paper Explore if we can use graph spectral methods to solve the diffusion equation commencing from a discrete setting.

6. Steps • Set-up diffusion process as problem involving weighted graph, where anisotropic smoothing is modelled by an edge-weight matrix. • Diffusion is heat-flow on the associated graph-structure. • Nodes are pixels, and diffusion of grey-scale information is along edges with time. • Solution is given by exponentiating the spectrum of the associated Laplacian matrix.

7. Graph Representation of Images • An image is represented using a weighted graph . • The nodes V are the pixels. An edge is formed between two nodes vi and vj . The weight of an edge, , is denoted by .

8. Graph Edge Weight • Characterise each pixel by a window of neighbors instead of using a single pixel alone. • The similarity between nodes vi and vj is measured by the Gaussian weighted Euclidean distance between windows, i.e. • Thus, edge weight is computed by

9. Laplacian of a graph • Weighted graph G = (V, E,W) with node-set V, edge-set E, adjacency weight matrix W and degree matrix D. • Diagonal degree matrix • Laplacian • Normalised Laplacian

10. Laplacian spectrum • Spectral decomposition of Laplacian Eigenvalues Eigenvectors

11. Graph Heat Kernel Heat equation on graph Solution of heat equation is heat kernel When t tends to 0, When t large,

12. Graph Heat Kernel Heat equation on graph Solution of heat equation is heat kernel When t tends to 0, When t large,

13. Graph Heat Kernel Heat equation on graph Solution of heat equation is heat kernel When t tends to 0, When t large,

14. Graph Heat Kernel Heat equation on graph Solution of heat equation is heat kernel When t tends to 0, When t large,

15. Lazy random walk on graph • Moves between nodes with probability , remains static with probability • Transition probability matrix • Let , then after N-steps of walk

16. Lazy random walk on graph • Moves between nodes with probability , remains static with probability • Transition probability matrix • Let , then after N-steps of walk

17. Continuous time random walk • Let we the state probability vector at time t. Probability of visiting ith node after time t is • State-vector is solution of the differential equation • Solution given by heat-kernel

18. Continuous time random walk • Let we the state probability vector at time t. Probability of visiting ith node after time t is • State-vector is solution of the differential equation • Solution given by heat-kernel

19. …we are going to use the random walk to model image smoothing via anisotropic diffusion. Transition probability is small when there is strong evidence of edge-structure, and small in uniform regions. Heat-kernel is used to smooth pixel values, and this means pixel values are related to state probabilities of random walk.

20. Anisotropic diffusion as heat flow on a graph • Vertices: pixel values • Edge weight: c.f. thermal conductivity • Diffusion-equation for pixel values: Pixel values stored as vector of stacked image columns. Connectivity structure encoded by Laplacian.

21. Graph spectral image smoothing • Solution of diffusion equation: • Small t behaviour

22. Graph spectral image smoothing • Solution of diffusion equation: • Small t behaviour Heat kernel weights the averaging of grey-scale values

23. Graph spectral image smoothing • Solution of diffusion equation: • Small t behaviour Effect of original pixel value decreases with time. Weighted average of neighbouring values.

24. Meaning What does this mean? - heat kernel weights initial pixel values and smooths image. - pixel values are strongly influenced by neighbours when they are connected by edges of large weight (small difference in grey-scale value). - time plays the role of scale, the longer the process is run the greater the smoothing.

25. Numerical Implementation • Laplacian L is very large, e.g. . • It is not tractable to calculate the heat kernel by matrix exponential. • Solution: Krylov subspace projection technique • Idea: approximate by an element of Krylov space • Approximation scheme: • : first column of identity matrix • : orthonormal basis of Krylov space • : Hessenberg matrix resulting from Lanczos process

26. Relation to Anisotropic Diffusion • An image is a 2D manifold M embedded in R3, i.e.

27. Relation to Signal Processing • An extension of Fourier analysis to images defined on graphs. • Decomposition of the image into a linear combination of eigenvectors • Attenuate the terms associating with high eigenvalues. Thus, Graph heat kernel can be regarded as a low-pass filter kernel.

28. Relation to Spectral Clustering • Large t behavior of the heat kernel is governed by the second eigenvector of the graph Laplacian [Normalised cut (Shi and Malik)]. • The algorithm projects the noisy image onto the space spanned by the first few eigenvectors. [Laplacian eigenmap (Belkin and Niyogi)]. • The second or first few eigenvectors of the Laplacian encode the segment-structure of the image.

29. Results It takes 3~6 seconds to process a 256 square image.

30. (a) Noisy Lenna (b) Zoomed portion (c) Our graph smoothing (d) Regularised Perona-Malik diffusion (RPM) (e) Nonlinear complex ramp-preserving diffusion (NCRD) (f) Coherence-enhancing diffusion (CED) (g) Total-variation denoising (TV) (h) Wavelet filtering (WAVELET)

32. Root-Mean-Square Error comparison

33. Root-Mean-Square Error comparison

36. Conclusion • Graph representation of images is a natural, discrete and effective way for image processing. • Diffusion on graphs can be efficiently used for image smoothing. The diffusion is determined by spectra of the graphs. • Graph smoothing can be readily solved using Krylov subspace technique. • Graph-spectral smoothing has close relationships with the continuous anisotropic diffusion, low-pass filtering and spectral clustering.