A T ransform-based V ariational F ramework

1. Pixel Club, November, 2013 A Transform-based Variational Framework Guy Gilboa

2. In a Nutshell Fourier inspiration: Fourier Scale Fourier Scale Spectral TV Scale TV Scale TV Flow

3. Relations to eigenvalue problems • General linear: (L linear operator) • Functional based Linear Nonlinear

4. What can a transform-based approach give us? • Scale analysis based on the spectrum. • New types of filtering– otherwise hard to design: nonlinear LPF, BPF, HPF. • Nonlinear spectral theory – relation to eigenfunctions and eigenvalues. • Deeper understandingof the regularization, optimal design with respect to data, noise and artifacts.

5. Examples of spectral applications today:Eigenfunctions for 3D processing Taken from L Cai, F Da, “Nonrigid deformation recovery..”, 2012. Taken from Zhang et al, “Spectral mesh processing”, 2010.

6. Image Segmentatoin Eigenvectors of the graph Laplacian [Taken from I. Tziakos et al, “Color image segmentation using Laplacianeigenmaps”, 2009 ]

7. Some Related Studies • Andreu, Caselles, Belletini, Novagaet al 2001-2012– TV flow theory. • Steidl et al 2004 – Wavelet – TV relation • Brox-Weickert 2006 – scale through TV-flow • Luo-Aujol-Gousseau 2009 – local scale measures • Benning-Burger 2012 – ground states (nonlinear spectral theory) • Szlam-Bresson – Cheeger cuts. • Meyer, Vese, Osher, Aujol, Chambolle, G. and many more – structure-texture decomposition. • Chambolle-Pock 2011, Goldstein-Osher2009– numerics.

8. Scale Space – a Natural Way to Define Scale We’ll talk specifically about total-variation (TV-flow, Andreuet al - 2001): Scale space as a gradient descent:

9. TV-Flow:A behavior of a disk in time[Andreu-Caselleset al–2001,2002, Bellettini-Caselles-Novaga-2002, Meyer-2001] Center of disk, first and second time derivatives: t … …

10. Spectral TV basic framework Phi(t) definition

11. Reconstruction Reconstruction formula Th. 1: The reconstruction formula recovers

12. Spectral response Spectrum S(t) as a function of time t: f S(t) t

13. Spectrum example f S(t)

14. Dominant scales

15. Eigenvalue problem The nonlinear eigenvalue problem with respect to a functional J(u) is defined by: We’ll show a connection to the spectral components .

16. Solution of eigenfunctions Th. 2: For is an eigenfunction with eigenvalue then:

17. What are the TV eigenfunctions? In 2D, is a characteristic function of a convex set. I then is an eigenfunction. [Giusti-1978], [Finn-1979],[Alter-Caselles-Chambolle-2003].

18. Filtering H(t) Let H(t) be a real-valued function of t. The filtered spectral response is The filtered spatial response is

19. Filtering, example 1: TV Band-Pass and Band-Stop filters f S(t) Band-pass Band-stop

20. Disk band-pass example S(t)

21. We have the basic framework

22. Numerics • Many ways to solve. • Variational approach was chosen: • Currently use Chambolle’s projection algorithm (some spikes using Split-Bregman, under investigation). • In time: • 2nd derivative - central difference • 1st derivative - forward differnce • Discrete reconstruction algorithm proved for any regularizing scale-space (Th. 4).

23. TV-Flow as a LPF Th. 3: The solution of the TV-flow is equivalent to spectral filtering with:

24. Nonlocal TV • Reminder: NL-TV (G.-Osher2008): Gradient Functional

25. Spectral NL-TV? • The framework can fit in principle many scale-spaces, like NL-TV flow. We can obtain a one-homogeneous regularizer. • What is a generalized nonlocal disk? • What are possible eigenfunctions? • It is expected to be able to process better repetitive textures and structures.

26. Sparseness in the TV sense • Sparse spectrum – the signal has only a few dominant scales. • Can be a large objects • Or many small ones (here TV energy is large) • Natural images – are not very sparse in general

27. Noise Spectrum Various standard deviations: S(t)

28. Noise + signal • Not additive. • Spreads original image spectrum. • Needs to be investigated. Band-pass filtered u f f-u

29. Spectral Beltrami Flow? Original Beltrami Flow Spectral Beltrami Initial trials on Beltrami flow with parameterization such that it is closer to TV Difference images: Spectral Beltrami • Keeps sharp contrast • Breaks extremum principle Values along one line (Green channel)

30. Segmentation priors • Swoboda-Schnorr2013 – convex segmentation with histogram priors. • We can have 2D spectrum with histograms • Use it to improve segmentation S(t,h)

31. Texture processing • Many texture bands • We can filter and manipulate certain bands and reconstruct a new image. • Generalization of structure-texture decomposition.

32. Processing approach • Deconstruct the image into bands • Identify salient textures • Amplify / attenuate / spatial process the bands. • Reconstruct image with processed bands

33. Color formulation • Vectorial TV – all definitions can be generalized in a straightforward manner to vector-valued images. • Bresson-Chan (2008) definition and projection algorithm is used for the numerics.

34. Orange example

35. Orange – close up Original Modes 2,3=0 Modes 2-5=x1.5

36. Selected phi(t) modes (1, 5, 15, 40) f residual

37. Old man

38. Old man – close up Original 2 modes attenuated 7 modes attenuated

39. Old Man - First 3 Modes Modes: 1 2 3

40. Take Home Messages • Introduction of a new TV transformand TV spectrum. • Alternative way to understand and visualize scales in the image. • Highly selective scale separation, good for processing textures. • Can be generalized to other functionals.

41. Thanks! • Refs. Google “Guy Gilboa publications” • Preliminary ideas are in SSVM 2013 paper. • Most material is in CCIT Tech report 803. • Up-to-date and organized - submitted journal version – contact me.