Michael Elad The CS Department The Technion – Israel Institute of technology Haifa 32000, Israel

1. Recent Trends in Signal Representations and Their Role in Image Processing Michael Elad The CS Department The Technion – Israel Institute of technology Haifa 32000, Israel Research Day on Image Analysis and Processing Industrial Affiliate Program (IAP) The Computer Science Department – The Technion January 27th, 2005

2. Today’s Talk is About Sparsity and Overcompleteness • We will show today that • Sparsity & Overcompleteness can & should be used to design new powerful signal/image processingtools (e.g., transforms, priors, models, …), • The obtained machinery works very well – we will show these ideas deployed to applications. Recent Advances in Signal Representations and Applications to Image Processing

3. Welcome to Sparseland Agenda • A Visit to Sparseland • Motivating Sparsity & Overcompleteness • 2. Problem 1: Transforms & Regularizations • How & why should this work? • 3. Problem 2: What About D? • The quest for the origin of signals • 4. Problem 3: Applications • Image filling in, denoising, separation, compression, … Recent Advances in Signal Representations and Applications to Image Processing

4. Every column in D (dictionary) is a prototype signal (Atom). N • The vector is generated randomly with few non-zeros in random locations and random values. N A sparse & random vector K A fixed Dictionary Generating Signals in Sparseland M Recent Advances in Signal Representations and Applications to Image Processing

5. M Multiply by D Sparseland Signals Are Special • Simple:Every generated signal is built as a linear combination of fewatoms from our dictionaryD • Rich:A general model: the obtained signals are a special type mixture-of-Gaussians (or Laplacians). Recent Advances in Signal Representations and Applications to Image Processing

6. M • Assume that x is known to emerge from . . M • How about “Given x, find the α that generated it in ” ? M T Transforms in Sparseland ? • We desire simplicity, independence, and expressiveness. Recent Advances in Signal Representations and Applications to Image Processing

7. We need to solve an under-determined linear system of equations: • Sparsity is measured using the L0 norm: Known So, In Order to Transform … • Among all (infinitely many) possible solutions we want the sparsest !! Recent Advances in Signal Representations and Applications to Image Processing

8. A sparse & random vector Multiply by D • Is ? Under which conditions? • Are there practical ways to get ? 4 Major Questions Signal’s Transform inSparseland • How effective are those ways? • How would we get D? Recent Advances in Signal Representations and Applications to Image Processing

9. M • Assume that x is known to emerge from . • Suppose we observe , a “blurred” and noisy version of x with . How will we recover x? M Q Noise Inverse Problems in Sparseland ? • How about “find the α that generated the x …” again? Recent Advances in Signal Representations and Applications to Image Processing

10. A sparse & random vector “blur” by H Multiply by D • Is ? How far can it go? 4 Major Questions (again!) • Are there practical ways to get ? • How effective are those ways? • How would we get D? Inverse Problems in Sparseland ? Recent Advances in Signal Representations and Applications to Image Processing

11. Sparselandis HERE Sparsity. Sparsity. Overcompleteness. Sparsity & Overcompleteness. Independence and Sparsity. Back Home … Any Lessons? Several recent trends worth looking at: • JPEG to JPEG2000 - From (L2-norm) KLT to wavelet and non-linear approximation • From Wiener to robust restoration – From L2-norm (Fourier) to L1. (e.g., TV, Beltrami, wavelet shrinkage …) • From unitary to richer representations – Frames, shift-invariance, bilateral, steerable, curvelet • Approximation theory – Non-linear approximation • ICA and related models Recent Advances in Signal Representations and Applications to Image Processing

12. M The Sparseland model for signals is interesting since it is rich and yet every signal has a simple description Who cares? So, what are the implications? Sparsity? looks complicated To Summarize so far … We do! this model is relevant to us just as well. We need to solve (or approximate the solution of) a linear system with more unknowns than equations, while finding the sparsest possible solution • Practical solvers? • How will we get D? • Applications? Recent Advances in Signal Representations and Applications to Image Processing

13. T Q Agenda • 1. A Visit to Sparseland • Motivating Sparsity & Overcompleteness • 2. Problem 1: Transforms & Regularizations • How & why should this work? • 3. Problem 2: What About D? • The quest for the origin of signals • 4. Problem 3: Applications • Image filling in, denoising, separation, compression, … Recent Advances in Signal Representations and Applications to Image Processing

14. Our dream for Now: Find the sparsest solution of Known Put formally, known Lets Start with the Transform … Recent Advances in Signal Representations and Applications to Image Processing

15. Questions to Address • Is ? Under which conditions? • Are there practical ways to get ? • How effective are those ways? • How would we get D? 4 Major Questions Recent Advances in Signal Representations and Applications to Image Processing

16. M Multiply by D Suppose we can solve this exactly Why should we necessarily get ? It might happen that eventually . Question 1 – Uniqueness? Recent Advances in Signal Representations and Applications to Image Processing

17. Definition:Given a matrix D, =Spark{D} is the smallestand and number of columns that are linearly dependent. Donoho & Elad (‘02) Example: Spark = 3 Matrix “Spark” Rank = 4 Recent Advances in Signal Representations and Applications to Image Processing

18. Suppose this problem has been solved somehow Uniqueness If we found a representation that satisfy Then necessarily it is unique (the sparsest). Donoho & Elad (‘02) M This result implies that if generates signals using “sparse enough” , the solution of the above will find it exactly. Uniqueness Rule Recent Advances in Signal Representations and Applications to Image Processing

19. Multiply by D Are there reasonable ways to find ? Question 2 – Practical P0 Solver? M Recent Advances in Signal Representations and Applications to Image Processing

20. Matching Pursuit (MP) Mallat & Zhang (1993) • The MPis a greedy algorithm that finds one atom at a time. • Next steps: given the previously found atoms, find the next one to best fit … • Step 1: find the one atom that best matches the signal. • The Orthogonal MP (OMP) is an improved version that re-evaluates the coefficients after each round. Recent Advances in Signal Representations and Applications to Image Processing

21. Instead of solving Solve Instead Basis Pursuit (BP) Chen, Donoho, & Saunders (1995) • The newly defined problem is convex. • It has a Linear Programming structure. • Very efficient solvers can be deployed: • Interior point methods [Chen, Donoho, & Saunders (`95)] , • Sequential shrinkage for union of ortho-bases [Bruce et.al. (`98)], • If computing Dx and DT are fast, based on shrinkage [Elad (`05)]. Recent Advances in Signal Representations and Applications to Image Processing

22. Multiply by D How effective are the MP/BP in finding ? Question 3 – Approx. Quality? M Recent Advances in Signal Representations and Applications to Image Processing

23. D = DT DTD Evaluating the “Spark” • Compute Assume normalized columns • The Mutual Incoherence M is the largest entry in absolute value outside the main diagonal of DTD. • The Mutual Incoherence is a property of the dictionary (just like the “Spark”). The smaller it is, the better the dictionary. Recent Advances in Signal Representations and Applications to Image Processing

24. BP and MP Equivalence Equivalence Given a signal x with a representation , Assuming that , BP and MP are Guaranteed to find the sparsest solution. Donoho & Elad (‘02) Gribonval & Nielsen (‘03) Tropp (‘03) Temlyakov (‘03) • MP is typically inferior to BP! • The above result corresponds to the worst-case. • Average performance results are available too, showing much better bounds [Donoho (`04), Candes et.al. (`04), Elad and Zibulevsky (`04)]. Recent Advances in Signal Representations and Applications to Image Processing

25. A sparse & random vector “blur” by H Multiply by D What About Inverse Problems? • We had similar questions regarding uniqueness, practical solvers, and their efficiency. • It turns out that similar answers are applicable here due to several recent works [Donoho, Elad, and Temlyakov (`04), Tropp (`04), Fuchs (`04)]. Recent Advances in Signal Representations and Applications to Image Processing

26. Find the sparsest solution? Why works so well? What next? To Summarize so far … The Sparseland model for signals is relevant to us. We can design transforms and priors based on it Use pursuit Algorithms A sequence of works during the past 3-4 years gives theoretic justifications for these tools behavior • How shall we find D? • Will this work for applications? Which? Recent Advances in Signal Representations and Applications to Image Processing

27. Agenda • 1. A Visit to Sparseland • Motivating Sparsity & Overcompleteness • 2. Problem 1: Transforms & Regularizations • How & why should this work? • 3. Problem 2: What About D? • The quest for the origin of signals • 4. Problem 3: Applications • Image filling in, denoising, separation, compression, … Recent Advances in Signal Representations and Applications to Image Processing

28. M Multiply by D Problem Setting • Given these P examples and a fixed size [NK] dictionary D: • Is D unique? • How would we find D? Recent Advances in Signal Representations and Applications to Image Processing

29. M • “Rich Enough”: The signals from could be clustered to groups that share the same support. At least L+1 examples per each are needed. Uniqueness? If is rich enough* and if then D is unique. Uniqueness Aharon, Elad, & Bruckstein (`05) Comments: • This result is proved constructively, but the number of examples needed to pull this off is huge – we will show a far better method next. • A parallel result that takes into account noise can be constructed similarly. Recent Advances in Signal Representations and Applications to Image Processing

30. D X  A Each example is a linear combination of atoms from D Each example has a sparse representation with no more than L atoms Practical Approach – Objective Recent Advances in Signal Representations and Applications to Image Processing

31. D Initialize D Sparse Coding Use MP or BP X Dictionary Update Column-by-Column by SVD computation The K–SVD Algorithm – General Aharon, Elad, & Bruckstein (`04) Recent Advances in Signal Representations and Applications to Image Processing

32. D For the jth example we solve X K–SVD Sparse Coding Stage Pursuit Problem !!! Recent Advances in Signal Representations and Applications to Image Processing

33. Gk: The examples in and that use the column dk. Let us fix all A apart from the kth column and seek both dk and the kth column to better fit the residual! The content of dk influences only the examples in Gk. K–SVD Dictionary Update Stage D Recent Advances in Signal Representations and Applications to Image Processing

34. We should solve: dk is obtained by SVD on the examples’ residual in Gk. K–SVD Dictionary Update Stage D ResidualE Recent Advances in Signal Representations and Applications to Image Processing

35. How D can be found? Will it work well? What next? To Summarize so far … The Sparseland model for signals is relevant to us. In order to use it effectively we need to know D Use the K-SVD algorithm We have established a uniqueness result and shown how to practically train D using the K-SVD • Deploy to applications • Generalize in various ways: multiscale, non-negative factorization, speed-up, … • (c) Performance guarantees? Recent Advances in Signal Representations and Applications to Image Processing

36. Agenda • 1. A Visit to Sparseland • Motivating Sparsity & Overcompleteness • 2. Problem 1: Transforms & Regularizations • How & why should this work? • 3. Problem 2: What About D? • The quest for the origin of signals • 4. Problem 3: Applications • Image filling in, denoising, separation, compression, … Recent Advances in Signal Representations and Applications to Image Processing

37. Source Outcome Inpainting (1) [Elad, Starck, & Donoho (`04)] predetermined dictionary: Curvelet+DCT Recent Advances in Signal Representations and Applications to Image Processing

38. Source Outcome Inpainting (2) predetermined dictionary: Curvelet+DCT Recent Advances in Signal Representations and Applications to Image Processing

39. Overcomplete Haar 10,000 sample 8-by-8 images. 441 dictionary elements. Approximation method: MP. K-SVD: K-SVD on Images [Aharon, Elad, & Bruckstein (`04)] Recent Advances in Signal Representations and Applications to Image Processing

40. 90% missing pixels K-SVD Results Average # coefficients 4.27 RMSE: 25.32 Haar Results Average # coefficients 4.48 RMSE: 28.97 Filling-In Missing Pixels Recent Advances in Signal Representations and Applications to Image Processing

41. Filling-In Missing Pixels Recent Advances in Signal Representations and Applications to Image Processing

42. Haar dictionary DCT dictionary K-SVD dictionary OMP with error bound Compression Recent Advances in Signal Representations and Applications to Image Processing

43. Compression K-SVD Dictionary BPP = 0.502 RMSE = 7.67 Haar Dictionary BPP = 0.784 RMSE = 8.41 DCT Results BPP = 1.01 RMSE = 8.14 Recent Advances in Signal Representations and Applications to Image Processing

44. Compression Recent Advances in Signal Representations and Applications to Image Processing

45. Today We Have Discussed • 1. A Visit to Sparseland • Motivating Sparsity & Overcompleteness • 2. Problem 1: Transforms & Regularizations • How & why should this work? • 3. Problem 2: What About D? • The quest for the origin of signals • 4. Problem 3: Applications • Image filling in, denoising, separation, compression, … Recent Advances in Signal Representations and Applications to Image Processing

46. There are difficulties in using them! The dream? Summary Sparsity and Over- completeness are important ideas that can be used in designing better tools in signal/image processing • We are working on resolving those difficulties: • Performance of pursuit alg. • Speedup of those methods, • Training the dictionary, • Demonstrating applications, • … Future transforms and regularizations will be data-driven, non-linear, overcomplete, and promoting sparsity. Recent Advances in Signal Representations and Applications to Image Processing