Analysis of the Basis Pustuit Algorithm and Applications*

1. Analysis of the Basis Pustuit Algorithm and Applications* Michael Elad The Computer Science Department – (SCCM) program Stanford University Multiscale Geometric Analysis Meeting IPAM - UCLA January 2003 * Joint work with: Alfred M. Bruckstein – CS, Technion David L. Donoho – Statistics, Stanford

2. Collaborators Dave Donoho Statistics Department Stanford Freddy Bruckstein CS Department Technion M.Elad and A.M. Bruckstein, “A Generalized Uncertainty Principle and Sparse Representation in Pairs of Bases", IEEE Trans. On Information Theory, Vol. 48, pp. 2558-2567, September 2002. D. L. Donoho and M. Elad, “Maximal Sparsity Representation via l1 Minimization”, to appear in Proceedings of the Naional Academy of Science.  Sparse representation and the Basis Pursuit Algorithm

3. General • Basis Pursuit algorithm [Chen, Donoho and Saunders, 1995]: • Effective for finding sparse over-complete representations, • Effective for non-linear filtering of signals. • Our work (in progress) – better understanding BP and deploying it in signal/image processing and computer vision applications. • We believe that over-completeness has an important role! • Today we discuss the analysis of the Basis Pursuit algorithm, giving conditions for its success. We then show some stylized applications exploiting this analysis. Sparse representation and the Basis Pursuit Algorithm

4. Understanding the BP Using the BP - Application Agenda • Introduction • Previous and current work • 2. Two Ortho-Bases • Uncertainty  Uniqueness  Equivalence • 3. Arbitrary dictionary • Uniqueness  Equivalence • 4. Stylized Applications • Separation of point, line, and plane clusters • 5. Discussion Sparse representation and the Basis Pursuit Algorithm

5. Define the forward and backward transforms by (assume one-to-one mapping) s – Signal (in the signal space CN) – Representation (in the transform domain CL, LN) Transforms • Transforms T in signal and image processing used for coding, analysis, speed-up processing, feature extraction, filtering, … Sparse representation and the Basis Pursuit Algorithm

6. Special interest - linear transforms (inverse) General transforms L Linear Square  = s N Unitary  Atoms from a Dictionary • In square linear transforms,  is an N-by-N & non-singular. The Linear Transforms Sparse representation and the Basis Pursuit Algorithm

7. Lack Of Universality • Many available square linear transforms – sinusoids, wavelets, packets, … • Successful transform – one which leads to sparse representations. • Observation: Lack of universality - Different bases good for different purposes. • Sound = harmonic music (Fourier) + click noise (Wavelet), • Image = lines (Ridgelets) + points (Wavelets). • Proposed solution: Over-complete dictionaries, and possibly combination of bases. Sparse representation and the Basis Pursuit Algorithm

8. 1 0.1 1 0.05 0 10 1.0 0 0.5 -0.05 -0.1 -2 10 0 0.3 0.1 2 0.1 0.05 0 -4 10 0 + -0.1 -0.05 |T{1+0.32+0.53+0.054}| -0.2 -0.1 -6 10 3 -0.3 0.5 |T{1+0.32}| -0.4 -8 10 -0.5 -0.6 -10 4 10 DCT Coefficients 0 20 40 60 80 100 120 1 0.05 0 64 128 0.5 0 Example – Composed Signal Sparse representation and the Basis Pursuit Algorithm

9. 0 10 -2 10 -4 10 -6 10 -8 10 -10 10 0 40 80 120 160 200 240 Spike (Identity) Coefficients DCT Coefficients Example – Desired Decomposition Sparse representation and the Basis Pursuit Algorithm

10. Combined representation per a signal s by • Non-unique solution  - Solve for maximal sparsity Matching Pursuit • Given d unitary matrices {k, 1kd}, define a dictionary  = [1, 2 , … d] [Mallat & Zhang (1993)]. • Hard to solve – a sub-optimal greedy sequential solver: “Matching Pursuit algorithm” . Sparse representation and the Basis Pursuit Algorithm

11. 0 10 -2 10 -4 10 -6 10 -8 10 -10 10 0 50 100 150 200 250 Dictionary Coefficients Example – Matching Pursuit Sparse representation and the Basis Pursuit Algorithm

12. Facing the same problem, and the same optimization task [Chen, Donoho, Saunders (1995)] • Hard to solve – replace the l0 norm by an l1 : “Basis Pursuit algorithm” Basis Pursuit (BP) • Interesting observation: In many cases it successfully finds the sparsest representation. Sparse representation and the Basis Pursuit Algorithm

13. Example – Basis Pursuit Dictionary Coefficients Sparse representation and the Basis Pursuit Algorithm

14. 0P<1 P=1 P>1 Why ? 2D-Example Sparse representation and the Basis Pursuit Algorithm

15. Wavelet part of the noisy image Ridgelets part of the image Example – Lines and Points* Original image * Experiments from Starck, Donoho, and Candes - Astronomy & Astrophysics 2002. Sparse representation and the Basis Pursuit Algorithm

16. Original Residual = + Ridgelets Curvelets Wavelet + + Example – Galaxy SBS 0335-052* * Experiments from Starck, Donoho, and Candes - Astronomy & Astrophysics 2002. Sparse representation and the Basis Pursuit Algorithm

17. From Transforming to Filtering Non-Linear Filtering via BP • Through the previous example – Basis Pursuit can be used for non-linear filtering. • What is the relation to alternative non-linear filtering methods, such as PDE based methods (TV, anisotropic diffusion …), Wavelet denoising? • What is the role of over-completeness in inverse problems? Sparse representation and the Basis Pursuit Algorithm

18. Provingtightness of E-B bounds [Feuer & Nemirovski] Improving previous results – tightening the bounds [Elad and Bruckstein] Proven equivalence between P0 and P1 under some conditions on the sparsity of the representation, and for dictionaries built of two ortho-bases [Donoho and Huo] Relaxing the notion of sparsity from l0 to lp norm [Donoho and Elad] 2000 2001 2002 1998 1999 time Generalized all previous results to any dictionary and Applications [Donoho and Elad] (Our) Recent Work Sparse representation and the Basis Pursuit Algorithm

19. Our goal is the solution of • Basis Pursuit alternative is to solve instead Before we dive … • Given a dictionary and a signal s, we want to find the sparse “atom decomposition” of the signal. • Our focus for now: Why should this work? Sparse representation and the Basis Pursuit Algorithm

20. N N N Agenda • 1. Introduction • Previous and current work • 2. Two Ortho-Bases • Uncertainty  Uniqueness  Equivalence • 3. Arbitrary dictionary • Uniqueness  Equivalence • 4. Stylized Applications • Separation of point, line, and plane clusters • 5. Discussion Sparse representation and the Basis Pursuit Algorithm

21. Our Objective is Our Objective Given a signal s, and its two representations using  and , what is the lower bound on the sparsity of both? We will show that such rule immediately leads to a practical result regarding the solution of the P0 problem. Sparse representation and the Basis Pursuit Algorithm

22. Properties • Generally, . • For Fourier+Trivial (identity) matrices . • For random pairs of ortho-matrices . Mutual Incoherence • M – mutual incoherence between  and . • M plays an important role in the desired uncertainty rule. Sparse representation and the Basis Pursuit Algorithm

23. * • Examples: • =: M=1, leading to . • =I, =FN (DFT): , leading to . Theorem 1 * Donoho & Huo obtained a weaker bound Uncertainty Rule Sparse representation and the Basis Pursuit Algorithm

24. For N=1024, . • The signal satisfying this bound: Picket-fence 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 200 400 600 800 1000 Example =I, =FN (DFT) Sparse representation and the Basis Pursuit Algorithm

25. Given a unit norm signal s, assume we hold two different representations for it using  • Thus • Based on the uncertainty theorem we just got: Towards Uniqueness Sparse representation and the Basis Pursuit Algorithm

26. In words: Any two different representations of the same signal CANNOT BE JOINTLY TOO SPARSE. * If we found a representation that satisfy Then necessarily it is unique (the sparsest). Theorem 2 * Donoho & Huo obtained a weaker bound Uniqueness Rule Sparse representation and the Basis Pursuit Algorithm

27. We are interested in solving • Somehow we obtain a candidate solution . • The uniqueness theorem tells us that a simple test on ( ) could tell us if it is the solution of P0. Uniqueness Implication • However: • If the test is negative, it says nothing. • This does not help in solving P0. • This does not explain why P1 may be a good replacement. Sparse representation and the Basis Pursuit Algorithm

28. We are going to solve the following problem Equivalence - Goal • The questions we ask are: • Will the P1 solution coincide with the P0 one? • What are the conditions for such success? • We show that if indeed the P0 solution is sparse enough, then P1 solver finds it exactly. Sparse representation and the Basis Pursuit Algorithm

29. Given a signal s with a representation , Assuming a sparsity on  such that (assume k1<k2) k1 non-zeros k2 non-zeros If k1 and k2 satisfy then P1 will find the correct solution. Theorem 3 * A weaker requirement is given by * Donoho & Huo obtained a weaker bound Equivalence - Result … … Sparse representation and the Basis Pursuit Algorithm

30. 32 28 K +K = 0.9142/M 1 2 2 2M K K +MK -1=0 1 2 2 24 K +K =1/M 20 1 2 K >K 16 K +K = 1 2 1 2 = 0.5(1+1/M) 12 8 4 0 4 8 12 16 20 24 28 32 The Various Bounds Signal dimension: N=1024, Dictionary: =I, =FN , Mutual incoherence M=1/32. • Results • Uniqueness: 31 entries and below, • Equivalence: • 16 entries and below (D-H), • 29 entries and below (E-B). 2 K K 1 Sparse representation and the Basis Pursuit Algorithm

31. For uniqueness we got the requirement • For equivalence we got the requirement Equivalence – Uniqueness Gap • Is this gap due to careless bounding? • Answer [by Feuer and Nemirovski, to appear in IEEE Transactions On Information Theory]: No, both bounds are indeed tight. Sparse representation and the Basis Pursuit Algorithm

32. L Every column is normalized to have an l2 unit norm N Agenda • 1. Introduction • Previous and current work • 2. Two Ortho-Bases • Uncertainty  Uniqueness  Equivalence • 3. Arbitrary dictionary • Uniqueness  Equivalence • 4. Stylized Applications • Separation of point, line, and plane clusters • 5. Discussion Sparse representation and the Basis Pursuit Algorithm

33. Why General Dictionaries? • In many situations • We would like to use more than just two ortho-bases (e.g. Wavelet, Fourier, and ridgelets); • We would like to use non-ortho bases (pseudo-polar FFT, Gabor transform, … ), • In many situations we would like to use non-square transforms as our building blocks (Laplacian pyramid, shift-invariant Wavelet, …). • In the following analysis we assume ARBITRARY DICTIONARY (frame). We show that BP is successful over such dictionaries as well. Sparse representation and the Basis Pursuit Algorithm

34. Given a unit norm signal s, assume we hold two different representations for it using  • The equation implies a linear combination of columns from  that are linearly dependent. What is the smallest such group?  = 0 v Uniqueness - Basics • In the two-ortho case - simple splitting and use of the uncertainty rule – here there is no such splitting !! Sparse representation and the Basis Pursuit Algorithm

35. Examples: Spark =2 Spark =N+1; Uniqueness – Matrix “Spark” Definition: Given a matrix , define =Spark{} as the smallest integer such that there exists at least one group of  columns from  that is linearly dependent. Sparse representation and the Basis Pursuit Algorithm

36. “Spark” versus “Rank” The notion of spark is confusing – here is an attempt to compare it to the notion of rank Generally: 2  =Spark{} Rank{}+1. Sparse representation and the Basis Pursuit Algorithm

37. For any pair of representations of s we have • By the definition of the spark we know that if v=0 then . Thus • From here we obtain the relationship Uniqueness – Using the “Spark” • Assume that we know the spark of , denoted by . Sparse representation and the Basis Pursuit Algorithm

38. If we found a representation that satisfy Then necessarily it is unique (the sparsest). Theorem 4 Uniqueness Rule – 1 Any two different representations of the same signal using an arbitrary dictionary cannot be jointly sparse. Sparse representation and the Basis Pursuit Algorithm

39. Define • (notice the resemblance to the previous definition of M). • We can show (based on Gerśgorin disks theorem) that a lower-bound on the spark is obtained by Lower bound on the “Spark” • Since the Gerśgorin theorem is un-tight, this lower bound on the Spark is too pessimistic. Sparse representation and the Basis Pursuit Algorithm

40. Any two different representations of the same signal using an arbitrary dictionary cannot be jointly sparse. * If we found a representation that satisfy Then necessarily it is unique (the sparsest). Theorem 5 * This is the same as Donoho and Huo’s bound! Have we lost tightness? Uniqueness Rule – 2 Sparse representation and the Basis Pursuit Algorithm

41. The Spark can be found by solving • Use Basis Pursuit • Clearly . Thus . “Spark” Upper bound Sparse representation and the Basis Pursuit Algorithm

42. Given a signal s with a representation , Assuming that , P1 (BP) is Guaranteed to find the sparsest solution. * Theorem 6 * This is the same as Donoho and Huo’s bound! Is it non-tight? Equivalence – The Result Following the same path as shown before for the equivalence theorem in the two-ortho case, and adopting the new definition of M we obtain the following result: Sparse representation and the Basis Pursuit Algorithm

43. forward transform? Why works so well? Practical Implications? To Summarize so far … Over-complete linear transforms – great for sparse representations Basis Pursuit Algorithm We give explanations (uniqueness and equivalence) true for any dictionary (a) Design of dictionaries, (b) Test of for optimality, (c) Applications - scrambling, signal separation, inverse problems, … Sparse representation and the Basis Pursuit Algorithm

44. Agenda • 1. Introduction • Previous and current work • 2. Two Ortho-Bases • Uncertainty  Uniqueness  Equivalence • 3. Arbitrary dictionary • Uniqueness  Equivalence • 4. Stylized Applications • Separation of point, line, and plane clusters • 5. Discussion Sparse representation and the Basis Pursuit Algorithm

45. Assume a 3D volume of size • Voxels. • S contains digital points, lines, and planes, defined algebraically ( ): • Point – one Voxel • Line – p Voxels (defined by a starting Voxel on the cube’s face, and a slope – wrap around is permitted), • Plane – p2 Voxels (defined by a height Voxel in the cube, and 2 slope parameters – wrap around is permitted). Problem Definition • Task: Decompose S into the point, line, and plane atoms – sparse decomposition is desired. Sparse representation and the Basis Pursuit Algorithm

46. Properties To Use • Digital lines and planes are constructed such that: • Any two lines are disjoint, or intersect in a single Voxel. • Any two planes are disjoint, or intersect in a line of p Voxels. • Any digital line intersect a with a digital plane not at all, in a single Voxel, or along the p Voxels being the line itself. Sparse representation and the Basis Pursuit Algorithm

47. Our Dictionary • We construct a dictionary to contain all occurrences of points, lines, and planes (O(p4) columns). • Every atom in our set is reorganized lexicographically as a column vector of size p3-by-1. • The spark of this dictionary is =p+1 (p points and one line, or p lines and one plane). • The mutual incoherence is given by M=p-0.5 (normalized inner product of a plane with a line, or line with a point). Sparse representation and the Basis Pursuit Algorithm

48. If we found a representation that satisfies then necessarily it is sparsest (P0 solution). Due to Theorem 4 Given a volume s with a representation such that , (BP) is guaranteed to find it. Due to Theorem 6 Thus we can say that … Sparse representation and the Basis Pursuit Algorithm

49. Implications • Assume p=1024: • A representation with fewer than 513 atoms IS KNOWN TO BE THE SPARSEST. • A representation with fewer than 17 atoms WILL BE FOUND BY THE BP. • What happens in between? We know that the second bound is very loose! BP will succeed far above 17 atoms. Further work is needed to establish this fact. • What happens above 513 atoms? No theoretical answer yet! Sparse representation and the Basis Pursuit Algorithm

50. Towards Practical Application • The algebraic definition of the points, lines, and planes atoms was chosen to enable the computation of  and M. • Although wrap around makes this definition non-intuitive, these results are suggestive of what may be the case for geometrically intuitive definitions of points, lines, and planes. • Further work is required to establish this belief. Sparse representation and the Basis Pursuit Algorithm