Sparse Representation and Compressed Sensing: Theory and Algorithms

1. Sparse Representation and Compressed Sensing:Theory and Algorithms Yi Ma1,2 Allen Yang3 John Wright1 2University of Illinois at Urbana-Champaign 3University of California Berkeley 1Microsoft Research Asia CVPR Tutorial, June 20, 2009 TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAA

2. MOTIVATION – Applications to a variety of vision problems • Face Recognition: • Wright et al PAMI ’09, Huang CVPR ’08, Wagner CVPR ’09 … • Image Enhancement and Superresolution: • Elad TIP ’06, Huang CVPR ‘08, … • Image Classification: • Mairal CVPR ‘08, Rodriguez ‘07, many others … • Multiple Motion Segmentation: • Rao CVPR ‘08, Elfamhir CVPR ’09 … • … and many others, including this conference

3. MOTIVATION – Applications to a variety of vision problems • Face Recognition: • Wright et al PAMI ’09, Huang CVPR ’09, Wagner CVPR ’09 … • Image Enhancement and Superresolution: • Elad TIP ’06, Huang CVPR ‘08, … • Image Classification: • Mairal CVPR ‘08, Rodriguez ‘07, … • Multiple Motion Segmentation: • Rao CVPR ‘08, Elfamhir CVPR ’09 … • … and many others, including this conference When and why can we expect such good performance? A closer look at the theory …

4. SPARSE REPRESENTATION – Model problem Underdetermined system of linear equations, ? ? ? ? … Observation Unknown Two interpretations: • Compressed sensing: A as sensing matrix • Sparse representation: A as overcomplete dictionary

5. SPARSE REPRESENTATION – Model problem Underdetermined system of linear equations, ? ? ? ? … Observation Unknown Many more unknowns than observations → no unique solution. • Classical answer: minimum -norm solution • Emerging applications: instead desire sparse solutions

6. SPARSE SOLUTIONS – Uniqueness Look for the sparsest solution: - number of nonzero elements

7. SPARSE SOLUTIONS – Uniqueness Look for the sparsest solution: - number of nonzero elements Is the sparsest solution unique? - size of smallest set of linearly dependent columns of A.

8. SPARSE SOLUTIONS – Uniqueness Look for the sparsest solution: - number of nonzero elements Is the sparsest solution unique? - size of smallest set of linearly dependent columns of A. Proposition[Gorodnitsky & Rao ‘97]: If with , then is the unique solution to

9. SPARSE SOLUTIONS – So How Do We Compute It? Looking for the sparsest solution: Bad News: NP-hard in the worst case, hard to approximate within certain constants [Amaldi & Kann ’95].

10. SPARSE SOLUTIONS – So How Do We Compute It? Looking for the sparsest solution: Bad News: NP-hard in the worst case, hard to approximate within certain constants [Amaldi & Kann ’95]. • Maybe we can still solve important cases? • Greedy algorithms: • Matching Pursuit, Orthogonal Matching Pursuit [Mallat & Zhang ‘93] • CoSAMP[Needell & Tropp ‘08] • Convex programming [Chen, Donoho & Saunders ‘94]

11. SPARSE SOLUTIONS – The Heuristic Looking for the sparsest solution: Intractable. convex relaxation Linear program, solvable in polynomial time. Why ? Convex envelope of over the unit cube: Rich applied history – geosciences, sparse coding in vision, statistics

12. EQUIVALENCE – A stronger motivation In many cases, the solutions to (P0) and (P1) are exactly the same: Theorem [Candes & Tao ’04, Donoho ‘04]: For Gaussian , with overwhelming probability, whenever “ -minimization recovers any sufficiently sparse solution”

13. GUARANTEES – “Well-Spread” A Mutual coherence: largest inner product between distinct columns of A Low mutual coherence: vectors are well-spread in the space

14. GUARANTEES – “Well-Spread” A Mutual coherence: Theorem[Elad & Donoho ’03, Gribvonel & Nielsen ‘03]: minimization uniquely recovers any with . Strong point: checkable condition. Weakness: low coherence can only guarantee recovery up to nonzeros.

15. GUARANTEES – Beyond Coherence Low coherence: “any submatrix consisting of two columns ofA is well-conditioned” Stronger bounds by looking at larger submatrices? Restricted Isometry Constants: s.t. for all -sparse , “Column submatrices of A are uniformly well-conditioned” Low RIC:

16. GUARANTEES – Beyond Coherence Restricted Isometry Constants: s.t. for all -sparse , Theorem [Candes & Tao ’04, Candes ‘07]: If , then -minimization recovers any k-sparse . For random A, this guarantees recovery up to linear sparsity:

17. GUARANTEES – Sharp Conditions? Necessary and sufficient condition: solves iff polytope spanned by columns of A and their negatives

18. GUARANTEES – Geometric Interpretation Necessary and sufficient condition: [Donoho ’06] [Donoho+ Tanner ’08] uniquely recovers with support and signs iff is a simplicial face of . Uniform guarantees for -sparse P centrally -neighborly.

19. GUARANTEES – Geometric Interpretation Geometric understanding gives sharp thresholds for sparse recovery with Gaussian A [Donoho & Tanner ‘08]: Weak threshold Failure almost always Sparsity Strong threshold Success almost always Success always Aspect ratio of A

20. GUARANTEES – Geometric Interpretation Explicit formulas in the wide-matrix limit [Donoho & Tanner ‘08]: Weak threshold: Strong threshold:

21. GUARANTEES – Noisy Measurements What if there is noise in the observation? Gaussian or bounded 2-norm Natural approach: relax the constraint: Studied in several literatures Statistics – LASSO Signal processing – BPDN.

22. GUARANTEES – Noisy Measurements What if there is noise in the observation? Natural approach: Theorem[Donoho, Elad & Temlyakov ‘06]: Recovery is stable: See also [Candes-Romberg-Tao ‘06], [Wainwright ‘06], [Meinshausen & Yu ’06], [Zhao & Yu ‘06], …

23. GUARANTEES – Noisy Measurements What if there is noise in the observation? Natural approach: Theorem[Candes-Romberg-Tao ‘06]: Recovery is stable – for A satisfying an appropriate condition, – best S-term approximation See also [Donoho ‘06], [Wainwright ‘06], [Meinshausen & Yu ’06], [Zhao & Yu ‘06], …

24. CONNECTIONS – Sketching and Expanders Similar sparse recovery problems explored in data streaming community: Combinatorial algorithms → fast encoding/decoding at expense of suboptimal # of measurements Based on ideas from group testing, expander graphs 0 2 0 5 0 0 0 1 Data stream Sketch … [Gilbert et al ‘06], [Indyk ‘08], [Xu & Hassibi ‘08]

25. CONNECTIONS – High dimensional geometry • Sparse recovery guarantees can also be derived via probabilistic • constructions from high-dimensional geometry: • The Johnson-Lindenstrauss lemma • Dvoretsky’s almost-spherical section theorem: • There exist subspaces of dimension as high as • on which the and norms are comparable: Given n points a random projection into dimensions preserves pairwise distances:

26. THE STORY SO FAR – Sparse recovery guarantees • Sparse solutions can often be recovered by linear programming. • Performance guarantees for arbitrary matrices with • “uniformly well-spread columns”: • (in)-coherence • Restricted Isometry • Sharp conditions via polytope geometry • Very well-understood performance for random matrices • What about matrices arising in vision… ?

27. PRIOR WORK - Face Recognition as Sparse Representation Linear subspace model for images of same face under varying illumination: Subject i Training If test image is also of subject , then for some . . Can represent any test image wrt the entire training set as corruption, occlusion coefficients Test image Combined training dictionary

28. PRIOR WORK - Face Recognition as Sparse Representation Underdeterminedsystem of linear equations in unknowns : Solution is not unique … but should besparse: ideally, only supported on images of the same subject expected to besparse: occlusion only affects a subset of the pixels Seek thesparsestsolution: convex relaxation Wright, Yang, Ganesh, Sastry, and Ma. Robust Face Recognition via Sparse Representation, PAMI 2008

29. GUARANTEES – What About Vision Problems? Behavior under varying levels of random pixel corruption: Recognition rate 99.3% 90.7% 37.5% Can existing theory explain this phenomenon?

30. PRIOR WORK - Error Correction by minimization Candes and Tao [IT ‘05]: • Apply parity check matrix s.t. , yielding • Set • Recover from clean system Underdetermined system in sparse e only Succeeds whenever in the reduced system .

31. PRIOR WORK - Error Correction by minimization Candes and Tao [IT ‘05]: • Apply parity check matrix s.t. , yielding • Set • Recover from clean system Underdetermined system in sparse e only Succeeds whenever in the reduced system . This work: • Instead solve Can be applied when A is wide (no parity check).

32. PRIOR WORK - Error Correction by minimization Candes and Tao [IT ‘05]: • Apply parity check matrix s.t. , yielding • Set • Recover from clean system Underdetermined system in sparse e only Succeeds whenever in the reduced system . This work: • Instead solve Succeeds whenever in the expanded system .

33. GUARANTEES – What About Vision Problems? Highly coherent ( volume ) very sparse: # images per subject, often nonnegative (illumination cone models). as dense as possible: robust to highest possible corruption. Results so far:should not succeed.

34. SIMULATION - Dense Error Correction? As dimension , an even more striking phenomenon emerges:

35. SIMULATION - Dense Error Correction? As dimension , an even more striking phenomenon emerges:

36. SIMULATION - Dense Error Correction? As dimension , an even more striking phenomenon emerges:

37. SIMULATION - Dense Error Correction? As dimension , an even more striking phenomenon emerges:

38. SIMULATION - Dense Error Correction? As dimension , an even more striking phenomenon emerges:

39. SIMULATION - Dense Error Correction? As dimension , an even more striking phenomenon emerges: Conjecture:If the matrices are sufficiently coherent, then for any error fraction , as , solving corrects almost any error with .

40. DATA MODEL - Cross-and-Bouquet Our model for should capture the fact that the columns are tightly clustered around a common mean : L^-norm of deviations well-controlled ( -> v ) Mean is mostly incoherent with standard (error) basis We call this the“Cross-and-Bouquet’’ (CAB)model.

41. ASYMPTOTIC SETTING - Weak Proportional Growth • Observation dimension • Problem size grows proportionally: • Error support grows proportionally: • Support size sublinear in :

42. ASYMPTOTIC SETTING - Weak Proportional Growth • Observation dimension • Problem size grows proportionally: • Error support grows proportionally: • Support size sublinear in : Sublinear growth of is necessary to correct arbitrary fractions of errors: Need at least “clean” equations. Empirical Observation: If grows linearly in , sharp phase transition at .

43. NOTATION - Correct Recovery of Solutions Whether is recovered depends only on Call -recoverable if with these signs and support and the minimizer is unique.

44. MAIN RESULT - Correction of Arbitrary Error Fractions Recall notation: “ recovers any sparse signal from almost any error with density less than 1”

45. SIMULATION - Arbitrary Errors in WPG Fraction of correct successes for increasing m ( , )

46. SIMULATION - Phase Transition in Proportional Growth What if grows linearly with m? Asymptotically sharp phase transition, similar to that observed by Donoho and Tanner for homogeneous Gaussian matrices

47. SIMULATION - Comparison to Alternative Approaches “L1 - [A I]”: “L1 -  comp”: “ROMP”: Regularized orthogonal matching pursuit Candes + Tao ‘05 Needell + Vershynin ‘08

48. SIMULATION - Error Correction with Real Faces For real face images, weak proportional growth corresponds to the setting where the total image resolution grows proportionally to the size of the database. Fraction of correct recoveries Above: corrupted images. ( 50% probability of correct recovery ) Below: reconstruction.

49. SUMMARY – Sparse Representation in Theory and Practice So far: Face recognition as a motivating example Sparse recovery guarantees for generic systems New theory and new phenomena from face data After the break: Algorithms for sparse recovery Many more applications in vision and sensor networks Matrix extensions: missing data imputation and robust PCA