Shrinkage for Redundant Representations: Today's Focus on Optimality and Relevance

1. SRINKAGE FOR REDUNDANT REPRESENTATIONS ? Michael Elad The Computer Science Department The Technion – Israel Institute of technology Haifa 32000, Israel SPARS'05 Signal Processing with Adaptive Sparse Structured Representations November 16-18, 2005 – Rennes, France

2. ? Remove Additive Noise Noise Removal Our story begins with signal/image denoising … • 100 years of activity – numerous algorithms. • Considered Directions include: PDE, statistical estimators, adaptive filters, inverse problems & regularization, sparse representations, … Shrinkage for Redundant representations?

3. LUT Apply Wavelet Transform Apply Inv. Wavelet Transform Shrinkage For Denoising • Shrinkage is a simple yet effective denoising algorithm [Donoho & Johnstone, 1993]. • Justification 1: minimax near-optimal over the Besov (smoothness) signal space (complicated!!!!). • Justification 2: Bayesian (MAP) optimal[Simoncelli & Adelson 1996, Moulin & Liu 1999]. • In both justifications, an additive Gaussian white noise and a unitary transform are crucial assumptions for the optimality claims. Shrinkage for Redundant representations?

4. Apply its (pseudo) Inverse Transform LUT Number of coefficients is (much) greater than the number of input samples (pixels) Apply Redundant Transform Redundant Transforms? • This scheme is still applicable, and it works fine (tested with curvelet, contourlet, undecimated wavelet, and more). • However, it is no longer the optimal solution for the MAP criterion. TODAY’S FOCUS: IS SHRINKAGE STILL RELEVANT WHEN HANDLING REDUNDANT (OR NON-UNITARY) TRANSFORMS? HOW? WHY? Shrinkage for Redundant representations?

5. Agenda • 1. Bayesian Point of View – a Unitary Transform • Optimality of shrinkage • 2. What About Redundant Representation? • Is shrinkage is relevant? Why? How? • 3. Conclusions ThomasBayes 1702 - 1761 Shrinkage for Redundant representations?

6. The MAP Approach Minimize the following function with respect to x: Prior or regularization Log-Likelihood term Unknown to be recovered Given measurements Shrinkage for Redundant representations?

7. Energy Smoothness Adapt+ Smooth Robust Statistics • Mumford & Shah formulation, • Compression algorithms as priors, • … Total-Variation Wavelet Sparsity Sparse & Redundant Image Prior? During the past several decades we have made all sort of guesses about the prior Pr(x): Today’s Focus Shrinkage for Redundant representations?

8. Define L2 is unitarily invariant We got a separable set of 1D optimization problems (Unitary) Wavelet Sparsity Shrinkage for Redundant representations?

9. Want to minimize this 1-D function with respect to z LUT Why Shrinkage? A LUT can be built for any other robust function (replacing the |z|), including non-convex ones (e.g., L0 norm)!! Shrinkage for Redundant representations?

10. Agenda n • 1. Bayesian Point of View – a Unitary Transform • Optimality of shrinkage • 2. What About Redundant Representation? • Is shrinkage is relevant? Why? How? • 3. Conclusions k Shrinkage for Redundant representations?

11. Tx== = An Overcomplete Transform • Redundant transforms are important because they can • Lead to a shift-invariance property, • Represent images better (because of orientation/scale analysis), • Enable deeper sparsity (and thus give more structured prior). Shrinkage for Redundant representations?

12. Define However Analysis versus Synthesis Analysis Prior: Synthesis Prior: Basis Pursuit Shrinkage for Redundant representations?

13. D-y= - Getting a sparse solution implies that y is composed of few atoms from D Basis Pursuit As Objective Our Objective: Shrinkage for Redundant representations?

14. Our objective Set j=1 Fix all entries of  apart from the j-th one Optimize with respect to j j=j+1 mod k Sequential Coordinate Descent • The unknown, , has k entries. • How about optimizing w.r.t. each of them sequentially? • The objective per each becomes Shrinkage for Redundant representations?

15. BEFORE: We had this 1-D function to minimize and the solution was Our 1-D objective is and the solution now is We Get Sequential Shrinkage NOW: Shrinkage for Redundant representations?

16. Set j=1 Fix all entries of  apart from the j-th one Optimize with respect to j j=j+1 mod k Sequential? Not Good!! • This method requires drawing one column at a time from D. • In most transforms this is not comfortable at all !!! Shrinkage for Redundant representations?

17. For j=1:k Compute the descent direction per j : vj. Update the solution by How About Parallel Shrinkage? Our objective • Assume a current solution n. • Using the previous method, we have k descent directions obtained by a simple shrinkage. • How about taking all of them at once, with a proper relaxation? • Little bit of math lead to … Shrinkage for Redundant representations?

18. Initialize (*) The Proposed Algorithm The synthesis error Back-projection to the signal domain Shrinkage operation Update by line-search Compute (*) At all stages, the dictionary is applied as a whole, either directly, or via its adjoint Shrinkage for Redundant representations?

19. Initialize Compute (*) For Example: Tight (DDH=cI) and normalized (W=I) frame (*) The First Iteration – A Closer Look Shrinkage for Redundant representations?

20. Apply its (pseudo) Inverse Transform LUT Apply Redundant Transform Relation to Simple Shrinkage Shrinkage for Redundant representations?

21. 700 Steepest Descent D: a 1001000, union of 10 random unitary matrices, y: D, with having 15 non- zeros in random locations, λ=1, 0=0, Line-Search: Armijo Sequential Shrinkage Parallel Shrinkage with line-search 600 MATLAB’s fminunc 500 400 Objective function value 300 200 100 0 0 5 10 15 20 25 30 35 40 45 50 Iteration A Simple Example Shrinkage for Redundant representations?

22. Image Denoising Objective function 9000 Iterative Shrinkage Steepest Descent Conjugate Gradient Truncated Newton 8000 7000 • The Matrix W gives a variance per each coefficient, learned from the corrupted image. • D is the contourlet transform (recent version). • The length of : ~1e+6. • The Seq. Shrinkage algorithm can no longer be simulated 6000 5000 4000 3000 2000 1000 0 2 4 6 8 10 12 14 16 18 Iterations Shrinkage for Redundant representations?

23. Denoising PSNR 32 31 30 29 28 27 26 Iterative Shrinkage Steepest Descent Conjugate Gradient Truncated Newton 25 24 23 22 Iterations Image Denoising Even though one iteration of our algorithm is equivalent in complexity to that of the SD, the performance is much better 0 2 4 6 8 10 12 14 16 18 Shrinkage for Redundant representations?

24. Image Denoising Noisy Image with σ=20 Original Image Iterated Shrinkage – First Iteration PSNR=28.30dB Iterated Shrinkage – second iteration PSNR=31.05dB Shrinkage for Redundant representations?

25. Closely Related Work • The “same” algorithm was derived in several other works: • Sparse representation over curvelet [Starck, Candes, Donoho, 2003]. • E-M algorithm for image restoration [Figueiredo & Nowak 2003]. • Iterated Shrinkage for problems of the form [Daubechies, Defrise, & De-Mol, 2004]. • The work proposed here is different in several ways: • Motivation: Shrinkage for redundant representation, rather than general inverse problems. • Derivation: We used a parallel CD-algorithm, and others used the EM or a sequence of surrogate functions. • Algorithm: We obtain a slightly different algorithm, where the norms of the atoms are used differently, different thresholds are used, the choice of μ is different. Shrinkage for Redundant representations?

26. Agenda • 1. Bayesian Point of View – a Unitary Transform • Optimality of shrinkage • 2. What About Redundant Representation? • Is shrinkage is relevant? Why? How? • 3. Conclusions Shrinkage for Redundant representations?

27. When optimal? What if the transform is redundant? How? How to avoid the need to extract atoms? Getting what? Conclusion Shrinkage is an appealing signal denoising technique For additive Gaussian noise and unitary transforms Compute all the CD directions, and use the average Go Parallel Option 1: apply sequential coordinate descent which leads to a sequential shrinkage algorithm We obtain a very easy to implement parallel shrinkage algorithm that requires forward transform, scalar shrinkage, and inverse transform. Shrinkage for Redundant representations?