Shrinkage/Thresholding Iterative Methods

1. Shrinkage/Thresholding Iterative Methods • Nonquadratic regularizers • Total Variation • lp- norm • Wavelet orthogonal/redundant representations • sparse regression • Majorization Minimization revisietd • IST- Iterative Shrinkage Thresolding Methods • TwIST-Two step IST

2. Linear Inverse Problems -LIPs

3. References J. Bioucas-Dias and M. Figueiredo, "A New TwIST: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration“ Submitted to IEEE Transactions on Image processing, 2007. M. Figueiredo, J. Bioucas-Dias, and R. Nowak, "Majorization-Minimization Algorithms for Wavelet-Based Image Deconvolution'', Submitted to IEEE Transactions on Image processing, 2006.

4. More References M. Figueiredo and R. Nowak, “An EM algorithm for wavelet-based image restoration,” IEEE Trans. on Image Processing, vol. 12, no. 8, pp. 906–916, 2003. J. Bioucas-Dias, “Bayesian wavelet-based image deconvolution: a GEM algorithm exploiting a class of heavy-tailed priors,” IEEE Trans. on Image Processing, vol. 15, pp. 937–951, 2006. A. Chambolle, “An algorithm for total variation minimization and applications,” Journal of Mathematical Imaging and Vision, vol. 20, pp. 89-97, 2004. P. Combettes and V. Wajs, “Signal recovery by proximal forwardbackward splitting,” SIAM Journal on Multiscale Modeling & Simulation vol. 4, pp. 1168–1200, 2005 I. Daubechies, M. Defriese, and C. De Mol, “An iterative thresholding algorithm for linear inverse problems with a sparsity constraint”, Communications on Pure and Applied Mathematics, vol. LVII, pp. 1413-1457, 2004

5. Easy to prove monotonicity: Majorization Minorization (MM) Framework Let Majorization Minorization algorithm: ....with equality if and only if should be easy to maximize Notes: EM is an algorithm of this type.

6. MM Algorithms for LIPs IST Class: Majorize IRS Class: Majorize IST/IRS: Majorize and

7. MM Algorithms: IST class Assume that

8. MM Algorithms: IST class Majorizer: Let: IST Algorithm

9. MM Algorithms: IST class Overrelaxed IST Algorithm Convergence: [Combettes and V. Wajs, 2004] • is convex • the set of minimizers, G, of • is non-empty • 2 ]0,1] • Then converges to a point in G

10. Denoising with convex regularizers Denoising function also known as the Moreou proximal mapping Classes of convex regularizers: 1- homogeneous (TV, lp-norm (p>1)) 2- p power of an lp norm

11. 1-Homogeneous regularizers Then where is a closed convex set and denotes the orthogonal projection on the convex set

12. Total variation regularization Total variation [S. Osher, L. Rudin, and E. Fatemi, 1992] is convex (although not strightly) and 1-homogeneous  Total variation is a discontinuity-preserving regularizer have the same TV

13. Total variation regularization Then [Chambolle, 2004]

14. Total variation denoising

15. Total variation deconvolution 2000 IST iterations !!!

16. Weighted lp-norms is convex (although not strightly) and 1-homogeneous  There is no closed form expression for excepts for some particular cases Thus

17. Soft thresholding: p=1 Thus

18. Soft thresholding: p=1

19. Soft thresholding: p=1

20. Another way to look at it: Since L is convex: the point is a global minimum of L iif where is the subdifferential of L at f’

21. Example: Wavelet-based restoration Approximation coefficients (g-low pass filter) g,h – quadrature mirror filters Wavelet basis Wavelet coefficients Detail coefficients (h – high pass filter) DWT, Harr, J=2

22. Example: Wavelet-based restoration Histogram of coefficients – log h Histogram of coefficients - h