QUASI MAXIMUM LIKELIHOOD BLIND DECONVOLUTION

1. QUASI MAXIMUM LIKELIHOOD BLIND DECONVOLUTION AlexanderBronstein

2. 2 BIBLIOGRAPHY A. Bronstein, M. Bronstein, and M. Zibulevsky, "Quasi maximum likelihood blind deconvolution: super- ans sub-Gaussianity vs. asymptotic stability", submitted to IEEE Trans. Sig. Proc. A. Bronstein, M. Bronstein, M. Zibulevsky, and Y. Y. Zeevi, "Quasi maximum likelihood blind deconvolution: asymptotic performance analysis", submitted to IEEE Trans. Information Theory. A. Bronstein, M. Bronstein, and M. Zibulevsky, "Relative optimization for blind deconvolution", submitted to IEEE Trans. Sig. Proc. A. Bronstein, M. Bronstein, M. Zibulevsky, and Y. Y. Zeevi, "Quasi maximum likelihood blind deconvolution of images acquired through scattering media", Submitted to ISBI04. A. Bronstein, M. Bronstein, M. Zibulevsky, and Y. Y. Zeevi, "Quasi maximum likelihood blind deconvolution of images using optimal sparse representations", CCIT Report No. 455 (EE No. 1399), Dept. of Electrical Engineering, Technion, Israel, December 2003. A. Bronstein, M. Bronstein, and M. Zibulevsky, "Blind deconvolution with relative Newton method", CCIT Report No. 444 (EE No. 1385), Dept. of Electrical Engineering, Technion, Israel, October 2003.

3. 3 QML blind deconvolution Asymptotic analysis Relative Newton Generalizations Problem formulation Applications AGENDA Introduction

4. 4 W H CONVOLUTION MODEL DECONVOLUTION BLIND DECONVOLUTION PROBLEM restoration kernel source estimate arbitrary scaling factor arbitrary delay source signal convolution kernel observed signal sensor noise signal

5. 5 APPLICATIONS Acoustics, speech processing  DEREVERBERATION Optics, image processing, biomedical imaging  DEBLURRING Communications  CHANNEL EQUALIZATION Control  SYSTEM IDENTIFICATION Statistics, finances  ARMA ESTIMATION

6. 6 Asymptotic analysis Relative Newton Generalizations ML vs. QML The choice of φ(s) Equivariance Gradient and Hessian AGENDA Introduction QML blind deconvolution

7. 7 1 2 3 4 ML BLIND DECONVOLUTION ASSUMPTIONS is i.i.d. with probability density function has no zeros on the unit circle, i.e. No noise (precisely: no noise model) is zero-mean MAXIMUM-LIKELIHOOD BLIND DECONVOLUTION:

8. 8 The true source PDF in usually unknown Many times is non-log-concave and not well-suited for optimization Substitute with some model function QUASI ML BLIND DECONVOLUTION PROBLEMS OF MAXIMUM LIKELIHOOD QUASI MAXIMUM LIKELIHOOD BLIND DECONVOLUTION

9. 9 SUPER-GAUSSIAN SUB-GAUSSIAN THE CHOICE OF  (s)

10. 10 EQUIVARIANCE QML estimator of given the observation Theorem: The QML estimator is equivariant, i.e., for every invertible kernel , it holds where stands for the impulse response of the inverse of . ANALYSIS OF ANALYSIS OF

11. 11 GRADIENT & HESSIAN OF GRADIENT where is the mirror operator. HESSIAN

12. 12 Relative Newton Generalizations Asymptotic Hessian structure Asymptotic error covariance Cramér-Rao bounds Superefficiency Examples AGENDA Introduction QML blind deconvolution Asymptotic analysis

13. 13 ASYMPTOTIC HESSIAN AT THE SOLUTION POINT At the solution point, For a sufficiently large sample size , the Hessian becomes where

14. 14 ASYMPTOTIC ERROR COVARIANCE Exact restoration kernel Estimation kernel from the data The scaling factor has to obey

15. 15 ASYMPTOTIC ERROR COVARIANCE Estimation error From second-order Taylor expansion, equivariance

16. 16 ASYMPTOTIC ERROR COVARIANCE Asymptotically ( ), Separable structure:

17. 17 ASYMPTOTIC ERROR COVARIANCE The estimation error covariance matrix asymptotically separates to

18. 18 ASYMPTOTIC ERROR COVARIANCE Asymptotic gradient covariance matrices where

19. 19 ASYMPTOTIC ERROR COVARIANCE Asymptotic estimation error covariance: Asymptotic signal-to-interference ratio (SIR) estimate:

20. 20 CRAMER-RAO LOWER BOUNDS True ML estimator: The distribution-dependent parameters simplify to Asymptotic error covariance simplifies to where

21. 21 CRAMER-RAO LOWER BOUNDS Asymptotic SIR estimate simplifies to

22. 22 SUPEREFFICIENCY Let the source be sparse, i.e., Let be the smoothed absolute value with smoothing parameter In the limit

23. 23 SUPEREFFICIENCY Similar results are obtained for uniformly-distributed source with Can be extended for sources with PDF vanishing outside some interval.

24. 24 ASYMPTOTIC STABILITY The QML estimator is said to be asymptotically stable if is a local minimizer of in the limit . Theorem: The QML estimator is asymptotically stable if the following conditions hold: and is asymptotically unstable if one of the following conditions hold:

25. 25 EXAMPLE Generalized Laplace distribution

26. 26 STABILITY OF THE SUPER-GAUSSIAN ESTIMATOR SUB-GAUSSIAN SUPER-GAUSSIAN

27. 27 STABILITY OF THE SUB-GAUSSIAN ESTIMATOR SUB-GAUSSIAN SUPER-GAUSSIAN

28. 28 PERFORMANCE OF THE SUPER-GAUSSIAN ESTIMATOR SUPER-GAUSSIAN

29. 29 PERFORMANCE OF THE SUB-GAUSSIAN ESTIMATOR SUB-GAUSSIAN

30. 30 Generalizations Relative optimization Relative Newton Fast Relative Newton AGENDA Introduction QML blind deconvolution Asymptotic analysis Relative Newton

31. 31 0 1 4 2 3 5 RELATIVE OPTIMIZATION (RO) Start with and For until convergence Start with Find such that Update source estimate End For Restoration kernel estimate: Source estimate:

32. 32 RELATIVE OPTIMIZATION (RO) Observation: The k-th step of the relative optimization algorithm depends only on Proposition: The sequence of target function values produced by the relative optimization algorithm is monotonically decreasing, i.e.,

33. 33 RELATIVE NEWTON Relative Newton = use one Newton step in the RO algorithm Near the solution point Newton system separates to

34. 34 FAST RELATIVE NEWTON Fast relative Newton = use one Newton step with approximate Hessian in the RO algorithm + regularized approximate Newton system solution. Approximate Hessian evaluation = order of gradient evaluation

35. 35 AGENDA Introduction QML blind deconvolution Asymptotic analysis Relative Newton Generalizations

36. 36 GENERALIZATIONS IIR KERNELS BLOCK PROCESSING  ONLINE DECONVOLUTION MULTI-CHANNEL DECONVOLUTIONBSS+BD DECONVOLUTION OF IMAGES + USE OF SPARSE REPRESENTATIONS