Robust Flash Deblurring

1. Robust Flash Deblurring ShaojieZhuo, Dong Guo, Terence Sim School of Computing, National University of Singapore CVPR2010 Key words: image deblur, flash/no-flash technique, blur kernel estimate, sharp image reconstruction Reporter:周 澄(A.J.) 01/16/2011

2. Outline • Goal • Major contributions • Related work • Flash deblurring framework • MAP optimization • Practical implementation • Results • Limitations • Future works

3. Goal • Deblur a shaken image by using a corresponding flash image generated from a conventional hand-held camera.

4. Major contributions • Propose a novel approach by adding a flash image as a key constraint. • Well handle both the flash artifacts and the deconvolution artifacts. • Additional constraints introduced. • High quality results. • Insensitive to noise, fine image details.

5. Related work • Non-blind image deconvolution (Single image method). Def: Given the estimated blur kernel, the second step is to reconstruct a sharp image from the blurred image.

6. Related work • [Fergus et al. TOG2006] Removing camera shake from a single photograph. Used a variationalBayes inference method with natural image statistics to estimate the motion blur kernel.

7. Related work • [Jia. CVPR 2007] Single image motion deblurring using transparency. Investigated the relationship between object boundary transparency and the image motion blur and estimated the blur kernel from the alpha matte of motion objects.

8. Related work • [Shan et al. TOG 2008] High-quality motion deblurring from a single image. Formulated the deblurring problem as an MAP problem and proposed a high-order derivatives image noise model and a local image prior to avoid trivial solution.

9. Related work • Fergus, Jia, and Shan’s method are all able to obtain accurate kernels when the blur is small. => Fail at large shaking & noise.

10. Related work • Wiener filter and the Richardson-Lucy (RL) deconvolution. [Levin et al. CVPR2009] Understanding and evaluating blind deconvolution algorithms.

11. Related work • Wiener filter and the Richardson-Lucy (RL) deconvolution. => Suffer from deconvolution artifacts such as amplified noise and ringing artifacts.

12. Related work • Regularization methods. • In order to reduce artifacts. [Wang et al. SIIMS2008] A new alternating minimization algorithm for total variation image reconstruction. [Yuan et al. TOG2008] Progressive interscale and intra-scale non-blind image deconvolution.

13. Related work • Regularization methods. => Lost fine image details since you cannot separate image details from artifacts like noise or ringing artifacts properly.

14. Related work • Additional hardware(Hybrid camera). High resolution camera + low resolution video camera. => Some image details still lost due to non-invertible motion blur.

15. Related work • Multiple images solution. [Yuan et al. TOG2007] Image deblurring with blurred/noisy image pairs. [Chen et al. CVPR2008] Robust dual motion deblurring. [YW Tai et al. CVPR2005] Local color transfer via probabilistic segmentation by expectation-maximization.

16. Related work • Flash/no-flash technique. [Agrawal et al. TOG2005] Removing photography artifacts using gradient projection and flashexposure sampling. [Petschnigg et al. TOG2004] Digital photography with flash and no-flash image pairs. [Eisemann et al. TOG2004] Flash photography enhancement via intrinsic relighting.

17. Related work • Traditional flash/no-flash technique. => Need good alignment between two images.

18. Flash deblurring framework • Input: A blur image B and corresponding flash image F. • Output: A visually pleasant sharp image I with least flash artifacts or deconvolution artifacts.

19. Flash deblurring framework • Problem formulation: Given the blurred image B and flash image F, our goal is to estimate a blur kernel K and a sharp image I, so that I,K and B can be represented by the convolution model and the gradients of I are close to those in F.(We talk about gradients later)

20. Flash deblurring framework Maximum-a-posteriori (MAP) framework

21. The convolution model • B = I * K + n, where n is the image noise which modeled as a set of independent and identically distributed(i.i.d.) Gaussian noise.

22. MAP optimization • Problem formulation: where L(．) = - log (p(．))

23. MAP optimization • Likelihood term(SSD): • Analyze the SSD of the estimated kernel convolution result and the blur image.

24. MAP optimization • Kernel prior term: where the parameter α≦1; α=0.8 used in the paper.

25. Flash gradient constraint • Key idea: The robust flash gradient constraint encourages the gradients of reconstructed image to be close to those in F, while at the locations of flash artifacts, ambient shadows or noise it allow their gradient to differ to avoid flash artifacts and keep ambient illumination.

26. MAP optimization • Flash gradient constraint: observation. 1D scanlines of intensities and gradients in R channel of the three images. In the gradient plot, ΔI(cyan) is much close toΔF(magenta), which acts as a guide to reconstruct the sharp image I.

27. MAP optimization • Flash gradient constraint: where is the Lorentzian robust estimator and ε is a predefined constant.

28. MAP optimization • Objective function: Here we have the likelihood term, flash gradient constraint and kernel prior term, where λf and λk are the used to balance the three terms.

29. Flash deblurring framework Maximum-a-posteriori (MAP) framework • Sharp image reconstruction • Kernel estimation

30. Sharp image reconstruction • Kernel estimation • Fix K, we can estimate I by solving: Where the weight of re-weight least square for each pixel i at each iteration:

31. Sharp image reconstruction • Kernel estimation • Fix I, we can estimate K by solving: • Both equation can be solved by iterative re-weight least squares(IRLS).

32. Sharp image reconstruction • Kernel estimation • The two steps are alternated until diff(K) is smaller than the threshold. • To avoid local minimum when blur kernel is large, the kernel estimation is performed in a coarse-to-fine manner in the scale space.

33. Sharp image reconstruction • Kernel estimation • Three common artifacts. • Flash artifact regions. • λf is set to be large to suppress noise or ringing artifacts. • Over-saturated regions.

34. Sharp image reconstruction • Kernel estimation • Build a mask image M to change the weight of flash gradient constraint locally.

35. Sharp image reconstruction • Kernel estimation • Flash artifacts detect. • || K * I - F ||2. • Only need to manually mark the the flash shadow edge, since the shadow regions still contains useful gradients.

36. Sharp image reconstruction • Kernel estimation • Add in the sparse gradient constraint. The objective func becomes: where “ο” denotes the pixel-wise multiplication operator. • Solved by IRLS.

37. Practical implementation • Take flash image first, and then use high speed capturing mode capture blur image. • As the time between two shots is small, the the motion during the shots is basically a translation, which just causes a shift in the estimated blur kernel. Therefore, no image alignment is required.

38. Resaults – kernel RMS error

39. Resaults

40. Resaults

43. Results

44. Limitations • It cannot handle the spatially invariant motion blur model. • Additional alignment needed. • Ex: Busy traffic. • The exposure time between flash/no-flash images should be close. • Temporal incoherence. • Two images should share same aperture value. • May generate blur artifacts caused by different focus.

45. Future works • Extend to video. • Coherent preserve. • Support hybrid system such as combining with IR depth sensor or stereo sensor to get more information from image.