Removing blur due to camera shake from images.

1. Removing blur due to camera shake from images. William T. Freeman Joint work with Rob Fergus, Anat Levin, Yair Weiss, Fredo Durand, Aaron Hertzman, Sam Roweis, Barun Singh Massachusetts Institute of Technology

2. Overview Our algorithm Original

3. Close-up Naïve Sharpening Our algorithm Original

4. Let’s take a photo Blurry result

5. Slow-motion replay

6. Slow-motion replay Motion of camera

7. Image formation process ⊗ = Blur kernel Blurry image Sharp image Input to algorithm Desired output Convolutionoperator Model is approximation

8. Why is this hard? Simple analogy: 11 is the product of two numbers. What are they? No unique solution: 11 = 1 x 11 11 = 2 x 5.5 11 = 3 x 3.667 etc….. Need more information !!!!

9. Multiple possible solutions = ⊗ = ⊗ Sharp image Blur kernel = ⊗ Blurry image

10. Is each of the images that follow sharp or blurred?

14. Another blurry one

17. Natural image statistics Characteristic distribution with heavy tails Histogram of image gradients

18. Blury images have different statistics Histogram of image gradients

19. Parametric distribution Histogram of image gradients Use parametric model of sharp image statistics

20. Uses of natural image statistics • Denoising [Roth and Black 2005] • Superresolution [Tappen et al. 2005] • Intrinsic images [Weiss 2001] • Inpainting [Levin et al. 2003] • Reflections [Levin and Weiss 2004] • Video matting [Apostoloff & Fitzgibbon 2005] Corruption process assumed known

21. Existing work on image deblurring Assumed forms of blur kernels Software algorithms: • Extensive literature in signal processing community • Mainly Fourier and/or Wavelet based • Strong assumptions about blur  not true for camera shake • Image constraints are frequency-domain power-laws

23. A focus on image constraints, not image priors

24. Some image constraints/priors

25. Toy example: observed “image”: 1.0 0.0

26. Toy example: observed “image”: 0.0 1.0

27. Toy example: observed “image”: 1.0 0.0

28. Three sources of information = ⊗ Estimatedblur kernel Estimated sharp image Input blurry image 3. Blur prior: Positive & Sparse 1. Reconstruction constraint: 2. Image prior: Distribution of gradients

29. Prior on image gradients (mixture of Gaussians giving a Laplacian-like distribution) Distribution of gradients (log-scale) Green curve is our mixture of gaussians fit.

30. Prior on blur kernel pixels (mixture of exponentials) P(b) b

31. How do we use this information? Obvious thing to do: • Combine 3 terms into an objective function • Run conjugate gradient descent • This is Maximum a-Posteriori (MAP)

32. Maximum A-Posteriori y – observed blurry image x – unobserved sharp image b – blur kernel Blur prior Latent image prior Likelihood Sparse and i – image patch index f – derivative filter Assumption: all pixels independent of one another

33. Results from MAP estimation Input blurry image Maximum a-Posteriori (MAP) Our method: Variational Bayes

34. Variational Bayes http://citeseer.ist.psu.edu/cache/papers/cs/16537/http:zSzzSzwol.ra.phy.cam.ac.ukzSzjwm1003zSzspringer_chapter8.pdf/miskin00ensemble.pdf

35. Miskin and Mackay, 2000

36. Setup of variational approach Need likelihood and prior in same space, so use gradients: Likelihood Prior on latent image gradients – mixture of Gaussians i – image pixel j – blur pixel We use C=4, D=4 Prior on blur elements – mixture of Exponentials Also have Gamma hyperpriors on

37. Variational inference • Approximate posterior with • Cost function • Assume • Use gradient descent, alternating between updating while marginalizing out over and vice versa • Adapted code from Miskin & Mackay 2000 is Gaussian on each pixel is rectified Gaussian on each pixel

38. Variational Bayesian method Based on work of Miskin & Mackay 2000 Keeps track of uncertainty in estimates of image and blur by using a distribution instead of a single estimate Helps avoid local maxima and over-fitting

39. Variational Bayesian method Objective function for a single variable Maximum a-Posteriori (MAP) Variational Bayes Score Pixel intensity

40. MAP vs Variational MAP Variational MAP using variational initialization

41. Blurry synthetic image

42. Inference – initial scale

43. Inference – scale 2

44. Inference – scale 3

45. Inference – scale 4

46. Inference – scale 5

47. Inference – scale 6

48. Inference – final scale

49. Our output

50. Ground truth