Single Image Super Resolution

1. Single Image Super Resolution Seminar presented by: TomerFaktor Advanced Topics in Computer Vision (048921) 12/01/2012

2. OutLine • What is Image Super Resolution (SR)? • Prior Art • Today – Single Image SR: • Using patch recurrence [Glasner et al., ICCV09] • Using sparse representations [Yang et al., CVPR08], [Zeyde et al., LNCS10]

3. From High to Low Res. and Back Blur (LPF) + Down-sampling Inverse problem

4. What is Image SR? • Inverse problem – underdetermined • Image SR - how to make it determined:

5. Goals of Image SR • Always: • More pixels - according to the scale factor • Low res. edges and details - maintained • Bonus: • Blurred edges - sharper • New high res. details missing in the low res.

6. Multi-Image SR “Classical” [Irani91,Capel04,Farisu04] Fusion

7. Example-based SR [Freeman01, Kim08] Example-based SR (Image Hallucination) Algorithm Blur kernel & scale factor External database of low and high res. image patches

8. Single Image SR Single-Image SR (Scale-Up) Algorithm Blur kernel & scale factor Image model/prior

9. Prior art • No blur, only down-sampling  Interpolation: • LSI schemes – NN, bilinear, bicubic, etc. • Spatially adaptive and non-linear filters [Zhang08, Mallat10] • Blur + down-sampling: Interpolation Deblurring

10. Prior art • LS problem: • Add a regularization: • Tikhonov regularization, robust statistics, TV, etc. • Sparsity of global transform coefficients • Parametric learned edge model [Fattal07, Sun08]

11. Super Resolution from a Single Image D. Glasner, S. Bagon and M. Irani ICCV 2009

12. BASIC Assumption Patches in a single image tend to redundantly recur many times within and across scales

13. Statistics of patch recurrence The impact of SR is mostly expressed here

14. Suggested Approach • Patch recurrence  A single image is enough for SR • Adapt ideas from classical and example-based SR: • SR linear constraints – in-scale patch redundancy instead of multi-image • Correspondence between low-high res. patches - cross-scale redundancy instead of external database Unified framework

15. Exploiting in-scale Patch redundancy • In the low res. – find K NN for each pixel • Compute their subpixel alignment • Different weight for each linear SR constraint (patch similarity)

16. Exploiting Cross-scale redundancy Unknown, increasing res. by factor α Copy Copy NN Parent Known, decreasing res. by factor α Parent NN

17. Algorithmic scheme

18. Important Implementation Details • Coarse-to-fine – gradual increase in resolution, for numerical stability, improves results • Back-projection [Irani91] – to ensure consistency of the recovered high res. image with the low res • Color images – convert RGB to YIQ: • For Y – the suggested approach • For I,Q – bicubicinterpolation

19. Results – Purely repetitive structure Input low res. image

20. Suggested Approach Bicubic Interpolation

21. Results – Purely repetitive structure Bicubic interp. Input low res. image In and cross-scale redundancies Only in-scale redundancy High res. detail

22. Results – TEXT IMAGE Suggested approach Ground Truth Bicubic Interp. Input low res. image Small digits recur only in-scale and cannot be recovered

23. Results – Natural IMAGE Edge model [Fattal07] Suggested Approach Example-based [Kim08] Bicubic Interp. Input low res. image

24. Paper evaluation • Reasonable assumption – validated empirically • Very novel – new SR framework, combining two widely used approaches in the SR field • Solution technically sound • Well written, very nice paper! • Many visual results (nice webpage)

25. Paper evaluation Not fully self-contained – almost no details on: • Subpixel alignment • Weighted classical SR constraints • Back-projection No numerical evaluation (PSNR/SSIM) of the results No code available No details on running time

26. On Single IMAGE scale-up using sparse-representations R. Zeyde, M. Elad and M. Protter Curves & Surfaces, 2010

27. Sparse-land Prior • Widely-used signal model with numerous applications • Efficient algorithms for pursuit and dictionary learning m n n d Dictionary signal Non-zeros sparse vector

28. BASIC Assumptions • Sparse-land prior for image patches • Patches in a high-low res. pair have the same sparse representation over a high-low res. dictionary pair [Yang08] & [Zeyde10] Same procedure for all patches Joint training!

29. Joint Representation - Justification • Each low res. patch is generated from the high res. using the same LSI operator (matrix): • A pair of high-low res. dictionaries with the correct correspondence leads to joint representation of the patches:

30. Research Questions • Pre-processing? • How to train the dictionary pair? • What is the training set? • How to utilize the dictionary pair for SR? • Post-processing? [Yang08] & [Zeyde10]

31. Pre-processing – Low res. 1 2 Low Res. Image Feature Extraction by HPFs Bicubic Interpolation

32. Pre-processing – Low Res. 3 Features for low res. patches Dimensionality Reduction via PCA Projection onto reduced PCA basis

33. Pre-processing – High res. - Bicubic Interpolation

34. Training Phase 1 2 Training the Dictionary Pair

35. Training the low res. dictionary 1 K-SVD Dictionary Learning

36. Training the high res. Dictionary 2

37. Which training SET? • External set of true high res. image (off-line training): • Generate low res. image • Collect • The image itself (bootstrapping): • Input low res. image = “high res.” • Proceed as before…

38. reconstruction Phase Low Res. Image Image Reconstruction Pre-processing OMP SR Image

39. Image reconstruction No need to perform back-projection!

40. Results – Text Image Suggested Approach Bicubic Interp. Ground Truth Input low res. image Dictionary pair learned off-line from another text image! PSNR=14.68dB PSNR=16.95dB

41. Numerical Results – Natural Images Dictionary pair learned off-line from a set of training images!

42. Visual Results – Natural Images Bicubic Interp. Yang et al. Ground Truth Suggested Approach Artifact

43. Comparing the two approaches

44. Visual Comparison Suggested On-line Approach Bicubic Interp. Suggested Off-line Approach Glasner et al. Input low res. image Appears in Zeyde et al.

45. Glasner et al. Doesn’t appear in Zeyde et al. Input low res. image Suggested On-line Approach Suggested Off-line Approach Bicubic Interp.

46. Paper evaluation • Reasonable assumptions – justified analytically • Novel – similar model as Yang et al., new algorithmic framework, improves runtime and image SR quality • Solution technically sound • Well written and self-contained • Code available • Performance evaluation – both visual and numerical

47. Paper evaluation Experimental validation – not full: • Comparison to other approaches – main focus on bicubic interp. and Yang et al. • No comparison between on-line and off-line learning of the dictionary pair on the same image Only “good” results are shown, running the code reveals weakness with respect to Glasner et al.

48. Future Directions • Extending the first approach to video: Space-time SR from a single video [Shahar et al., CVPR11] • Merging patch redundancy + sparse representations in the spirit of non-local K-SVD [Mairal et al., ICCV09] • Coarse-to-fine also for the second approach – to improve numerical stability and SR quality

49. Thank you for your attention!