1 / 74

Super-Resolution

Super-Resolution. Dr. Yossi Rubner yossi@rubner.co.il. Many slides from Miki Elad - Technion. Example - Video. 53 images, ratio 1:4. Example – Surveillance. 40 images ratio 1:4. Example – Enhance Mosaics. Super-Resolution - Agenda. The basic idea Image formation process

tybalt
Download Presentation

Super-Resolution

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Super-Resolution Dr. Yossi Rubner yossi@rubner.co.il Many slides from Miki Elad - Technion

  2. Example - Video 53 images, ratio 1:4

  3. Example – Surveillance 40 images ratio 1:4

  4. Example – Enhance Mosaics

  5. Super-Resolution - Agenda • The basic idea • Image formation process • Formulation and solution • Special cases and related problems • Limitations of Super-Resolution • SR in time

  6. For a given band-limited image, the Nyquist sampling theorem states that if a uniform sampling is fine enough (D), perfect reconstruction is possible. D D Intuition

  7. 2D 2D Intuition Due to our limited camera resolution, we sample using an insufficient 2D grid

  8. 2D 2D Intuition However, if we take a second picture, shifting the camera ‘slightly to the right’ we obtain:

  9. Intuition Similarly, by shifting down we get a third image: 2D 2D

  10. Intuition And finally, by shifting down and to the right we get the fourth image: 2D 2D

  11. Intuition It is trivial to see that interlacing the four images, we get that the desired resolution is obtained, and thus perfect reconstruction is guaranteed.

  12. Rotation/Scale/Disp. What if the camera displacement is Arbitrary ? What if the camera rotates? Gets closer to the object (zoom)?

  13. Rotation/Scale/Disp. There is no sampling theorem covering this case

  14. A Small Example 3:1 scale-up in each axis using 9 images, with pure global translation between them

  15. Complicated motion perspective, local motion, … Blur sampling is not a point operation Spatially variant blur Temporally variant blur Noise Changes in the scene Further Complications

  16. Super-Resolution - Agenda • The basic idea • Image formation process • Formulation and solution • Special cases and related problems • Limitations of Super-Resolution • SR in time

  17. Geometric transformation Optical Blur Scene Sampling Noise Image Formation LR HR Can we write these steps as linear operators?

  18. Geometric transformation Geometric Transformation • Any appropriate motion model • Every frame has different transformation • Usually found by a separate registration algorithm Scene

  19. = Geometric Transformation Can be modeled as a linear operation

  20. Geometric transformation Optical Blur Optical Blur • Due to the lens PSF and pixel integration • Usually

  21. H H PSF * PIXEL =

  22. = Optical Blur Can be modeled as a linear operation

  23. Optical Blur Sampling Sampling • Pixel operation consists of area integration followed by decimation • D is the decimation only • Usually

  24. o 1 = o Decimation Can be modeled as a linear operation

  25. Super-Resolution - Agenda • The basic idea • Image formation process • Formulation and solution • Special cases and related problems • Limitations of Super-Resolution • SR in time

  26. Super-Resolution - Model Geometric warp Blur Decimation Y High- Resolution Image 1 F =I H D D H V V Low- Resolution Exposures 1 1 1 1 N X Additive Noise Y N F N N N

  27. Simplified Model Geometric warp Blur Decimation Y High- Resolution Image 1 F =I H H D D V V Low- Resolution Exposures 1 1 N X Additive Noise Y N F N

  28. The Super-Resolution Problem • Given Yk – The measured images (noisy, blurry, down-sampled ..) H – The blur can be extracted from the camera characteristics D – The decimation is dictated by the required resolution ratio Fk – The warp can be estimated using motion estimation n– The noise can be extracted from the camera / image • Recover X – HR image

  29. =[10M×1] =[10M×16M] =[16M×1] The Model as One Equation r = resolution factor MXM = size of the frames N = number of frames r = resolution factor = 4 MXM = size of the frames = 1000X1000 N = number of frames = 10 Linear algebra notation is intended only to develop algorithm

  30. Smoothness constraint regularization SR - Solutions • Maximum Likelihood (ML): Often ill posed problem! • Maximum Aposteriori Probability (MAP)

  31. ML Reconstruction (LS) Minimize: Thus, require:

  32. Back projection Simulated error All the above operations can be interpreted as operations performed on images. There is no actual need to use the Matrix-Vector notations as shown here. LS - Iterative Solution • Steepest descent

  33. LS - Iterative Solution • Steepest descent For k=1..N - inverse geometry wrap geometry wrap convolve with H down sample up sample convolve with HT -

  34. LR + noise X4 Least squares Example HR image Simulated example from Farisu at al. IEEE trans. On Image Processing, 04

  35. Robust Reconstruction • Cases of measurements outlier: • Some of the images are irrelevant • Error in motion estimation • Error in the blur function • General model mismatch

  36. Minimize: Robust Reconstruction

  37. Robust Reconstruction • Steepest descent For k=1..N - inverse geometry wrap geometry wrap convolve with H down sample up sample convolve with HT sign -

  38. Least squares LR + noise X4 Robust Reconstruction Example - Outliers HR image Simulated example from Farisu at al. IEEE trans. On Image Processing, 04

  39. L2 norm based L1 norm based Example – Registration Error 20 images, ratio 1:4

  40. MAP Reconstruction • Regularization term: • Tikhonov cost function • Total variation • Bilateral filter

  41. Minimize: Robust Estimation + Regularization

  42. Robust Estimation + Regularization For k=1..N - inverse geometry wrap geometry wrap convolve with H down sample up sample convolve with HT sign - For l,m=-P..P - - horizontal shift l vertical shift m horizontal shift -l vertical shift -m sign From Farisu at al. IEEE trans. On Image Processing, 04

  43. Example • 8 frames • Resolution factor of 4 From Farisu at al. IEEE trans. On Image Processing, 04

  44. Example Images from Vigilant Ltd.

  45. Handling Color in SR Handling color: the classic approach is to convert the measurements to YCbCr, apply the SR on the Y and use trivial interpolation on the Cb and Cr. Better treatment can be obtained if the statistical dependencies between the color layers are taken into account (i.e. forming a prior for color images). In case of mosaiced measurements, demosaicing followed by SR is sub-optimal. An algorithm that directly fuse the mosaic information to the SR is better.

  46. SR for Full Color 20 images, ratio 1:4

  47. SR+Demosaicing 20 images, ratio 1:4 Mosaiced input Mosaicing and then SR Combined treatment

  48. Super-Resolution - Agenda • The basic idea • Image formation process • Formulation and solution • Special cases and related problems • Limitations of Super-Resolution • SR in time

  49. Special Case – Translational Motion • In this case H and F commute: • SR is decomposed into 2 steps • Find blur HR image from LR images  non-iterative • Deconvolve the result using H  iterative

More Related