Super Resolution (SR) methods

1. Super Resolution (SR) methods Muharrem Mercimek Lab Presentation 27 July 2009

2. Contents • Why Super-resolution? • Image Recovery Ideas • Main SR Approaches • Standard SR vs. BSR • Theory of BSR • Conclusions

3. Why Super-resolution? • Imaging plays a key role in many diverse application areas. • Due to the imperfections of measuring devices instability of the observed scene (motion, media turbulence) acquired images are blurred noisy and corrupted with insufficient spatial temporal resolution • the resolution w.r.t time. There is a trade off between spatial resolution and temporal resolution. Objective: • In order to recover the original image techniques called blind deconvolution and super-resolution, to remove the blur and to increase the resolution respectively. • For one single observation the problem is ill-posed. W assume we have multiple LR observations, differences between images are very important to provide information abut the original scene. : Observed image LR [1] : Decimation Operator : Volatile PSF : Original Scene image : Geometric deformation : Acquisition index : Noise

4. Sroubek’s BSR Method In practical use Original Image Sequence Rough Registration (preprocessing) 2x Interpolation 2x SR image (output) PSFs for the frames 2x Optical zoom (ground truth) [1]

5. SR Principle to obtain higher resolution videos: [1]

6. Image Recovery Ideas • Deconvolution techniques can hardly cope with low resolution images in this case the standard convolution model is violated. • On the contrary state of the super-resolution techniques achieve remarkable results in resolution enhancement by estimating the sub-pixel shifts between images for calculating the blurs but lack any apparatus for calculating the blurs. • General assumptions: 1- We face with a registration problem, if we want to resolve the geometric degradation. 2-if the decimation operator and the geometric transform are not considered, we face a multichannel blind deconvolution problem. 3-Third, if the volatile blur is not considered or assumed known and is suppressed up to a sub-pixel translation we obtain a classical SR problem. And if all problems are considered we will face a blind super-resolution problem.

7. Standard SR vs. BSR Sub-pixel shifts during acquisition [2] Pixels on the LR grid Polyphase decomposition • The standard SR approach consist of sub-pixel registration overlaying the LR images on an HR grid. And interpolating the missing values. • In Sroubek’s super-resolution method, complex geometric transforms are removed in the preprocessing step and only a small misalignment is left behind. • And this translation is included into the estimation of volatile blurs.

8. Main SR Approaches Many papers address the standard SR problem Maximum likelihood (ML), maximum a posteriori (MAP), the set theoretic approach using POCS (projection on convex sets), and fast Fourier techniques can all provide a solution to the SR problem. Earlier approaches assumed that sub-pixel shifts are estimated by other means. Other approaches focus on fast implementations; space-time SR or SR of compressed video. In general, most of the SR techniques assume a priori known blurs. SR that can handle parametric PSFs, PSFs modeled with one parameter This restriction is unfortunately very limiting for most real applications. First attempts for BSR with an arbitrary PSF appeared in [3]. The interesting idea proposed therein is the conversion of the SR problem from SIMO to multiple input multiple output (MIMO) using so-called polyphase components.

9. Theory of BSR Acquisition Model Regularization Volatile blur matrix Decimation Operator Unknown translation and Volatile PSF Image regularization PSF regularization Regularization is carried out in both the image and the blur domains. AM method is use to tackle the minimization problem.

10. Sroubek’s BSR method Is applied to Army video sequence One Frame from Test Video (LR Image) The single HR output obtained using 20 frames 20 PSF

11. Conclusions • This is the only method that can perform deconvolution and resolution enhancement simultaneously. • The experiments with promising results give the reader a precise notion of the quality of the BSR methodology and wide applicability of the proposed algorithm to all sorts of real problems.

12. References [1] Šroubek Filip, Cristóbal G., Flusser Jan: A Unified Approach to Superresolution and Multichannel Blind Deconvolution, IEEE Transactions on Image Processing vol.16, 9 (2007), p. 2322-2332 [2] Sroubek F., Flusser J., Cristobal G.: Multiframe blind deconvolution coupled with frame registration and resolution enhancement. In: Blind Image Deconvolution: Theory and Applications. (Campisi P., Egiazarian K. eds.). CRC Press, to be published in 2007. [3] A.E. Yagle. Blind superresolution from undersampled blurred measurements. In Advanced Signal Processing Algorithms, Architectures, and Implementations XIII, volume 5205, pages 299–309, Bellingham, 2003, SPIE.