Super Resolution Image Reconstruction

1. Super Resolution Image Reconstruction MuharremMercimek 8 September 2010

2. Contents • Motives for Super-resolution (SR) • How does SR works? • Classification of SR Methods, Background • Ideas on Proposal • Conclusions

3. Motivation for Super resolution • The advances in electronics, sensors, and optics have enabled widespread availability of monitoring and video based surveillance systems. • Many applications ranging from security to broadcasting are driving need for better understanding of the objects especially to highlight individuals and threats from video data. • video surveillance systems, • CCTV (Closed Circuit TV) based surveillance, • Web cameras attached to your PC, • IP (Internet Protocol) camera technology. www.ballardian.com, www.cctv-leeds.co.uk

4. Motivation for Super resolution • Often there is a tradeoff between temporal resolution of a measurement and its spatial resolution. • The user can risk having high spatial resolution and favor having as many as possible frames A section of a video with multiple LR frames Temporal Resolution [fr/sec] or duration of a frame Spatial Resolution [pix/inch] or length of a pixel • For surveillance systems, to record the scene for a longer time the spatial resolution can intentionally be kept low, due to practical storage restrictions. • For medical applications the patient would be the target of high amount of rays, to achieve high resolution data. • This leads sensor manufacturing techniques to be replaced with signal processing techniques to increase the spatial resolution

5. SR Method in Practical Use Raw Image Sequence Rough Image Registration SR Image Reconstruction 2x Optical zoom (ground truth) Raw Image (Cropped) 2x Interpolation 2x SR image (output) [F. Sroubek, 2007]

6. Application Areas Application Areas • Synthetic zooming of region of interest (ROI in surveillance, forensic, scientific, medical, and satellite imaging. • A digital video recorder (DVR) is currently replacing the close circuit television (CCTV) system, and it is often needed to magnify objects in the scene such as the face of a criminal or the license plate of a car for surveillance or forensic purposes. • The SR technique is also useful in medical imaging such as computed tomography (CT) and magnetic resonance imaging (MRI) since the acquisition of multiple images is possible while the resolution quality is limited. • In satellite imaging applications such as remote sensing and LANDSAT, several images of the same area are usually provided, and the SR technique to improve the resolution of target can be considered. • Conversionfrom an NTSC video signal to an HDTV signal.

7. Observation Model HR image acquisition plane LR image acquisition plane [Schultz94] the continuous scene or image to be projected onto a sensor plane LR image HR image HR image

8. Observation Model Scene Wkf DVkWkf Spatial Sampling Warping Blur Decimation Optical Blur Motion Blur etc. Continuous to discrete Translation Rotation CCD scan nk noise gk = DVkWkf +nk kth LR Image f HR Image VkWkf matrix represents the behavior of the system at the kth time instant. The geometric deformation or warping LR frame at the kth time instant, or kth frame The decimation operator that models the function of the CCD sensors and consists of convolution process with sensor’s PSF. Volatile blurs atthe kth time instant HRdiscrete representation of the continuous scene function The scene image The world coordinates The image grid Noise at the kth time instant

9. Observation Model Instabilities in videos motion degradations such as atmospheric turbulence, change of the camera position Due to the physical conditions such as the • imperfections of measuring devices • instability of the observed scene Vibrating imaging systems Geometric deformation –warping- Media turbulence Movement of local objects out of focus blur relative camera-scene motion blur due to shutter speed; the acquired images are often blurred, noisy and corrupted with insufficient spatial resolution.

10. f1 fk fk fK f2 A1k A1 Akk Ak AK AKk … … … … A2k A2 g1 g1 gk gk gK gK g2 g2 • Temporal Coincident and Temporal non-coincident observation models • excluding the noise term (adopted from [Borman99].) matrix represents the behavior of the system at the kth time instant.

11. How Super resolution works • Holding a digital camera and taking a series of shots in a very short time. • The small vibrations of the user’s hand during image acquisition are sufficient to reconstruct a HR image. License Plate video 4.5x zoom from 50 ft License Plate video 9x zoom from 50 ft

12. How Super resolution works • The straightforward idea of interpolating a single image does not produce a sufficient high resolution (HR) image. • A single image interpolation cannot recover the high frequency components (specifically spatial frequency components) lost or degraded during the LR sampling process • For this reason, image interpolation methods are not considered as super resolution (SR) techniques. • The idea behind the SR image reconstruction is to combine the information from a set of slightly different LR images of the same scene and use it to construct a HR image or a sequence, (better representation of the scene) with more resolving power. • Having plentiful images having INTER-DEPENDENCE available adds stability to SR image reconstruction techniques.

13. How Super resolution works LR grid HR grid HR Image Registered Frames • Motion Estimation • Registration • Reconstruction or • Mapping to HR grid • {Iterative, Non-iterative} LR Frames Basic premise for traditional SR image reconstruction; all frames are aligned onto the reference frame -top left-. Sub-pixel transformations take place. Image reconstruction is applied to map registered LR images to HR image grid. Two (or three) stage SR methods are consist sub-pixel registration for motion suppression and overlay of the registered LR images on a high grid with interpolating the missing values, are named as traditional SR approaches.

14. SR METHODS Classification of SR methods Sequential Methods Simultaneous Methods LR frames and other previously estimated HR frames can be used in SR algorithm Frequency Domain Methods Spatial Domain Methods Wavelet –based Methods Learning Based Methods Reconstruction Based Methods Powerful transforms can be used to highlight salient structures of LR frames Imaging system can be modeled using single input output pairs

15. Classification of SR methods • Several classifications under the light of different aspects of observation function • In [Juan09, Vandawalle06, Keren88] SR methods are divided into two main categories; frequency and spatial domain methods. • Discrete cosine transform (DCT)-based SR method which was proposed in [Rhee99]. • Wavelet-based SR methods are employed so many times recently [Wheeler07, Whillet03, Nguyen00]. • The authors in [Kim10, Yu08] state two SR categories based on whether a training stage is employed in SR restoration, as reconstruction based SR methodsand learning based SR methods. • In [Zibetti05, Borman99] the authors classify the SR methods as; sequential algorithms estimating the HR frames one at time, using many LR frames and other HR frames previously estimated, and simultaneous algorithms which estimate the entire sequence where all HR frames are restored, in one process. • Sroubek et. al [Sroubek07] proposed a Blind SR method, considering all the components of the image observation model. In that paper two (or three) stage SR methods are named as traditional SR approaches, which are not considering all the components of the observation model.

16. SR methods • Wavelet-based SR methods • The ideal algorithm for SR should be fast, and should add sharpness and details, both at edges and in regions without adding artifacts. • In [Hsu04] A wavelet based SR is divided mainly into three stages; image registration, wavelets based fusion and image deblurring. • The wavelet based fusion is performed to overcome the need for retaining edges like detailswhen going from LR images to HR images. • High frequency coefficients across all registered frames are combined by a fusion scheme • Reference image is up-sampled using interpolation DWT is applied. • High frequency coefficients are replaced with the fused high frequency coefficients which is the essence of the studies employing wavelet based super resolution [Wheeler07. Hsu04] • Cons.: Discrete Wavelet Transform (DWT) performs poorly when the frames contain motion like blurs.

17. SR methods • Frequency domain methods • Fourier analysis governs the use of periodicity of the patterns in an image. • A major class of SR methods utilizes a frequency domain formulation • Frequency domain SR methods provide the advantages • theoretically simple, • low computational complexity • an intuitive de-aliasing SR mechanism. • Frequency domain methods contribute to the solution of the problem with the light of the basic principles; • the shifting propertyof FT, • aliasing relationship frames [Borman98]. The resolving power can also be increased by bringing in high frequency information based on the image model or by removing the aliasing ambiguity [Vandawalle06].

18. SR methods • Frequency domain methods • Yang and Schonfield[Yang09] investigated to improve performance analysis of super-resolution (SR) image reconstruction. • They derived lower bounds on the resolution enhancement factor based on a frequency-domain SR algorithm. 18

19. SR methods • Frequency domain methods • Cons.: • Frequency domain model cannot accommodate local motions, and cannot compansate the effects of spatially varying phenomena [Borman98]. • The reason for favoring spatial domain methods over frequency domain methods was indicated in Keren et.al [Keren88] as it proved to be less sensitive to our noisy environment.

20. SR methods • Spatial domain methods • In this class of SR reconstruction methods, the observation model is formulated, and reconstruction is employed in the spatial domain. • Unlike frequency domain methods The linear spatial domain observation model can accommodate both global and non-global motions, and handle compensate spatially varying degradations. • Spatial domain reconstruction allows natural inclusion of (possibly nonlinear) spatial domain a-prioriconstraints (e.g. Markov random fields or convex sets) which result in bandwidth extrapolation in reconstruction. Statistical methods such as the algorithms using image priori’s can fall into this class. • But up to a degree for naïve application scenarios many spatial features like edges or lines to be matched can be located in the LR images. • Cons: These methods do not try to bring back the high frequency components lost after coarse sampling of the imaging system during data acquisition.

21. SR methods • Blind SR Methods The observation model covers three distinct cases frequently encountered in literature [Sroubek07]. • to resolve the geometric degradation, we face a (1) registration problem. • the decimation operator and the geometric transform are not considered, we face a (2) multi-frame blind deconvolutionproblem. • the volatile blur is not considered or assumed known, and is suppressed up to a sub-pixel translation, we obtain a (3) traditional SR formulation.

22. SR methods • Blind SR Methods Sroubek et.al proposed blind super resolution (BSR) method considering all of these cases in The main assumption was that the complex geometric deformations are removed in the preprocessing steps leaving a small misalignment. In this case the small misalignment (translation) is included into the simultaneous estimation of blur function and HR function. An iterative regularization is carried out in image and blur domain.

23. SR METHODS SR Ideas 1 Sequential Methods Simultaneous Methods Frequency Domain Methods Spatial Domain Methods Wavelet –based Methods • Empirical mode decomposition (EMD) algorithm for registered LR frames. • Reconstruction of the HR image using the reliable IMF Components from LR frames LR grid HR grid Learning Based Methods Reconstruction Based Methods HR Image Registered Frames • Motion Estimation • Registration • Reconstruction or • Mapping to HR grid • {Iterative, Non-iterative} LR Frames

24. SR Ideas 2 • Investigate the temporal smoothness (continuous motion of the frames in time) of the frames. Estimation of motion trajectory function. Model the continuous motion in time. • Find outliers which may break the continuous motion. Such as the sudden changes seen in the frames. • Rather than using registration algorithms, find the inter - dependence function of sequence LR grid HR grid HR Image Registered Frames • Motion Estimation • Registration • Reconstruction or • Mapping to HR grid • {Iterative, Non-iterative} LR Frames

25. SR Ideas 3 • Investigate if there are any LR frames, not suitable for the reconstruction algorithm, such as finding the bad candidates and eliminating them in the reconstruction step. Find the degree of compatibility of each LR frame. LR grid HR grid HR Image Registered Frames • Motion Estimation • Registration • Reconstruction or • Mapping to HR grid • {Iterative, Non-iterative} LR Frames

26. References • [Borman99] S. Borman, and R. L. Stevenson, "Simultaneous multi-frame MAP super-resolution video enhancement using spatio-temporal priors," Image Processing,. Proceedings. IEEE International Conference on, ICIP 99. pp. 469-473, 1999 • [Keren88] D. Keren, S. Peleg, and R. Brada, “Image sequence enhancement using sub-pixel displacements,” in Proceedings of IEEEComputer Society Conference on Computer Vision and PatternRecognition (CVPR ’88), pp. 742–746, Ann Arbor, Mich, USA, June 1988. • [Hsu04] Hsu J. T., Yen C. C., Li C. C., Sun M., Tian B., and Kaygusuz M., “Application of wavelet-based POCS super resolution for cardiovascular MRI image enhancement”, Proc. 3rd International Conference on Image and Graphics (Hong Kong, China), pp. 572 – 575, Dec. 2004. • [Fan06] C. Fan, J. Zhu, J. Gong, and C. Kuang, “POCS super-resolution sequence image reconstruction based on improvement approach of Keren registration method”, Proceedings of the Sixth International Conference on Intelligent Systems Design and Applications (ISDA ’06), pp. 333 – 337, 2006. • [Kim10] K. I. Kim, Y. Kwon, "Single-image super-resolution using sparse regression and natural image prior" Pattern Analysis and Machine Intelligence, IEEE Transactions on , vol.32, no.6, pp.1127-1133, June 2010. • [Nguyen00] N. Nguyen and P. Milanfar, “A wavelet based interpolation restoration method for super resolution”, Circuits Systems Signal Processing, vol. 19, pp.321-338, 2000. • [Rhee99] S.H. Rhee and M.G. Kang, “Discrete cosine transform based regularized high-resolution image reconstruction algorithm,” Opt. Eng., vol. 38, no. 8, pp. 1348-1356, Aug. 1999. • [Park03] S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Processing Magazine, vol. 20, no. 3, pp. 21–36, 2003.

27. References • [Reddy96] B. S. Reddy, B. N. Chatterji, “An FFT-based technique for translation, rotation, and scale-invariant image registration,” Image Processing, IEEE Transactions on, vol. 5, no. 8, pp. 1266– 1271, 1996. • [Schultz94] R. R. Schultz and R. L. Stevenson, “A Bayesian Approach to Image Expansion for Improved Definition”, Image Processing, IEEE Transactions on, vol. 3, no. 3, pp. 333-342, May, 1994. • [Sroubek07] F. Sroubek, G. Cristobal, J. Flusser, “A unifed approach to super resolution and multichannel blind deconvolution”, Image Processing, IEEE Transactions on , vol.16, no.9, pp.2322-2332, 2007. • [Vandawalle06] P. Vandawalle, S. Süsstrunk, and M. Vetterli, “A frequency domain approach to registration of aliased images with application to super-resolution”, EURASIP Journal on Applied Signal Processing, pp. 1-14. 2006. • [Wheeler07] Wheeler F.W., Liu X.M., and Tu P.H., Multi-frame super-resolution for face recognition, Proc. 1st International Conference on Biometrics Theory, Applications and Systems (Washington D.C), Sep. 27-29 pp. 1-6, 2007. • [Willet03] R. Willet, I. Jermyn, R. Nowak, and J. Zerubia, Wavelet based super resolution in astronomy, Proc. Astronomical Data Analysis Software and Systems, vol. 314, pp. 107-116, 2003. • [Yang09] J. Yang and D. Schonfeld, "New results on performance analysis of super-resolution image reconstruction," Image Processing (ICIP), 2009 16th IEEE International Conference on, pp.1517-1520, 7-10 Nov. 2009. • [Yu08] J. Yu and B. Bhanu, "Super-resolution of deformed facial images in video," Image Processing, IEEE International Conference on, ICIP 2008, pp.1160-1163, 12-15 Oct. 2008. • [Zibetti05] M.V.W. Zibetti, and J. Mayer, "Simultaneous super-resolution for video sequences," Image Processing, IEEE International Conference on, ICIP 2005 , vol.1, pp.11-14, 14 Sept. 2005