Super-Resolution

1. Super-Resolution Authors: Wagner, Waagen and Cassabaum Presented By: Mukul Apte

2. Introduction • Definition: Create High Resolution visual output from Low Resolution visual input. • Mathematical assistance to features viz. motion detection, face recognition, person detection. • Action Packed Sports Images • Astronomy • Medical Imaging • Surveillance • Range of resources: low (single camera, polaroid lenses) to high (super-resolution chips, androids-ASIMO)

3. Basic idea Given: A set of degraded (warped, blurred, decimated, noised) images: Required: Fusion of the measurements into a higher resolution image:

4. Types • Static Super-Resolution (SSR) - The creation of a single improved image, from the finite measured sequence of images • Dynamic Super-Resolution (SSR) - Low Quality Movie In - High Quality Movie Out

5. Simple Example • Concepts I: Raster, CCD, Image in mathematical domain, Matlab • Concepts II: Bresenham’s line algorithm, non-uniform interpolation, frequency domain

6. For a given band-limited image, the Nyquist sampling theorem states that if a uniform sampling is fine enough (pixel density = twice highest static frequency), perfect reconstruction is possible. D D Simple Example

7. 2D 2D Simple Example Due to our limited camera resolution, we sample using an insufficient 2D grid

8. 2D 2D Simple Example However, we are allowed to take a second picture and so, shifting the camera ‘slightly to the right’ we obtain

9. Simple Example Similarly, by shifting down we get a third image 2D 2D

10. Simple Example And finally, by shifting down and to the right we get the fourth image 2D 2D

11. This is Super-Resolution in its simplest form Simple Example - Conclusion It is trivial to see that interlacing the four images, we get that the desired resolution is obtained, and thus perfect reconstruction is guaranteed.

12. Uncontrolled Displacements In the previous example we counted on exact movement of the camera by D in each direction. What if the camera displacement is uncontrolled?

13. Uncontrolled Displacements It turns out that there is a sampling theorem due to Yen (1956) and Papoulis (1977) covering this case, guaranteeing perfect reconstruction for periodic uniform sampling if the sampling density is high enough (1 sample per each D-by-D square).

14. Uncontrolled Rotation/Scale/Disp. In the previous examples we restricted the camera to move horizontally/vertically parallel to the photograph object. What if the camera rotates? Gets closer to the object (zoom)?

15. There is no sampling theorem covering this case Uncontrolled Rotation/Scale/Disp.

16. Further Complications • Sampling is not a point operation – there is a blur • Motion may include perspective warp, local motion, etc. • Samples may be noisy – any reconstruction process must take that into account.

17. Static Super-Resolution Static Super-Resolution t Low Resolution Measurements Static Super-Resolution Algorithm High Resolution Reconstructed Image

18. Dynamic Super-Resolution Algorithm Dynamic Super-Resolution Low Resolution Measurements t High Resolution Reconstructed Images t

19. Approach • Image Registration • Motion Estimation • Projection onto High-Resolution Grid • Non-Uniform Interpolation • Frequency domain – alias correction Projection Registration Low-res Images Registration (sub-pixel grid) High Res Grid

20. 1.1 Registration (angle) • Rotation Calculation • Correlate 1st LR image with all LR images at all angles • OR • Calculate energy at all angles for all LR images. Correlate energy vector to find the rotation angle Anglei = max index(correlation(I1(θ), Ii (θ))) LR image 1 LR image 2 i = 2,3,..,N (number of LR images) Energy at angle Ii(θ) Energy at angle I2(θ)

21. Fi (uT) = ej2πuΔsF1(uT) Δs = angle( Fi (uT) / F1(uT) ) 2πu 1.1 Registration (shift) • Shift Calculated using Frequency Domain Method Δs  [Δx Δy]T u [fx fy] • Used only 6% lower u (high freq could be aliased) • Used least square to calculate Δs

22. 2.1 Frequency Domain • Input: Down-sampled aliased images • Goal I: Correct the low-freq aliased data • Goal II: Predict the lost high freq values -π π -π π Down-sampled Original High-Res Aliased (fix it) Lost (find it) -π/2 π/2 π -π π Up-sampled Desired High-Res

23. I (known pixel positions) = Known Values I_fft = fft2(I) I_fft(higher Freq) = 0 I= ifft2 (I_fft) 2.2 Projection onto High-res grid • Papoulis-Gerchberg Algorithm • Projection onto convex sets • Known pixel values • Known Cut-off freq in the HR image • Algorithm:

24. Geometric Warp Blur Decimation Y High- Resolution Image 1 F =I H V D D V H Low- Resolution Images 1 1 1 N 1 X Additive Noise Y N F N N N SSR – The Model

25. 0 0 F[j,i]=1 Warp – Linear Operation Z X Per every point in X find a matching point in Z

26. Model Assumptions We assume that the images Yk and the operators Hk, Dk, Fk,& Wk are known to us, and we use them for the recovery of X. Yk – The measured images (noisy, blurry, down-sampled ..) Hk – The blur can be extracted from the camera characteristics Dk – The decimation is dictated by the required resolution ratio Fk – The warp can be estimated using motion estimation Wk – The noise covariance can be extracted from the camera characteristics

27. Special Condition - Noiseless Clearly, this linear system of equations should have more equations than unknowns in order to make it possible to have a unique solution.

28. SSR – Handling Problems Single image de-noising Single image restoration Single image scaling Motion compensation average Using

29. All the above operations can be interpreted as iterative operations performed on images. (Typically 15-20 iterations are performed.) Back projection Simulated error Weighted edges SSR Standard Equation

30. Thumb Rule on Desired Resolution • Assume that we have N images of M-by-M pixels, and we would like to produce an image X of size L-by-L. Then –

31. Papoulis – Gerchberg Algorithm Initial Setup FFT (Initial image) Taj Mahal – Low-res image I

32. Papoulis – Gerchberg Algorithm Known Pixel Values Image at iteration 0 Image after 1st iteration FFT I(high freq) =0

33. Papoulis – Gerchberg Algorithm Known Pixel Values Image at iteration 1 Image after 10 iterations FFT I(high freq) =0

34. Papoulis – Gerchberg Algorithm After 50 iterations Bilinear Interpolation Bicubic Interpolation SR Reconstructed image

35. Results (Images - I) • Input: 4 snaps using a high-res digital camera • Cropped the same part of each image • SR algorithm compared with bicubic interpolation Original Low-res images (Courtesy: Patrick Vandewalle)

36. Results (Images - I) Bicubic Interpolation

37. Results (Images - I) Super-resolution

38. Results (Images - II) Low-Res Image I Low-Res Image II • SR Algorithm didn’t work as expected !!! • Reason: • Motion was not restricted to shifts & rotation • Images had affine mapping • Rule I - Need Correct Registration

39. Results (Frames - I) Original Bicubic SR • Why didn’t SR work??? • Low-res images were created by forcing shifts at critical velocities • Rule II - If low-res frames are at critical velocities, can’t create good high-res frame

40. Results (Frames - II) Original Bicubic SR • Why did SR work so well??? • Low-res images were created by forcing shifts at non-critical velocities • Rule III  If low-res images have all the info about high-res then HR image can be perfectly constructed

41. Future Work • Superresolution with multiple motions between frames - create high res video • Predict the high-res frequency components using wavelet methods Predict Predict Predict

42. Acknowledgements • Prof John Apostolopoulos • Prof Susie Wee • Patrick Vandewalle

43. THANK YOU!!! QUESTIONS? COMMENTS?