Image and Video Upscaling from Local Self Examples

1. Image and Video Upscaling from Local Self Examples Gilad Freedman Raanan Fattal Hebrew University of Jerusalem

2. Background and overview Algorithm description Local self similarity Non-dyadic filter bank Filter design Results

3. Single image upscaling • 1. large • 2. realistic • 3. faithful • 4. fast

4. Previous work parametric image model example based noisy result Freeman et al. 2002 Sun et al. 2008 Shan et al 2008 Fattal 2007 Glasner et al. 2009 generic looking edges

5. New approach: locale example based local self similarity small upscaling ratios corner step edge 4/5 Increase exemplar quality and size maintain search locality novel components: local self similarity non-dyadic filter bank line step edge 1/2 non smooth shading new non-dyadic filter bank

6. Background and overview Algorithm description Local self similarity Non-dyadic filter bank Filter design Results

7. Local self-examples upscaling interpolated image frequency content original image low pass high pass

8. Local self-examples upscaling For each patch: Add to interpolated image Take corresponding patch Search a local area for best example interpolated image frequency content low pass high pass

9. Local self-examples upscaling Repeat for all patches, to fill the high frequencies interpolated image frequency content low pass high pass

10. Overview Algorithm description Local self similarity Non-dyadic filter bank Filter design Results

11. Local self similarity croppeddownscaled

12. Local self similarity Patches in original image can matched locallywith ones in downscaled version

13. Local examples are enough query dbimage local 4.0 2.9 3.55 1.6 1.05 1.05 2.7 2.05 2.05 image database full image best matches 3.3 2.96 3.06 6.5 5.61 5.61

14. external database local search Visual assessment – external, exact NN, local Large external example database global search Searching the entire image Searching local regions in image

15. Comparison of example search methods

16. Background and overview Algorithm description Local self similarity Non-dyadic filter bank Filter design Results

17. Need for non-dyadic scalings large ratios mixed ratios small ratios

18. Dyadic filters higher half full frequency content 1:2 lower half dyadic filter bank

19. Non-dyadic filter bank higher part 4:5 full frequency content 1:2 lower part non-dyadic filter bank

20. Non-dyadic filters: downscaling example for the 2:3 ratio: dyadic case: 1. convolve with 2 filters 1. convolve with one filter 2. subsample by 2 2. subsample each by 3

21. Non-dyadic filters: upscaling example for the 2:3 ratio: dyadic case: 1. zero upsample by 1 2. convolve with 1 filter 1. zero upsample by 2 2. convolve with 2 filters 3. sum

22. Use of the filters in upscaling Upscaling using inverse scaling filters Smoothing by downscaling and upscaling

23. Background and overview Algorithm description Local self similarity Non-dyadic filter bank Filter design Results

24. 1. Uniform stretch • When interpolating, smooth areas come from inputUniformly spaced grids should remain uniform 255 brightness 0 grid coordinates

25. 2. Consistency The interpolated image, if downscaled should be equal to the input. Formally, upsample downsample Previous methods achieve consistency by solving large linear systems to achieve this property

26. 3. PSF modeling Difference between point spread functions Large image - small camera point spread function Small image - large camera point spread function

27. 4. Low frequency span frequency When upsampling don’t add new frequencies Upsamplingfilter should be low-pass original interpolated

28. 5. Singularities preservation similar amount of blur blurred Image interpolated image

29. Real time video upsampling on GPU main GPU memory Search and filter-banks are both local operations GPU cores NTSC to full HD @ 24 fps

30. Background and overview Algorithm description Local self similarity Non-dyadic filter bank Filter design Results

31. Ours X3 (zoomed in) Bicubic x3 (zoomed in)

32. Ours X3 Bicubic x3

33. Genuine Fractals™ x4 Ours X4

34. Ours X4 Glasner et al. 2009 x4

37. Thank you! Paper & additional results can be found at: www.cs.huji.ac.il/~giladfreedmn