1 / 23

Optimized neighbor embeddings for single-image super-resolution

Optimized neighbor embeddings for single-image super-resolution. Mehmet Türkan , Dominique Thoreau, Philippe Guillotel , Charly Poyac. Summary. Basic concepts Principle Quality Improvements Complexity improvements Results Conclusion & remaining work. Basic concepts.

hayes
Download Presentation

Optimized neighbor embeddings for single-image super-resolution

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Optimized neighbor embeddings for single-image super-resolution Mehmet Türkan, Dominique Thoreau, Philippe Guillotel, Charly Poyac

  2. Summary • Basic concepts • Principle • Quality Improvements • Complexity improvements • Results • Conclusion & remaining work

  3. Basic concepts Original low res image Nearest neighbor interpolation To resize an image, standard algorithms are interpolation based (linear, cubic…), they are fast but they cannot recovery high frequencies and the result is often blur. Bicubic interpolation

  4. Basic concepts Super-resolution is class of techniques that enhance the resolution of an imaging system. Methods for SR can be broadly classified into two families of methods: The classical multi-image super-resolution[3], and the Example-Based super-resolution[4][5][7][8].

  5. Objectives of the internship • Starting from a reference algorithm designed by Mehmet Turkan, Dominique Thoreau, and PhillipeGuillotel, my job was: • Switch original software developed from Matlab to C/C++. • Speed up the processing time and enhance the quality as much as possible. Final objective is to upsampleFullHD sequences to UHD in a suitable time.

  6. Summary • Basic concepts • Principle • Quality Improvements • Complexityimprovements • Results • Conclusion & remaining work

  7. Principle Small patches are very likely to repeat themselves within and across different scales of an image. We can reconstruct a coherent high resolution image with patches correspondence between low res and downsampled images.

  8. Principle : minimization With the knn-search, we have K couples of low res and high res patches. LLE (locally linear embedding) is computed with low res patches to approximate the input patch and weights obtained are used to merge high res patches. The best K patches combination is obtained by a downsampling of the merged patch and a SSD comparison with the input LR patch.

  9. Summary • Basic concepts • Principle • Quality Improvements • Complexityimprovements • Results • Conclusion & remaining work

  10. QualityImprovements: patch orientation To increase the result of the knn-search, it’s possible to seek the input patch plus his transformations (transpose, symmetric X, symmetric Y and combinations). Result is better but the knn-search is about 8 times slower.

  11. Quality Improvements: textural features We can improve the opportunity of find similar patches in the knn-search by using a textural version of all the patches. standard textural In the same way, the LLE is computed with textural patches, and the mean value of the input patch is added to the result because the upsampling of the mean is itself.

  12. Quality Improvements: attempts • Using a bicubic neighborhood instead of padding for the patch downsampling. • Compute a first pass with a big value of K, and a second pass with k varying and using the neighborhood of the first pass. • Downsampling result and compare it with the input image to detect error areas in order to recompute with better parameters. • Continue to minimize the patch combination until it’s over a maximum error. • Nonlinear minimization instead of LLE.

  13. Summary • Basic concepts • Principle • Quality Improvements • Complexityimprovements • Results • Conclusion & remaining work

  14. Complexity improvements: windows search Instead of the full phases surface, it’s possible to significantly speedup the knn-search using a window. Results are close to a full search because corresponding patches are often beside the input patch position.

  15. Complexity improvements: single channel SR Because the Human visual system is principally sensible in luminance, it’s possible to convert an image from RGB to YUV colorspace and work only in Y channel, U and V channels are interpolated with Lanczos algorithm. HR image Visually, results are very close to a multi channel processing but the compute time is faster. LR image channels

  16. Complexity improvements: minimization Histogram of best value of K Number of patches Standard minimization : K takes all value until Optimized minimization, only small values of k are computed + max value K → Result is very close to a full minimization with all values of K.

  17. Summary • Basic concepts • Principle • Quality Improvements • Speed improvements • Results • Conclusion & remaining work

  18. Results : Upsampling x4 for Oldman LANCZOS OUR[8] Original NEAREST NEIGHBOR

  19. LANCZOS OUR ORIGINAL

  20. Summary • Basic concepts • Principle • Quality Improvements • Complexityimprovements • Results • Conclusion & remaining work

  21. Conclusion & Remaining work • Advantages: • Results are excellent compared to standards interpolation methods (bicubic, Lanczos…), average is 1.5dB better, many details appear. • It’s visually better than all other super resolution algorithms. • Drawbacks: • Very slow to compute, about 1 hour to upsample a FullHD image into UHD, but it’s 20 times faster than the reference Matlab software. • Remaining work: • CUDA implementation: • Knn-search, matrix inversion and patch combination are veryparallelizable. • Discontinuity problems with sequences: Lot of artifactsappear, especially on edges.

  22. References H. Chang, D.-Y. Yeung, and Y.Xiong, “Super-resolution through neighbor embedding,” in IEEE Comp. Soc. Conf. Comp. Vis. Pattern Recogn., 2004, pp. 275-282. W. T. Freeman, E. C. Pasztor, and O. T. Carmichael, “Learning low-level vision,” Int. J. Comp. Vis., vol. 40, no. 1, pp. 83-97, 2004. J. Yang, J. Wright, T. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Trans. Image Process., vol. 19, pp. 2861-2873, 2010. D. Glasner, S. Bagon, and M. Irani, “Super-resolution from a single image,” in IEEE Int. Conf. Comp. Vis., 2009, pp. 349-356. G. Freedman and R. Fattal, “Image and video upscaling from local self-examples,” ACM Trans. Graph., vol. 28, no. 3, pp.1-10, 2010. S. Roweis and L. Saul, “Nonlinear dimensionality reduction by locally linear embedding,” Science, vol. 290, pp. 2323-2326, 2000. M. Turkan, D. Thoreau, and P. Guillotel, “Optimized neighbor embeddings for single-image super-resolution,” To be presented in ICIP 2013. M. Turkan, D. Thoreau, and P. Guillotel, “Self-content super-resolution for ultra-HD up-sampling,” in Proc. European Conf. Visual Media Prod., 2012, pp. 49–58.

More Related