1 / 38

“Locally Adaptive Patch-based Image and Video Restoration”

Thank you for participating in and contributing to our mini-symposium on . “Locally Adaptive Patch-based Image and Video Restoration”. Session I: Today (Mon) 10:30 – 1:00 Session II: Wed Same Time, Same Room. Local Adaptivity + Patch-Based Approaches .

radley
Download Presentation

“Locally Adaptive Patch-based Image and Video Restoration”

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Thank you for participating inand contributing to our mini-symposium on “Locally Adaptive Patch-based Image and Video Restoration” Session I: Today (Mon) 10:30 – 1:00 Session II: Wed Same Time, Same Room Milanfar et al. EE Dept, UCSC

  2. Local Adaptivity + Patch-Based Approaches • State of the Art Performance A Convergence of Ideas Extremely Popular Milanfar et al. EE Dept, UCSC

  3. Patch-based methods have become so popular in fact …. Patchy the Pirate Milanfar et al. EE Dept, UCSC

  4. Multi-dimensional Kernel Regression for Video Processing and Reconstruction Peyman Milanfar* EE Department University of California, Santa Cruz *Joint work with Hiro Takeda (UCSC), Mattan Protter and Michael Elad (Technion), Peter van Beek (Sharp Labs of America) SIAM Imaging Science Meeting, July 7, 2008 Milanfar et al. EE Dept, UCSC

  5. Outline • Background and Motivation • Classic Kernel Regression • Data-Adaptive Regression • Adaptive Implicit-Motion Steering Kernel (AIMS) • Motion-Aligned Steering Kernel (MASK) • Conclusions Milanfar et al. EE Dept, UCSC

  6. Summary • Motivation: • Existing methods make strong assumptions about signal and noise models. • Develop “universal”, robust methods based on adaptive nonparametric statistics • Goal: • Develop the adaptive Kernel Regression framework for a wide class of problems, including video processing; producing algorithms competitive with state of the art. Milanfar et al. EE Dept, UCSC

  7. Outline • Background and Motivation • Classic Kernel Regression • Data-Adaptive Regression • Adaptive Implicit-Motion Steering Kernel (AIMS) • Motion-Aligned Steering Kernel (MASK) • Conclusions Milanfar et al. EE Dept, UCSC

  8. Kernel Regression Framework • The data model Zero-mean, i.i.d noise (No other assump.) The sampling position The number of samples A sample The regression function • The specific form of • may remain unspecified for now. Milanfar et al. EE Dept, UCSC

  9. Local Approximation in KR • The data model • Local representation (N-terms Taylor expansion) • Note • With a polynomial basis, we only need to estimate the first unknown, • Other localized representations are also possible, and may be advantageous. Unknowns Milanfar et al. EE Dept, UCSC

  10. Optimization Problem • We have a local representation with respect to each sample: • Minimization The regression order N+1 terms The choice of the kernel function is open, e.g. Gaussian. This term give the estimated pixel value at x. Milanfar et al. EE Dept, UCSC

  11. Locally Linear Estimator • The optimization yields a pointwise estimator: • The bias and variance are related to the regression order and the smoothing parameter: • Large N  small bias and large variance • Large h  large bias and small variance The smoothing parameter Kernel function The weighted linear combinations of the given data Equivalent kernel function The regression order Milanfar et al. EE Dept, UCSC

  12. Outline • Background and Motivation • Classic Kernel Regression • Data-Adaptive Regression • Adaptive Implicit-Motion Steering Kernel (AIMS) • Motion-Aligned Steering Kernel (MASK) • Conclusions Milanfar et al. EE Dept, UCSC

  13. (2D) Data-Adaptive Kernels • Take not only spatial distances, but also radiometric distances (pixel value differences) into account • Data-adaptive kernel function • Yields locally non-linear estimators Classic kernel Data-adapted kernel Milanfar et al. EE Dept, UCSC

  14. Simplest Case: Bilateral Kernels Spatial kernel Radiometric kernel . = . = . = Low noise case Milanfar et al. EE Dept, UCSC

  15. Better: Steering Kernel Method Local dominant orientation estimate based on local gradient covariance H. Takeda, S. Farsiu, P. Milanfar, “Kernel Regression for Image Processing and Reconstruction”, IEEE Transactions on Image Processing, Vol. 16, No. 2, pp. 349-366, February 2007. Milanfar et al. EE Dept, UCSC

  16. Steering Kernel Steering matrix • Kernel adapted to locally dominant structure • The steering matrices scale, elongate, and rotate the kernel footprints locally. Local dominant orientation estimation Elongate Rotate Scale Milanfar et al. EE Dept, UCSC

  17. Steering Kernel (Low Noise) • Kernel weights and footprints: Steering kernel as a function of xiwith x held fixed Steering kernel as a function of xwith xi and Hi held fixed Weights Footprints Low noise case Milanfar et al. EE Dept, UCSC

  18. Steering approach provides stable weights even in the presence of significant noise. Steering Kernel (High Noise) • Kernel weights and footprints: Steering kernel as a function of xiwith x held fixed Steering kernel as a function of xwith xi and Hi held fixed Weights Footprints High noise case Milanfar et al. EE Dept, UCSC

  19. Some Related (0th-order) Methods • Non-Local Means (NLM) • A. Buades, B. Coll, and J. M. Morel. “A review of image denoising algorithms, with a new one.” Multiscale Modeling & Simulation, 4(2):490-530, 2005. • Optimal Spatial Adaptation (OSA) • C. Kervrann, J. Boulanger “Optimal spatial adaptation for patch-based image denoising.” IEEE Trans. on Image Processing, 15(10):2866-2878, Oct 2006. SKR NLM OSA Milanfar et al. EE Dept, UCSC

  20. Adaptive Kernels for Interpolation • When there are missing pixels: • We cannot have the radiometric distance. • Using a “pilot” estimate, fill the missing pixels: • Classic kernel regression • Cubic or bilinear interpolation ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Milanfar et al. EE Dept, UCSC

  21. Outline • Background and Motivation • Classic Kernel Regression • Data-Adaptive Regression • Regression in 3-D • Adaptive Implicit-Motion Steering Kernel (AIMS) • Motion-Aligned Steering Kernel (MASK) • Conclusions Milanfar et al. EE Dept, UCSC

  22. Kernel Regression in 3-D • Setup is similar to 2-D, but….. • Data samples come from various (nearby) frames • Signal “structure” is now in 3-D • We can perform • Denoising • Spatial Interpolation • Frame rate upconversion • Space-time super-resolution Temporal gradients Spatial gradients Milanfar et al. EE Dept, UCSC

  23. Kernel Regression in 3-D Cont. • Two ways to proceed • Adaptive Implicit-Motion Steering Kernel (AIMS) • Roughly warp the data to “neutralize” large motions • Implicitly capture sub-pixel motions in 3-D Kernel • Motion-Aligned Steering Kernel (MASK) • Estimate motion with subpixel accuracy • Accurately warp the kernel (instead of the data) Milanfar et al. EE Dept, UCSC

  24. Outline • Background and Motivation • Classic Kernel Regression • Data-Adaptive Regression • Regression in 3-D • Adaptive Implicit-Motion Steering Kernel (AIMS) • Motion-Aligned Steering Kernel (MASK) • Conclusions Milanfar et al. EE Dept, UCSC

  25. AIMS Kernel in 3-D • Steering kernel visualization examples Isosurface A plane structure Steering kernel weights A tube structure Milanfar et al. EE Dept, UCSC

  26. Large displacements make orientation estimation difficult. By neutralizing the large displacement, the steering kernel can effectively spread again. The local kernel after motion compensation. Shift down Shift up AIMS Motion Compensation The local kernel effectively spread along the local motion trajectory. The local kernel effectively spread along the local motion trajectory. Small motions Large motions Important: The compensation does not require subpixel accurate motion estimation, nor does it require interpolation Milanfar et al. EE Dept, UCSC

  27. AIMS Contains Implicit Motion “Small” motion vector Space-time gradients of roughly compensated data Optical flow equation (Eigenvalues of C) Assuming the patch moves with approximate uniformity Homogeneous Optical Flow Vector Milanfar et al. EE Dept, UCSC

  28. AIMS Summary • AIMS is a two-tiered approach. • Neutralize whole-pixel motions. • 3-D SKR with implicit subpixel motion information Steering matrices estimated from the motion compensated data in 3-D. Milanfar et al. EE Dept, UCSC

  29. Foreman Example Input video (QCIF: 144 x 176 x 28) Lanczos (frame-by-frame upscaling) AIMS Factor of 2 upscaling Milanfar et al. EE Dept, UCSC

  30. Spatial Upscaling Example Input (200 x 200) Upscaled image by AIMS (multi-frame, 5 frames), 400x400 Milanfar et al. EE Dept, UCSC

  31. Spatiotemporal Upscaling Single frame steering kernel regression (400 x 400 x 20) Input video (200 x 200 x 20) Spatiotemporal classic kernel regression (400 x 400 x 40) AIMS regression (400 x 400 x 40) Milanfar et al. EE Dept, UCSC

  32. Outline • Background and Motivation • Classic Kernel Regression • Data-Adaptive Regression • Regression in 3-D • Adaptive Implicit-Motion Steering Kernel (AIMS) • Motion-Aligned Steering Kernel (MASK) • Conclusions Milanfar et al. EE Dept, UCSC

  33. Motion-Aligned Steering Kernel Accurate, explicit motion estimates “2-D motion-steered” (spatial) kernel 1-D (temporal) kernel Milanfar et al. EE Dept, UCSC • Motion is explicitly estimated to subpixel accuracy • Kernel weights are aligned with the local motion vectors using warping/shearing • The warped kernel acts directly on the data • Handles large and/or complex motions

  34. Intuition Behind the MASK 2-D “motion-steered” (spatial) kernel 1-D (temporal) kernel Milanfar et al. EE Dept, UCSC

  35. The Shapes of MASK Slices of MASK kernels Local data Milanfar et al. EE Dept, UCSC Spreads along spatial orientations and local motion vectors.

  36. A Comparison of AIMS and MASK Input video (200 x 200 x 20) AIMS (400 x 400 x 40) MASK (400 x 400 x 40) Milanfar et al. EE Dept, UCSC Spin Calendar video

  37. A Comparison of AIMS and MASK Input video (QCIF: 144 x 176 x 28) AIMS + BTV deblurring (CIF: 288 x 352 x 28) MASK + BTV deblurring (CIF: 288 x 352 x 28) Milanfar et al. EE Dept, UCSC Foreman video

  38. Conclusions Milanfar et al. EE Dept, UCSC • We extended the 2-D kernel regression framework to 3-D. • Illustrated 2 distinct approaches • AIMS: Avoids subpixel motion estimation, needs comp. for large motions • MASK: Needs subpixel motion estimation, deals directly with large motions • Which is better? Depends on the application. • The overall 3-D SKR framework is simultaneously well-suited for spatial, temporal, and spatiotemporal • upscaling, denoising, blocking artifact removal, superresolution • not only in video but in general 3-D data sets. • Future work • Integration of deblurring directly in the 3-D framework • Computational complexity

More Related