390 likes | 695 Views
Thank you for participating in and contributing to our mini-symposium on . “Locally Adaptive Patch-based Image and Video Restoration”. Session I: Today (Mon) 10:30 – 1:00 Session II: Wed Same Time, Same Room. Local Adaptivity + Patch-Based Approaches .
E N D
Thank you for participating inand contributing to our mini-symposium on “Locally Adaptive Patch-based Image and Video Restoration” Session I: Today (Mon) 10:30 – 1:00 Session II: Wed Same Time, Same Room Milanfar et al. EE Dept, UCSC
Local Adaptivity + Patch-Based Approaches • State of the Art Performance A Convergence of Ideas Extremely Popular Milanfar et al. EE Dept, UCSC
Patch-based methods have become so popular in fact …. Patchy the Pirate Milanfar et al. EE Dept, UCSC
Multi-dimensional Kernel Regression for Video Processing and Reconstruction Peyman Milanfar* EE Department University of California, Santa Cruz *Joint work with Hiro Takeda (UCSC), Mattan Protter and Michael Elad (Technion), Peter van Beek (Sharp Labs of America) SIAM Imaging Science Meeting, July 7, 2008 Milanfar et al. EE Dept, UCSC
Outline • Background and Motivation • Classic Kernel Regression • Data-Adaptive Regression • Adaptive Implicit-Motion Steering Kernel (AIMS) • Motion-Aligned Steering Kernel (MASK) • Conclusions Milanfar et al. EE Dept, UCSC
Summary • Motivation: • Existing methods make strong assumptions about signal and noise models. • Develop “universal”, robust methods based on adaptive nonparametric statistics • Goal: • Develop the adaptive Kernel Regression framework for a wide class of problems, including video processing; producing algorithms competitive with state of the art. Milanfar et al. EE Dept, UCSC
Outline • Background and Motivation • Classic Kernel Regression • Data-Adaptive Regression • Adaptive Implicit-Motion Steering Kernel (AIMS) • Motion-Aligned Steering Kernel (MASK) • Conclusions Milanfar et al. EE Dept, UCSC
Kernel Regression Framework • The data model Zero-mean, i.i.d noise (No other assump.) The sampling position The number of samples A sample The regression function • The specific form of • may remain unspecified for now. Milanfar et al. EE Dept, UCSC
Local Approximation in KR • The data model • Local representation (N-terms Taylor expansion) • Note • With a polynomial basis, we only need to estimate the first unknown, • Other localized representations are also possible, and may be advantageous. Unknowns Milanfar et al. EE Dept, UCSC
Optimization Problem • We have a local representation with respect to each sample: • Minimization The regression order N+1 terms The choice of the kernel function is open, e.g. Gaussian. This term give the estimated pixel value at x. Milanfar et al. EE Dept, UCSC
Locally Linear Estimator • The optimization yields a pointwise estimator: • The bias and variance are related to the regression order and the smoothing parameter: • Large N small bias and large variance • Large h large bias and small variance The smoothing parameter Kernel function The weighted linear combinations of the given data Equivalent kernel function The regression order Milanfar et al. EE Dept, UCSC
Outline • Background and Motivation • Classic Kernel Regression • Data-Adaptive Regression • Adaptive Implicit-Motion Steering Kernel (AIMS) • Motion-Aligned Steering Kernel (MASK) • Conclusions Milanfar et al. EE Dept, UCSC
(2D) Data-Adaptive Kernels • Take not only spatial distances, but also radiometric distances (pixel value differences) into account • Data-adaptive kernel function • Yields locally non-linear estimators Classic kernel Data-adapted kernel Milanfar et al. EE Dept, UCSC
Simplest Case: Bilateral Kernels Spatial kernel Radiometric kernel . = . = . = Low noise case Milanfar et al. EE Dept, UCSC
Better: Steering Kernel Method Local dominant orientation estimate based on local gradient covariance H. Takeda, S. Farsiu, P. Milanfar, “Kernel Regression for Image Processing and Reconstruction”, IEEE Transactions on Image Processing, Vol. 16, No. 2, pp. 349-366, February 2007. Milanfar et al. EE Dept, UCSC
Steering Kernel Steering matrix • Kernel adapted to locally dominant structure • The steering matrices scale, elongate, and rotate the kernel footprints locally. Local dominant orientation estimation Elongate Rotate Scale Milanfar et al. EE Dept, UCSC
Steering Kernel (Low Noise) • Kernel weights and footprints: Steering kernel as a function of xiwith x held fixed Steering kernel as a function of xwith xi and Hi held fixed Weights Footprints Low noise case Milanfar et al. EE Dept, UCSC
Steering approach provides stable weights even in the presence of significant noise. Steering Kernel (High Noise) • Kernel weights and footprints: Steering kernel as a function of xiwith x held fixed Steering kernel as a function of xwith xi and Hi held fixed Weights Footprints High noise case Milanfar et al. EE Dept, UCSC
Some Related (0th-order) Methods • Non-Local Means (NLM) • A. Buades, B. Coll, and J. M. Morel. “A review of image denoising algorithms, with a new one.” Multiscale Modeling & Simulation, 4(2):490-530, 2005. • Optimal Spatial Adaptation (OSA) • C. Kervrann, J. Boulanger “Optimal spatial adaptation for patch-based image denoising.” IEEE Trans. on Image Processing, 15(10):2866-2878, Oct 2006. SKR NLM OSA Milanfar et al. EE Dept, UCSC
Adaptive Kernels for Interpolation • When there are missing pixels: • We cannot have the radiometric distance. • Using a “pilot” estimate, fill the missing pixels: • Classic kernel regression • Cubic or bilinear interpolation ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Milanfar et al. EE Dept, UCSC
Outline • Background and Motivation • Classic Kernel Regression • Data-Adaptive Regression • Regression in 3-D • Adaptive Implicit-Motion Steering Kernel (AIMS) • Motion-Aligned Steering Kernel (MASK) • Conclusions Milanfar et al. EE Dept, UCSC
Kernel Regression in 3-D • Setup is similar to 2-D, but….. • Data samples come from various (nearby) frames • Signal “structure” is now in 3-D • We can perform • Denoising • Spatial Interpolation • Frame rate upconversion • Space-time super-resolution Temporal gradients Spatial gradients Milanfar et al. EE Dept, UCSC
Kernel Regression in 3-D Cont. • Two ways to proceed • Adaptive Implicit-Motion Steering Kernel (AIMS) • Roughly warp the data to “neutralize” large motions • Implicitly capture sub-pixel motions in 3-D Kernel • Motion-Aligned Steering Kernel (MASK) • Estimate motion with subpixel accuracy • Accurately warp the kernel (instead of the data) Milanfar et al. EE Dept, UCSC
Outline • Background and Motivation • Classic Kernel Regression • Data-Adaptive Regression • Regression in 3-D • Adaptive Implicit-Motion Steering Kernel (AIMS) • Motion-Aligned Steering Kernel (MASK) • Conclusions Milanfar et al. EE Dept, UCSC
AIMS Kernel in 3-D • Steering kernel visualization examples Isosurface A plane structure Steering kernel weights A tube structure Milanfar et al. EE Dept, UCSC
Large displacements make orientation estimation difficult. By neutralizing the large displacement, the steering kernel can effectively spread again. The local kernel after motion compensation. Shift down Shift up AIMS Motion Compensation The local kernel effectively spread along the local motion trajectory. The local kernel effectively spread along the local motion trajectory. Small motions Large motions Important: The compensation does not require subpixel accurate motion estimation, nor does it require interpolation Milanfar et al. EE Dept, UCSC
AIMS Contains Implicit Motion “Small” motion vector Space-time gradients of roughly compensated data Optical flow equation (Eigenvalues of C) Assuming the patch moves with approximate uniformity Homogeneous Optical Flow Vector Milanfar et al. EE Dept, UCSC
AIMS Summary • AIMS is a two-tiered approach. • Neutralize whole-pixel motions. • 3-D SKR with implicit subpixel motion information Steering matrices estimated from the motion compensated data in 3-D. Milanfar et al. EE Dept, UCSC
Foreman Example Input video (QCIF: 144 x 176 x 28) Lanczos (frame-by-frame upscaling) AIMS Factor of 2 upscaling Milanfar et al. EE Dept, UCSC
Spatial Upscaling Example Input (200 x 200) Upscaled image by AIMS (multi-frame, 5 frames), 400x400 Milanfar et al. EE Dept, UCSC
Spatiotemporal Upscaling Single frame steering kernel regression (400 x 400 x 20) Input video (200 x 200 x 20) Spatiotemporal classic kernel regression (400 x 400 x 40) AIMS regression (400 x 400 x 40) Milanfar et al. EE Dept, UCSC
Outline • Background and Motivation • Classic Kernel Regression • Data-Adaptive Regression • Regression in 3-D • Adaptive Implicit-Motion Steering Kernel (AIMS) • Motion-Aligned Steering Kernel (MASK) • Conclusions Milanfar et al. EE Dept, UCSC
Motion-Aligned Steering Kernel Accurate, explicit motion estimates “2-D motion-steered” (spatial) kernel 1-D (temporal) kernel Milanfar et al. EE Dept, UCSC • Motion is explicitly estimated to subpixel accuracy • Kernel weights are aligned with the local motion vectors using warping/shearing • The warped kernel acts directly on the data • Handles large and/or complex motions
Intuition Behind the MASK 2-D “motion-steered” (spatial) kernel 1-D (temporal) kernel Milanfar et al. EE Dept, UCSC
The Shapes of MASK Slices of MASK kernels Local data Milanfar et al. EE Dept, UCSC Spreads along spatial orientations and local motion vectors.
A Comparison of AIMS and MASK Input video (200 x 200 x 20) AIMS (400 x 400 x 40) MASK (400 x 400 x 40) Milanfar et al. EE Dept, UCSC Spin Calendar video
A Comparison of AIMS and MASK Input video (QCIF: 144 x 176 x 28) AIMS + BTV deblurring (CIF: 288 x 352 x 28) MASK + BTV deblurring (CIF: 288 x 352 x 28) Milanfar et al. EE Dept, UCSC Foreman video
Conclusions Milanfar et al. EE Dept, UCSC • We extended the 2-D kernel regression framework to 3-D. • Illustrated 2 distinct approaches • AIMS: Avoids subpixel motion estimation, needs comp. for large motions • MASK: Needs subpixel motion estimation, deals directly with large motions • Which is better? Depends on the application. • The overall 3-D SKR framework is simultaneously well-suited for spatial, temporal, and spatiotemporal • upscaling, denoising, blocking artifact removal, superresolution • not only in video but in general 3-D data sets. • Future work • Integration of deblurring directly in the 3-D framework • Computational complexity