300 likes | 446 Views
Brendan Duncan. Motion-corrected image denoising for digital photography. Noise in digital cameras. Noise in digital cameras:. P – # photons Q e – quantum efficiency of sensor t – exposure time D – dark current noise N r – read noise. Averaging multiple exposures to reduce noise.
E N D
Brendan Duncan Motion-corrected image denoising for digital photography
Noise in digital cameras • Noise in digital cameras: • P – # photons • Qe – quantum efficiency of sensor • t – exposure time • D – dark current noise • Nr – read noise
Averaging multiple exposures to reduce noise • Averaging images effectively reduces exposure time, which increases the SNR • Averaging several short exposures is better than a single long exposure when the camera is handheld; a single exposure would result in blur • This requires aligning images prior to averaging
Combining images without motion blur or ghosting: Previous solutions • Removing moving objects • O. Gallo, et al. Artifact-free high dynamic range imaging. ICCP, 2009. • Different use case, we want to preserve any moving object present in the base image
Combining images without motion blur or ghosting: Previous solutions • Optical flow-based methods • B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. Image Understanding Workshop, 1981. • A. Ogale, et al. Motion segmentation using occlusions. Pattern Analysis and Machine Intelligence,2005. • Wong and Spetsakis. Motion segmentation and tracking. International Conference on Vision Interface, 2002. • These require a high framerate
Combining images without motion blur or ghosting: Previous solutions • Video denoising algorithms that extend existing image denoising algorithms • Seo and Milanfar. Video denoising using higher order optimal space-time adaptation. International Conference on Acoustics, Speech and Signal Processing, 2008. • Potter and Elad. Image denoising via learned dictionaries and sparse representation. Transactions on Image Processing, 2009. • These think of video as a single 3D volume • Cannot extend regions of like pixels in time dimension in regions with motion because of low framerate
Combining images without motion blur or ghosting: Previous solutions • Adaptive Spatio-Temporal Accumulation (ASTA) Filter • Bennett and McMillan. Video enhancement using per-pixel virtual exposures. SIGGRAPH, 2005. • Uses spatial filtering, which will cause loss in texture and detail
Proposed solution • Align both foreground and background objects using SIFT features and RANSAC instead of optical flow techniques • Use a weighted average that will reduce the contribution from noisy pixels and from unmatched pixels
Algorithm overview • SIFT feature detection • Calculate perspective fit using RANSAC • Apply perspective warp • Calculate RANSAC fit on outliers • Apply new perspective warp • Repeat from step 4 as needed • Calculate denoised estimate • Perform weighted average
SIFT feature detection • Detect SIFT features in each of the images • D. Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004. • The features of the base image are inserted into a k-d tree • J. Friedman, et al. An algorithm for finding best matches in logarithmic expected time. ACM Transactions on Mathematical Software, 1977. • Use best bin first search to find shared features across images
Calculate perspective fit using RANSAC • RANSAC randomly chooses a small subset of shared features to calculate a perspective warp that matches the features • Fischler and Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981. • It chooses the perspective warp with the lowest error across all features after a predetermined number of iterations
Perspective warp • Equation for perspective warp • RANSAC finds the least-squares solution for each random sample
Calculate RANSAC fit on outliers • Previous inliers are removed from the feature set, and alignment is performed on the previously unmatched features • If background objects were aligned in the first RANSAC fit, now foreground objects will be aligned
Repeat if needed • Continue calculating RANSAC fits if there are unmatched features • Perform a perspective warp after each RANSAC stage • This accounts for multiple differently moving objects in the scene
Goals for weighted average • To improve upon simple average, two goals • Assign negligible weight to unmatched areas • Reduce contribution from noisy pixels
Use bilateral filter to get estimate • Bilateral filter is a nonlinear filter that • Does not blur across edges • Can be used to denoise images • Therefore it roughly meets our goals, except that it performs spatial averaging, which reduces texture and details • It is used as an estimate to help calculate weighted average
Bilateral filter p, q – x,y pixel coordinates I – three channel image Sp– set of pixels surrounding Ip σs – spatial standard deviation – user defined σr – range standard deviation – user defined Wp– total weight for all pixels in Sp Tomasi and Manduchi. Bilateral filtering for gray and color images. International Conference on Computer Vision, 1998.
Proposed weighted average I0 – base image I1 … In – warped images R – result image • The bilateral filtered estimate is used to determine expected mean of normal distribution and does not contribute to average
Example of weights for an image black – low weight white – high weight
Close-up comparison Above – original base image Right – processed image
Parallax • Parallax across images makes a simple average impossible
Parallax Above – original base image Right – processed image
Contribution from noisy pixels is reduced Simple average of 3 images Weighted average
Texture is preserved Above – bilateral filter Right – result of algorithm (same σr)