Frequency domain methods for demosaicking of Bayer sampled color images Eric Dubois

1. Frequency domain methods for demosaicking of Bayer sampled color imagesEric Dubois

2. Problem Statement • Problem: Most digital color cameras capture only one color component at each spatial location. The remaining components must be reconstructed by interpolation from the captured samples. Cameras provide hardware or software to do this, but the quality may be inadequate. • Objective: Develop new algorithms to interpolate each color plane (called demosaicking) with better quality reconstruction, and with minimal computational complexity. Frequency-domain Bayer demosaicking

3. Retinal Cone Mosaic The human visual system must solve a similar problem! Frequency-domain Bayer demosaicking

4. Construction of color image from color planes + Frequency-domain Bayer demosaicking

5. Lighthouse original

6. Lighthouse red original

7. Lighthouse green original

8. Lighthouse blue original

9. Formation of Color planes Frequency-domain Bayer demosaicking

10. Lighthouse red subsampled

11. Lighthouse green subsampled

12. Lighthouse blue subsampled

13. Lighthouse Bayer CFA image

14. Color plane interpolation Green channel: bilinear interpolation GA GL GR GI GB Frequency-domain Bayer demosaicking

15. Color plane interpolation Red channel: bilinear interpolation RNE RNW RC RSE RS RSW Frequency-domain Bayer demosaicking

16. Lighthouse red interpolated

17. Lighthouse green interpolated

18. Lighthouse blue interpolated

19. Lighthouse Interpolated color image

20. Lighthouse original

21. Can we do better? • Color planes have severe aliasing. Better interpolation of the individual planes has little effect. Frequency-domain Bayer demosaicking

22. Lighthouse red interpolated with bilinear interpolator

23. Lighthouse red interpolated with bicubic interpolator

24. Can we do better? • Color planes have severe aliasing. Better interpolation of the individual planes has little effect. • We could optically prefilter the image (blur it) so that aliasing is less severe. Frequency-domain Bayer demosaicking

25. Lighthouse red interpolated with bilinear interpolator

26. Lighthouse prefiltered red interpolated with bilinear interpolator

27. Lighthouse Interpolated color image

28. Lighthouse Prefiltered & Interpolated color image

29. Lighthouse original

30. Can we do better? • Color planes have severe aliasing. Better interpolation of the individual planes has little effect. • We could optically prefilter the image (blur it) so that aliasing is less severe. • We can process the three color planes together to gather details from all three components. Frequency-domain Bayer demosaicking

31. Can we do better? • There have been numerous papers and patents describing different algorithms to interpolate the color planes – they all work on the three planes together, exploiting the correlation between the three components. • Gunturk et al. published an extensive survey in March 2005. The best methods were the projection on convex sets (POCS) algorithm (lowest MSE) and the adaptive homogeneity directed (AHD) algorithm (best subjective quality). • We present here a novel frequency-domain algorithm. Frequency-domain Bayer demosaicking

32. Spatial multiplexing model subsampling multiplexing Frequency-domain Bayer demosaicking

33. Spatial multiplexing model Frequency-domain Bayer demosaicking

34. Frequency-domain multiplexing model Re-arranging the spatial multiplexing expression Frequency-domain Bayer demosaicking

35. Frequency-domain multiplexing model David Alleysson, EPFL Frequency-domain Bayer demosaicking

36. Luma and chrominance components Frequency-domain Bayer demosaicking

37. Luma and chrominance components Luma fL Chroma_1 fC1 Chroma_2 fC2 Frequency-domain Bayer demosaicking

38. Lighthouse Bilinearly Interpolated color image

39. Frequency-domain demosaicking algorithm • Extract modulated C1 using a band-pass filter at (0.5,0.5) and demodulate to baseband • Extract modulated C2 using band-pass filters at (0.5,0.0) and (0.0, 0.5), demodulate to baseband, and combine in some suitable fashion (the key) • Subtract modulated C1 and remodulated C2 from the CFA to get the estimated luma component L. • Matrix the L, C1 and C2 components to get the RGB representation. Frequency-domain Bayer demosaicking

40. Spectrum of CFA signal b a Frequency-domain Bayer demosaicking

41. Using C2b only Using C2a only

42. Demosaicking using C2a only or C2b only -- details From C2b only Original From C2a only Frequency-domain Bayer demosaicking

43. fC2am fC2a  h2a fC2 fC2 (-1)n1 fR combine fC2bm fC2b  h2b fCFA (-1)n1-(-1)n2 -(-1)n2  matrix - fG + + fL - fB  h1 fC1m fC1 fC1 (-1)n1+n2 Demosaicking Block Diagram Frequency-domain Bayer demosaicking

44. Spectrum of CFA signal b a Frequency-domain Bayer demosaicking

45. Design Issues • How to choose the filters h1, h2a and h2b • Frequency domain design methods • Least-squares design methods • Size of the filters • How to combine the two estimates and • Choice of features to guide weighting • The two above issues may be inter-related. Frequency-domain Bayer demosaicking

46. Filter design • Gaussian filters (Alleysson) • Window design or minimax design • Define ideal response, with pass, stop and transition bands • Approximate using the window design method • Refine using minimax or least pth optimization • Can design low-pass filters and modulate to the center frequency Frequency-domain Bayer demosaicking

47. Filter specification u v val 0.000 0.00 1.0 0.110 0.00 1.0 0.110 0.02 1.0 0.000 0.10 1.0 0.030 0.10 1.0 0.070 0.06 1.0 0.338 0.00 0.0 0.338 0.05 0.0 0.050 0.36 0.0 0.000 0.36 0.0 0.184 .205 0.0 0.500 0.00 0.0 0.000 0.50 0.0 0.500 0.50 0.0 Frequency-domain Bayer demosaicking

48. Ideal response – perspective view Frequency-domain Bayer demosaicking

49. Ideal response – contour plot Frequency-domain Bayer demosaicking

50. Window design – perspective view Frequency-domain Bayer demosaicking