EE 7700

1. EE 7700 Demosaicking Problem in Digital Cameras

2. Lens Scene Beam-splitters Spectral filters Sensors Multi-Chip Digital Camera • To produce a color image, at least three spectral components are needed at each pixel. • One approach is to use beam-splitters and multiple chips.

3. Single-Chip Digital Camera • Multi-chip approach is expensive. Precise chip alignment is required. • The alternative is to use a color filter array. Lens Sensors Color filter array Scene

4. Single-Chip Digital Camera • The missing color samples must be estimated to produce the full color image. • Since a mosaic of samples are available, this estimation (interpolation) process is called demosaicking.

5. Single-Chip Digital Camera • Images suffer from color artifacts when the samples are not estimated correctly. Original image Bilinearly interpolated from CFA-filtered samples

6. Demosaicking Approaches • Non-Adaptive Single-Channel Interpolation: Interpolate each color channel separately using a standard technique, such as nearest-neighbor interpolation, bilinear interpolation, etc. • Edge-Directed Interpolation: Estimate potential edges, avoid interpolating across the edges. • Edge-directed interpolation • Calculate horizontal gradient ΔH = |G1 – G2| • Calculate vertical gradient ΔV = |G3 – G4| • If ΔH > ΔV, • Gx = (G3 + G4)/2 • Else if ΔH < ΔV, • Gx = (G1 + G2)/2 • Else • Gx = (G1 + G2 + G3 + G4)/4 3 1 x 2 4

7. Demosaicking Approaches • Edge-Directed Interpolation: Based on the assumption that color channels have similar texture, various edge detectors can be used. • Edge-directed interpolation • Calculate horizontal gradient ΔH = | (R3 + R7)/2 – R5 | • Calculate vertical gradient ΔV = | (R1 + R9)/2 – R5 | • If ΔH > ΔV, • G5 = (G2 + G8)/2 • Else if ΔH < ΔV, • G5 = (G4 + G6)/2 • Else • G5 = (G2 + G8 + G4 + G6)/4 1 2 4 3 5 6 7 8 9

8. Demosaicking Approaches • Constant-Hue-Based Interpolation: Hue does not change abruptly within a small neighborhood. • Interpolate green channel first. • Interpolate hue (defined as either color differences or color ratios). • Estimate the missing (red/blue) from the interpolated hue. Interpolate Interpolated Red Red Interpolate Green

9. Demosaicking Approaches • Edge-Directed Interpolation of Hue: It is a combination of edge-directed interpolation and constant-hue-based interpolation. Hue is interpolated as in constant-hue-based interpolation approach, but this time, hue is interpolated based on the edge directions (as in the edge-directed interpolation algorithm).

10. Demosaicking Approaches • Using Laplacian For Enhancement: Use the second-order gradients of red/blue channels to enhance green channel. • Calculate horizontal gradient ΔH = |G4 – G6| + |R5 – R3 + R5 – R7| • Calculate vertical gradient ΔV = |G2 – G8| + |R5 – R1 + R5 – R9| • If ΔH > ΔV, • G5 = (G2 + G8)/2 + (R5 – R1 + R5 – R9)/4 • Else if ΔH < ΔV, • G5 = (G4 + G6)/2 + (R5 – R3 + R5 – R7)/4 • Else • G5 = (G2 + G8 + G4 + G6)/4 + (R5 – R1 + R5 – R9 + R5 – R3 + R5 – R7)/8 1 2 4 3 5 6 7 8 9

11. Aliasing Frequency spectrum of an image: After CFA sampling: Green channel Red/Blue channel

12. Demosaicking Approach • Alias Cancelling: Based on the assumption that red, green, and blue channels have similar frequency components, the high-frequency components of red and blue channels are replaced by the high-frequency components of green channel. Red/Blue channel

13. Experiment HL HL HL Full Red/Green/Blue channels LL LL LL Subband decomposition HH LH LH LH CFA Sampling Interpolate HL HL HL LL LL LL Subband decomposition HH LH LH LH

14. Constraint Sets • Detail Constraint Set: Detail subbands of the red and blue channels must be similar to the detail subbands of the green channel. HL HL HH HH LH LH

15. Constraint Sets • Observation Constraint Set: Interpolated channels must be consistent with the observed data. Sensors CFA

16. Projection Operations • Projection onto the Detail Constraint Set: • Decompose the color channels. • Update the detail subbands of red and blue channels. HL HH LH • Apply synthesis filters to reconstruct back the channels.

17. Projection Operations • Projection onto theObservation Constraint Set: • Insert the observed data to their corresponding positions. Sensors CFA

18. Alternating Projections Algorithm Samples of color channels Initial interpolation Projection onto the detail constraint set Projection onto the observation constraint set Insert the observed data Update Iteration

19. Results Original Hibbard 1995 Laroche and Prescott 1994 Hamilton and Adams 1997 Kimmel 1999 Gunturk 2002

20. Results Laroche and Prescott 1994 Hibbard 1995 Original Hamilton and Adams 1997 Gunturk 2002 Kimmel 1999

21. Previous Methods [Gunturk02] Gunturk et al, “Demosaicking: Color Filter Array Interpolation in Single-Chip Digital Cameras,” to appear in IEEE Signal Processing Magazine.

22. References • [Gunturk02] Gunturk et al, “Color Plane Interpolation Using Alternating Projections,” IEEE Trans. Image Processing, 2002. • [Hibbard 1995] R. H. Hibbard, “Apparatus and method for adaptively interpolating a full color image utilizing luminance gradients,” U.S. Patent 5,382,976, January, 1995. • [Laroche and Prescott 1994] C. A. Laroche and M. A. Prescott, “Apparatus and method for adaptively interpolating a full color image utilizing chrominance gradients,” U.S. Patent 5,373,322, December, 1994. • [Hamilton and Adams 1997] J. F. Hamilton Jr. and J. E. Adams, “Adaptive color plane interpolation in single sensor color electronic camera,” U.S. Patent 5,629,734, May, 1997. • [Kimmel 1999] R. Kimmel, “Demosaicing: Image reconstruction from CCD samples,” IEEE Trans. Image Processing, vol. 8, pp. 1221-1228, 1999.