Modelling, calibration and rendition of colour logarithmic CMOS image sensors

1. Modelling, calibration and rendition of colour logarithmic CMOS image sensors Dileepan Joseph and Steve Collins Department of Engineering Science University of Oxford

2. Outline • Logarithmic CMOS image sensors • Modelling sensor response • Image sensor calibration • Fixed pattern noise • Sensation of colour • Rendition of images • CIE Lab (perceptual error) • IEC sRGB (standard display) • Summary and future work IMTC 2002

3. Logarithmic CMOS image sensors • CMOS displacing CCD because of integration of signal processing and economies of scale • Logarithmic sensors offer high dynamic range and high frame rate • Linear sensors offer low fixed pattern noise and good colour rendition • Example images taken from IMS Chips website Linear CCD sensor Logarithmic sensor Logarithmic sensor IMTC 2002

4. Modelling sensor response • Since Ik = ∫ fk(λ) s(λ) dλ • For photocurrent Ik, spectral response fk(λ) and light stimulus s(λ) at a pixel, where k = R, G or B • And fk(λ) = gL(λ) gk(λ) gP(λ) • For spectral responses of lens gL(λ), colour filter gk(λ) and photodiode gP(λ) • Approximating a linear combination of three CIE XYZ basis functions • Then Ik = dk• x • For mask coefficients dk and tricolour vector x, i.e. s(λ) in CIE XYZ space • Ideally, y = a + b ln (c + Ik) + ε • For digital response y of pixel with offset a, gain b, bias c and error ε • Pixel-to-pixel variation of a, b or c causes fixed pattern noise (FPN) IMTC 2002

5. FPN calibration • Three types of FPN of interest: • Offset variation • Offset and gain variation • Offset, gain and bias variation • Partition pixels by colour filter to permit FPN calibration of three monochromatic sensors • Take images of uniform stimuli under different illuminances • Calibrate each pixel’s response to average response of all pixels by least squares estimation of varying model parameters • Fuga 15RGB sensor exhibits offset, gain and bias variation IMTC 2002

6. Colour calibration • Take and segment images of a standard chart, having patches of known CIE XYZ colour, under different illuminances • Calibrate pixel responses to colour by estimating non-varying model parameters (e.g. mask dk), using estimates of varying parameters • Ideal model fails for Fuga 15RGB because absolute relationship between y and Ik invalid (strong inversion component?) • Empirical model y = a + b ln (c + (α + dk • x)β) worked well, with no change to relative responses of pixels or FPN calibration IMTC 2002

7. Image rendition (CIE Lab) • Images of a Macbeth Colour Chart, taken by the Fuga 15RGB, were rendered into CIE Lab space with the calibrated empirical model • The perceptual error increases in dim lighting as the bias term c dominates the photocurrent Ik • Excluding the dimmest image (i.e. 5 lux), the error equals 12 over a 60 dB dynamic range for offset, gain and bias variation • Images in Digital Photographer show that conventional (linear) digital cameras have an error of 15 over a 30 dB dynamic range IMTC 2002

8. Image rendition (IEC sRGB) • A Fuga 15RGB image of the Macbeth Chart, taken in 11 lux of illuminance, was rendered into IEC sRGB space with the calibrated empirical model • Results for offset variation (top-left), offset and gain variation (top-right), offset, gain and bias variation (bottom-left) and true colours (bottom-right) are shown • Two types of residual deviation for the rendered patches are visible: • Fixed pattern noise • Colour desaturation IMTC 2002

9. Summary and future work • Logarithmic image sensors offer high dynamic range and frame rate • Combine theories of colour linear sensors and monochromatic logarithmic sensors to model colour logarithmic sensors • Calibrate FPN, using images of uniform stimuli, by relative estimation of model parameters that vary from pixel to pixel • Calibrate colour, using images of a colour chart, by absolute estimation of model parameters that do not vary • Fuga 15RGB results expose limitations of ideal model in absolute estimation but reveal empirical model that works well • Macbeth Chart results show colour rendition with calibrated Fuga 15RGB competes with conventional digital cameras • Seek to minimise bias variation, so simple FPN models suffice, and bias magnitude, to improve colour rendition in dim lighting IMTC 2002

10. Acknowledgements • The authors gratefully acknowledge the support of the Natural Sciences and Engineering Research Council (Canada) and the Engineering and Physical Sciences Research Council (UK) IMTC 2002