Examining Photo Response Non-Uniformity for the Comparison of Cameras

1. Examining Photo Response Non-Uniformity for the Comparison of Cameras Zeno Geradts PhD / Maarten van der Mark BS / Wiger van Houten MS Partially funded by the European Commission within the project FIDIS www.fidis.net Nederlands Forensisch Instituut Laan van Ypenburg 6, 2497 GB Den Haag

2. Overview • Introduction • Noise and PRNU sources • Comparison of denoising methods • Application to YouTube videos • Performance • Statistics (Bayesian) • Conclusions

3. FIDIS • Network of Excellence within the European Union, with 20 partners ranging from Universities, Privacy protection agencies and companies such as IBM and Microsoft • www.fidis.net

4. Introduction • Each image sensor has a unique `fingerprint’, the PRNU pattern, that is detectable in all images the sensor produces • This `fingerprint’ is used to establish the image origin

5. Noise and PRNU sources • Different types of noise: • 2 Categories: • Random noise • Temporally variable • Statistical distributions • Can be reduced by averaging multiple frames • Pattern noise (PRNU and FPN) • Not temporally variable but systematic (spatial variable) • How do image sensors work? • CCD image sensors • CMOS Active Pixel Sensors

6. CCD image sensors • Basic building block (gate) • Silicon substrate • Apply positive voltage to the gate • Electron-hole pairs generated in depletion region are confined • Read out • Charge converted to voltage at the sense node

7. CMOS APS image sensors • Cpd: photodiode capacitor • M1: reset transistor • M2: source follower • M3: row select transistor • M4: bias transistor

8. Noise and PRNU sources (2) • The characteristic pattern extracted from the images contains 2 systematic contributions: • Multiplicative PRNU, depends on illumination • Fixed Pattern Noise (FPN), does not depend on illumination • Also: common components in the extracted pattern (e.g. CFA interpolation)

9. Noise and PRNU sources (3) • FPN (general): • Crystal defects in the silicon lattice introduced during growth • Impurities • The size of the detector/potential well • Contamination during fabrication • Non-uniform oxide/gate thickness • In CMOS: additional sources (each transistor)

10. Noise and PRNU sources (4) • PRNU • The depth of the detector/potential well • Larger active area: more incident photons • Non-uniform oxide layer: results in non-uniform potential wells • Deeper potential well: more photons absorbed (wavelength dependent) • In CMOS: additional sources

11. Image denoising – pattern extraction • Denoising algorithms do not discriminate between noise and image details. There is always a tradeoff. • PRNU is a nonperiodic discontinuous signal • Pattern = Image – F(Image) , F is the denoising filter • Correlate the patterns from `questioned’ videos with reference patterns

12. Old method • Gaussian smoothing filter • Advantages: • Simple • Very fast • Disadvantages: • Distorts edge integrity • Image residue left behind in the pattern

13. New method • Wavelet based denoising filter [Lukas et al]* • Disadvantages • Slower • Diadic images only • Advantages • Preserves edges (edge detection) • Spatial adaptive • Works really well *Digital Camera Identification from Sensor Pattern Noise (2005) – Lukas, Fridrich, Goljan

14. YouTube (1) • Accepts large amount of input formats: Xvid, DivX, WMV, 3GP, … • Downloadable as H.264/MPEG-4 AVC with e.g. keepvid.com • Maximum resolution (H.264): 480x360 • Aspect ratio generally does not change

15. YouTube (2) • When the resolution of the video uploaded to YouTube is smaller than 480x360, generally no resolution change occurs (exceptions) • When the resolution of the video exceeds 480x360, the resolution is changed to 480x360 or lower (depending on the aspect ratio) e.g. 640x480  480x360 640x360  480x270

16. In practice – natural video • Record 30 seconds of natural video, recorded with XVID/WMV • Upload this video to YouTube • Download the video with keepvid.com • Extract individual frames • Estimate noise pattern

17. In practice – flatfield video • If natural video downloaded has a resolution lower than 480x360, record the flatfield video (RAW) with the same resolution, use these frames • Otherwise (e.g. 640x480), record flatfield video in native resolution, and upload (download) them to (from) YouTube • Alternative: resize flatfield video to match the dimension of the YouTube video (e.g. 640x480  480x360)

18. Performance (1) • Works really well when natural video is recorded in native resolution, both with XVID and WMV9 (Messenger) with large range of codec settings, even for shorter samples • Works reasonably well when aspect ratio has been changed • Problematic with very low resolution (Vodafone: 176x144) • Does not always work when the video was binned during recording (e.g. native resolution of 640x480, recorded in 320x240x)

19. Performance (2) – Creative Live! IM Webcam • Problem: need to set two parameters for extracting the patterns • These parameters depend on a large amount of variables: • Content • Compression • Codec • Resolution • etc. • Only from empirical data; impossible in casework

20. Comparison – Creative Live! IM, 640x480 native, recorded in 352x288 • Old method (σ=0.6, threshold 5, 4x4 averaging): New Method (σ_n=4.5, σ_f=4.5):

21. Logitech Communicate STX (1) • Logitech : 640x480 (native), XVID q4 • Logitech : 640x480 (native), variable XVID quality

22. Logitech Communicate STX (2) • Logitech : 320x240, XVID q4 • Logitech : 320x240, variable XVID quality

23. Test yourself sourceforge.net search PRNU

24. Likelihood ratio Comparing patterns Value of the evidence:

25. Likelihood ratio In practice: No overlapping histograms We do not know the origin of the questioned image

26. Likelihood ratio To artificially find the pdf of Hd at Hp we use an estimator Example: add-constant estimator Suppose you look at the colours of passing cars: 1 blue car, 1 red car, and 1 green car Q: what is the chance the next car is blue? A: 1/3? Or smaller? A: (1+1)/7 for blue; 1/7 for a new colour

27. Likelihood ratio More complicated estimators available, but the essence is the same: reserve a probability for unseen species, based upon the data we encountered so far Simple Good-Turing approach

28. Likelihood ratio SGT: problems: Car colours (green, blue, red, etc) is discrete, but some colours exist that can be seen as both (blue or green?) We are considering correlation values, continuous data. We have to divide the values into `species’ ourselves

29. Likelihood ratio, recap

30. Conclusion (1) We understand the origins of the pattern noise that is used to perform device identification We understand which random noise sources are present A likelihood ratio can be found, but there are some unanswered questions before this approach can be used

31. Conclusions (2) • It is possible to identify the source video camera based on the PRNU pattern, even after the video has been uploaded to YouTube • The new method, although computationally more intensive, performs much better

32. Conclusions (3) • Limitations: (very) low resolution videos, changed aspect ratios, ‘subsampled’ recordings • No way to find the ideal parameter directly from the video or frames: this is a problem when the abovementioned limitations are met

33. Future research • Using the methods on databases of child pornography to link cameras • Improvement of algorithms • Validation in casework

34. Questions ? • zeno@holmes.nl