Chang-Tsun Li Department of Computer Science University of Warwick UK

1. Image Clustering Based on Camera Fingerprints Chang-Tsun Li Department of Computer Science University of Warwick UK

2. Lens CFA Interpolation Post-Processing CFA Sensor Scene Digital Image Acquisition Process Photo CFA: Colour Filter Array mapping of CFA to sensor pixels Bayer CFA

3. Lens CFA Interpolation Post-Processing CFA Sensor Scene What is Camera Fingerprint • Lens aberration • Sensor pattern noise • Colour filter array (CFA) interpolation artefacts • Camera response function • Quantisation table of JPEG compression Photo

4. Camera Fingerprint for Multimedia Forensics • Multimedia Forensics: The use of “fingerprints” left in images by the imaging devices for • source device identification • source device linking • content integrity verification • image classification

5. What is Sensor Pattern Noise • Sensor Pattern Noise (SPN) is the noise left in the images by the sensors of digital imaging devices such as cameras, camcorders and scanners. • SPN is mainly caused by • manufacturing imperfection and • different sensitivity of pixels to light due to the inhomogeneity of silicon wafers. • Sensors made from the same silicon wafer possess unique SPN because of the non-uniform imperfection. • SPN can differentiate cameras of the same model.

6. “Traditional” SPN Extraction Method • Lukáš et al’s model for SPN extraction (IEEE TIFS 2006) • I is the original image • I’is the low-pass filtered version of I by the Weiner filter applied in the wavelet transform domain • n is the extracted SPN • SPN is the high-frequency component of the image.

7. Interference from Scene Details • Scene details, e.g., brick walls, tree leaves, or other kinds of textures, contribute to the high-frequency components of images. natural image SPN a clean SPN a contaminated SPN

8. SPN Enhancement at Warwick • C.-T. Li, "Source Camera Identification Using Enhanced Sensor Pattern Noise," IEEE Trans. on Information Forensics and Security, June 2010 • C.-T. Li and Y. Li, "Color-Decoupled Photo Response Non-Uniformity for Digital Image Forensics," IEEE Trans. on Circuits and Systems for Video Technology, 2012 • X. Lin and C.-T. Li, "Preprocessing Reference Sensor Pattern Noise via Spectrum Equalization," IEEE Trans. on Information Forensics and Security, 2016 . • X. Lin and C.-T. Li, "Enhancing Sensor Pattern Noise via Filtering Distortion Removal," IEEE Signal Processing Letter, accepted for publication in 2016

9. Image Classification/Clustering • Scenario: A forensic investigator • has a large set of images taken by an unknown number of unknown digital cameras and • wishes to cluster those images into a number of classes, each including the images acquired by the same camera. Each data point represents one image Each cluster present one unknown device

10. Challenges Facing Image Classification • The forensic investigator does not have the cameras that have taken the images to generate reference SNPsfor comparison. • No prior knowledge about the number and types of the imaging devicesare available. • With a large dataset, exhaustive fingerprint comparison is computationally prohibitive. • Given the shear number of images, analysing each image in its full size is computationally infeasible.

11. Image Classification – a MRF Approach • Step 1. Extract and enhance the fingerprint of each block cropped • from the images • Step 2. Establish a similarity matrix ρ for a Focus Set of M images • Step 3. Train the classifier based on the similarity matrix ρ. • For each fingerprint i, which is treated as a random variable • 3.1. Assign a unique random class label • 3.2. Calculate a reference similarity (i.e., a “soft” threshold) • 3.3. Establish a membership committee (neighbourhood) • 3.4. Update the class label iteratively based on the information • from the membership committee until there are no changes • of class labels to any SPN throughout a entire iteration • Step 4. Classify the rest of the dataset using the classifier

12. Establishing Similarity Matrix To establish an M × M similarity matrix ρ, the similarity between any two enhanced SPNs iandj in the Focus Set is calculated using

13.  inter-class similarity  intra-class similarity similarity Similarity Classifier Training • Each fingerprint (SPN) is treated as a random variable. • 3.1. Assign a unique random class label to each SPN • 3.2. Calculate a reference similarity r • Normally intra-class similarities > • inter-class similarities. • A similarity less than rindicates that the two images are taken by different devices, otherwise by the same device. μ1 rμ2

14. Classifier Training 3.3. Establish a membership committee For each SPN i , a membership committee Ciwith cSPN members from the focus set that are most similar to i is established. vvvvvv ×

15. Classifier Training 3.4. Update the class label iteratively according to p(fi |ρ(i, Ci ), Li) until there is no change of class label to any SPN in xconsecutive iterations • fi : class label of SPN i • C • ρ(i, Ci ) is the similarities between SPNi and the members of Ci, i.e., • p(fi |ρ(i, Ci ), Li): probability of assigning fi given the conditions • ri: reference similarity (“soft” threshold) of I

16.  inter-class similarity  intra-class similarity similarity Similarity Objective Function / Cost Function • The combination of the s(.) and ρ(.) says, • a penalty (i.e. positive value) will be incurred • if ρ(i,j) >rand a different label than fi is to be assigned to i or • if ρ(i,j)<rand the same label as fj is to be assigned to i • a reward (i.e. negative value) will be given • if ρ(i,j)<rand a different label than fi is to be assigned to i or • if ρ(i,j)>r and the same label as fjis to be assigned to i μ1 rμ2

17. Image Classification • The centroids of the image clusters provided by the classifier training process at the end of Step 3.4 are used to classify the images. • To classify an image x, we compare the similarity of its SPN to the centroid of each identified cluster and classify it to the class with its centroid closest to the image.  

18. Classifier Training - Simulation Initial label configure: Each pattern is assigned an unique label / colour Clustering in progress …..

19. Final Classification

20. Table 1. Classification error rate. c is the size of the membership committee. Experimental Results Misclassification rates: 1200 images taken by six cameras, each taking 200. Class identification stops when there are no class label changes throughout an iteration. c: the size of the membership committee M: the size of the focus set A misclassification rate in the range (1.2 ~ 1.6)is likely to be the best the system can achieve.

21. Conclusions • Multimedia forensics using “fingerprint” left in the images by the imaging devices has emerged as a new area of research in the last few years. • Sensor pattern noise (SPN) is one of the most promising types of fingerprint. • The “traditional” SPN extraction method is unable to cope with the interference of scene details. • The proposed classifier is feasible, but is unable to classify images without clean SPNs provided by the proposed SPN enhancer.