Advisor: Dr. Hagit Hel-Or By: Ido Yerushalmy December 2006

1. Image Processing Seminar Still and Video Image Forgery Detection Using JPEG Characteristics Advisor: Dr. Hagit Hel-Or By: Ido Yerushalmy December 2006

2. Using JPEG Characteristics Previous Methods In previous lectures we learnt about ways to detect forgeries, based upon the data in the image (statistical tools or watermarking). Today I will present methods that take advantage of the format the image is saved in. I will explore the JPEG format. JPEG Format Quant. Table MPEG Format GOP correlation Double Quantization Summary

3. Using JPEG Characteristics • JPEG – A Lossy Format • JPEG is characterized by loss of information at every compression. This is the result of 2 factors: • Some data is more accurately saved (e.g. the Y channel), some is sub-sampled (e.g. the Cb channel) • Each Channel is transformed into its frequency domain (DCT) and then undergoes a quantization process JPEG Format Quant. Table MPEG Format GOP correlation Double Quantization Summary

4. Using JPEG Characteristics Compression Example JPEG Format • The image is converted to YCrCb color map and sub-sampled as desired. From now on each channel is worked on separately • The image is dissected into 8x8 patches, like this one: Quant. Table MPEG Format GOP correlation Double Quantization Summary • The patch’s DCT is taken (frequency domain):

5. Using JPEG Characteristics Compression Example – Cont. JPEG Format The DCT bases: Quant. Table MPEG Format GOP correlation Double Quantization Summary • A quantization matrix is used to eliminate high frequencies:

6. Using JPEG Characteristics Compression Example – Cont. JPEG Format • Lets have a look at the DC quantization: Quant. Table MPEG Format GOP correlation • The compressed patch in this case will be: Double Quantization The patch is entropy encoded, so the zeros are not actually written Summary Note: Most of the noise in the image is added by the quantization process

7. Using JPEG Characteristics File Format JPEG Format Quant. Table MPEG Format GOP correlation Double Quantization Summary Tables from: http://www.codeproject.com/bitmap/iptc.asp?print=true

8. Using JPEG Characteristics Uniqueness of the Quantization Table JPEG Format Quant. Table • As shown, every JPEG image has a quantization table attached to it, which defines the de-coding algorithm. • Theoretically there are: • 64 quantization values in every table • 3 tables per image (one per color channel) • Quantization values usually range from 1 to 25 • Multiply all of the above and we get: different tables. Quant. Table MPEG Format GOP correlation Double Quantization Summary

9. Using JPEG Characteristics Uniqueness of the Quantization Table JPEG Format Quant. Table However, In practice there are much less different tables, since certain rules are applied to all tables. For example: the DC quantization value will always be low None the less, the quantization table can give us a sense of which compression software was used to compress and save this image last. Quant. Table MPEG Format GOP correlation Double Quantization Summary

10. Using JPEG Characteristics How Unique is the Quantization Table? JPEG Format Quant. Table The article Digital Image Ballistics from JPEG Quantization by H. Farid has identified the quantization tables for more than 200 cameras and software. It found that on the average, every such table is common to 1.4 cameras / software. Using this method we may get a rough idea if the image was edited and re-compressed. Quant. Table MPEG Format GOP correlation Double Quantization Summary

11. Using JPEG Characteristics Many Drawbacks JPEG Format Quant. Table • Drawbacks: • The results are only crude, since the uniqueness of the tables is not promised • Much work should be done to identify every possible device’s quantization table. Once more devices will be added the 1st drawback will get even worst • This method can be easily tackled by externally editing the quantization table. Since most tables are similar, chances are there will be no visual difference Quant. Table MPEG Format GOP correlation Double Quantization Summary

12. Using JPEG Characteristics Moving On To Video JPEG Format Quant. Table JPEG compression is heavily used not only for still images, but also for video compressions such as MPEG. Using the same method there may assist in a similar manner Quant. Table MPEG Format GOP correlation Double Quantization Summary

13. Using JPEG Characteristics Group Of Pictures (GOP) MPEG Frame Types JPEG Format • In an MPEG encoded video there are three types of frames: • Intra (I) frames • Predictive (P) frames • Bi-Directional (B) frames • The frames are typically sequenced as follows: Quant. Table MPEG Format GOP correlation Double Quantization Summary Time B B I P B B P B B P I B

14. Using JPEG Characteristics MPEG – I frames JPEG Format Quant. Table I-frames are typically the highest quality frames of a video sequence but afford the least amount of compression. I frames are encoded using a fairly standard JPEG compression scheme. MPEG Format GOP correlation Double Quantization Summary

15. Using JPEG Characteristics MPEG – P frames JPEG Format • P-frames encode the motion between frames. It takes advantage of the small temporal difference between frames. • An estimation of the motion between the previous frame and the current one is done Quant. Table MPEG Format GOP correlation Double Quantization Summary

16. Using JPEG Characteristics MPEG – P frames JPEG Format Quant. Table • The estimated motion is described by a motion vector encoded into the P frame. This results in a “predicted image” MPEG Format GOP correlation Double Quantization Summary

17. Using JPEG Characteristics MPEG – P frames JPEG Format Quant. Table • The error between the estimation and the actual current frame is also encoded into the P frame MPEG Format GOP correlation Double Quantization Summary

18. Using JPEG Characteristics MPEG – B frames JPEG Format Quant. Table Similar to a P-frame, a B-frame is encoded using motion compensation. Unlike a P-frame, however, a B-frame employs a past, future, or both of its neighboring I- or P frames for motion estimation. MPEG Format GOP correlation Double Quantization Summary

19. Using JPEG Characteristics B B I P B B P B B P I B Group Of Pictures (GOP) Correlation within a GOP JPEG Format Quant. Table Lets recall the MPEG Group Of Pictures (GOP) MPEG Format GOP correlation Double Quantization Summary Each frame in a GOP is correlated to the I-frame at its beginning. This is, in part, because the artifacts generated during the encoding of the I-frame propagate through the P and B frames.

20. Using JPEG Characteristics Detecting Temporal Changes JPEG Format Consider the following video with 5 frames: Quant. Table MPEG Format GOP correlation Double Quantization I 1 P2 P3 P4 I 5 Summary Group Of Pictures (GOP) The motion error for P2 will be: M = motion compensation

21. Using JPEG Characteristics P4 I 5 P3 P4 The motion error for the newly encoded P4 will be: M = motion compensation Detecting Temporal Changes JPEG Format A forgery has been performed, 1 frame is deleted: Quant. Table MPEG Format GOP correlation Double Quantization I 1 P2 P3 Summary Group Of Pictures (GOP)

22. Using JPEG Characteristics Detecting Temporal Changes JPEG Format The motion error m2 was: Quant. Table MPEG Format GOP correlation The additive noise of frames 1 and 2 was correlated and so was somewhat canceled in m2. In contrast, m4 was: Double Quantization Summary And the frame noises of 3’ and 4’, were not correlated, since they were produced from two different GOPs at the original video.

23. Using JPEG Characteristics Detecting Temporal Changes JPEG Format Quant. Table MPEG Format GOP correlation Double Quantization This causes the error encoding in m4 to be relatively bigger than that in m2. Summary

24. Using JPEG Characteristics The mean motion error over time The magnitude of its Fourier transform Changes Appear in a Periodic Manner JPEG Format This larger error will be encoded in a periodic manner from the point of deletion and on, where the period will be the number of frames in a GOP. For example: A video with GOP = 12 frames, where 6 frames have been removed. Quant. Table MPEG Format GOP correlation Double Quantization Summary

25. Using JPEG Characteristics Drawbacks JPEG Format This conclusion will fail when a whole GOP will be erased from the original video. Quant. Table MPEG Format GOP correlation Double Quantization Summary

26. Using JPEG Characteristics Moving to Static Artifacts JPEG Format Up to now we have discussed temporal artifacts caused by forgery. These artifacts were present in P-frames. Quant. Table MPEG Format GOP correlation At this point will move to look for static artifacts, which will be present in the I-frame. Lets recall that the I-frame is JPEG compressed, meaning (as discussed before) that quantization is used for the DCT coefficients. Double Quantization Summary

27. Using JPEG Characteristics De-Quantization is: Artifacts Caused by Double Quantization JPEG Format Quant. Table Quantization is in fact a point-wise operation: MPEG Format Where a is the quantization step GOP correlation Double Quantization Summary

28. Using JPEG Characteristics So, a double quantization, using 2 different quantization steps (a and b) is: Artifacts Caused by Double Quantization JPEG Format Quant. Table MPEG Format GOP correlation Double Quantization Summary • Double quantization can be described as a sequence of three operations: • quantization with step b • de-quantization with step b • quantization with step a.

29. Using JPEG Characteristics What happens to the histogram bins ? Different Quantization Steps JPEG Format Quant. Table • Lets consider the two cases of a and b: • a > b (the 2nd quantization uses a bigger step than the 1st) MPEG Format GOP correlation Double Quantization Summary

30. Using JPEG Characteristics a = 3 b = 2 The even bins receive samples from four original histogram bins, while the odd bins receive samples from only two. Different Quantization Steps JPEG Format Quant. Table • a > b (the 2nd quantization uses a bigger step than the 1st) MPEG Format GOP correlation Double Quantization Summary

31. Using JPEG Characteristics a = 3 b = 2 The first quantization places the samples of the original image into 42 bins, while the second quantization redistributes them into 64 bins. Different Quantization Steps JPEG Format Quant. Table • a < b (the 2nd quantization uses a smaller step than the 1st) MPEG Format GOP correlation Double Quantization Summary

32. Using JPEG Characteristics histogram Of DCT coefficients Fourier transform magnitude of the above histogram Forgery Detection Application JPEG Format Quant. Table If a video is de-coded and re-compressed using a different MPEG encoder, the result may be a double-quantization of the I-frame DCT coefficients. For example: A double quantization with the 1st step = 8 and 2nd step = 3 MPEG Format GOP correlation Double Quantization Summary

33. Using JPEG Characteristics Drawbacks JPEG Format Quant. Table • Some MPEG encoders use different quantization levels for different blocks in the same frame. In such a case even if the same encoder was used again to save the video, double quantization will be detected • As with the still images, the quantization table may be manually edited MPEG Format GOP correlation Double Quantization Summary

34. Using JPEG Characteristics Summary JPEG Format Quant. Table MPEG Format • JPEG compression format is a lossy format which introduces noise due to quantization of its DCT coefficients • The Quantization table is relatively unique, so it can narrow down the group of possible devices that last compressed a given image GOP correlation Double Quantization Summary

35. Using JPEG Characteristics Summary JPEG Format Quant. Table MPEG Format • MPEG video files contain GOPs that include frames which are correlated to each other. Partially deleting a GOP results in intensified error which is encoded into the P-frames • Different MPEG encoders may use different quantization tables for the I-frames. This introduces artifacts of double-quantization of the DCT coefficients GOP correlation Double Quantization Summary

36. Using JPEG Characteristics References JPEG Format Quant. Table • Exposing Digital Forgeries in Video by Detecting Double MPEG Compression W. Wang and H. Farid ACM Multimedia and Security Workshop, Geneva, Switzerland, 2006 • Digital Image Ballistics from JPEG Quantization H. Farid Technical Report, TR2006-583, Dartmouth College, Computer Science • http://www.codeproject.com/bitmap/iptc.asp?print=true • JPEG format, from Wikipedia – the free encyclopedia MPEG Format GOP correlation Double Quantization Summary