Multimedia Specification Design and Production

1. Multimedia Specification Design and Production 2012 / Semester 1 / L2 Lecturer: Dr. Nikos Gazepidis gazepidis@ist.edu.gr

2. Digital Audio, Image & Video • Digital Media • In computers, audio, image and video are stored as files just like other text files. • For images, these files can have an extension like: • BMP, JPG, GIF, TIFF, PNG, etc. • For audios, the file extensions include: • WAV, MP3, … • The videos files usually have extensions: • AVI, MOV, MKV

3. Digital Audio, Image & Video • An Example of Digital Image • Let’s open an image file is its “raw” format: * P6: (this is a ppm image) * Resolution: 512x512 * Depth: 255 (8bits per pixel in each channel)

4. Digital Audio, Image & Video An Example of Digital Image (cont.) Color depth or Bit depth is the number of bits used to indicate the color of a single pixel in a bitmapped image or video frame buffer. • 1-bit color (21 = 2 colors): monochrome • 2-bit color (22 = 4 colors): CGA, gray-scale • 3-bit color (23 = 8 colors): many early home computers with TV displays • 4-bit color (24 = 16 colors): least common denominator VGA • 5-bit color (25 = 32 colors) • 6-bit color (26 = 64 colors) • 8-bit color (28 = 256 colors): most early color Unix workstations, VGA at low-res • 12-bit color (212 = 4096 colors): some Silicon Graphics systems The HDMI 1.3 specification defines bit depths of 30 bits (1.073 billion colors), 36 bits (68.71 billion colors), and 48 bits (281.5 trillion colors) …& transparency

5. Digital Audio, Image & Video An Example of Digital Image (cont.) 1 bit (2 colors) 2 bits (4 colors) 4 bits (16 colors) 8 bits (256 colors) 24 bits (16,777,216 colors, "truecolor")

6. Digital Audio, Image & Video An Example of Digital Image (cont.) 8 bits 16 bits 24 bits

7. Digital Audio, Image & Video An Example of Digital Image (cont.) An image contains a header and a bunch of (integer) numbers.

8. Digital Audio, Image & Video Digital Media Capturing • To get a digital image, an audio or a video clip, we need some media • capturing device such as: • a digital camera or a scanner, • a digital audio recorder, • or a digital camcorder. • All these devices have to complete tasks: • Sampling: To convert a continuous media into discrete formats. • Digitization: To convert continuous samples into finite number of digital numbers. • There are probably some further compression process.

9. Digital Audio, Image & Video An Audio Signal

10. Digital Audio, Image & Video Sampling for an Audio Signal Signal Period T, f = 1/T The sampling frequency or sampling rate fs is defined as:the number of samples obtained in one second, or fs = 1/T…and is measured in Hertz Sampling period Ts, fs =1/Ts

11. Digital Audio, Image & Video Sampling for an Audio Signal fs = 2.5f A new component is added fs = 1.67f This is denoted as aliasing. Original signal

12. Digital Audio, Image & Video Sampling for an Audio Signal fs = 2f There are infinite number of possible sin waves going through the sampling points

13. Digital Audio, Image & Video Frequency Decomposition • We can use “Fourier Transform” to compute these frequency components. • Nyquist Theorem

14. Digital Audio, Image & Video Image Sampling • The sampling theorem applies to 2D signal (images) too • Nearest point interpolation.

15. Digital Audio, Image & Video Image Sampling (original image)

16. Digital Audio, Image & Video Image Sampling (Aliasing due to sampling)

17. Digital Audio, Image & Video Digitization • The samples are continuous and have infinite number of possible values. • The digitization process approximates these values with a fixed number of • numbers. • To represent N numbers, we need log2N bits.

18. Digital Audio, Image & Video Digital Audio • You often hear that an audio is 16bits at 44kHz. • 44KHz is the sampling frequency. Music has more high frequency • components than speech. 8kHz sampling is good enough for telephone • quality speech. • 16bits means each sample is represented as a 16bit integer. • Digital audio could have more than one channels.

19. Digital Audio, Image & Video Digital Images An image contains 2D samples of a surface, which can be represented as matrices. Each sample in an image is called a pixel.

20. Digital Audio, Image & Video Types of Digital Images • Grayscale image • Usually we use 256 levels for each pixel. Thus we need 8bits to represent each pixel (2^8 == 256) • Some images use more bits per pixel, for example MRI (Magnetic resonance imaging) images could use 16bits per pixel. A 8bit grayscale Image.

21. Digital Audio, Image & Video Binary Image • A binary image has only two values (0 or 1). • Binary image is quite important in image analysis and object • detection applications.

22. Digital Audio, Image & Video Binary Image [ b7 b6 b5 b4 b3 b2 b1 b0] b7 Most Significance Bitplane - MSB Less Significance Bitplane - LSB • Each bit plane is a binary image.

23. Digital Audio, Image & Video Dithering • A technique to represent a grayscale image with a binary one. 1  0  Convert image to 4 levels: I’ = floor(I/64) 2  3 

24. Digital Audio, Image & Video Dithering Matrix DefinitionA square matrix of threshold values that is repeated as a regular array to provide a threshold pattern for an entire image in the dither method of image representation 1  0  2  3  0 2 3 1 The dithering matrix is:

25. Digital Audio, Image & Video Color Image r g b 24 bit image There are other color spaces: YUV, HSV etc.

26. Digital Audio, Image & Video Color Table Image with 256 colors b It is possible to use much less colors To represent a color image without much degradation. r g Clusters of colors

27. Digital Audio, Image & Video Human Vision Human eye has two kinds of light sensitive cells. The rods and The cones Rods response curve (black and white vision) Cones response curve (color vision) R = s E(l) Sr(l)dl G = s E(l) Sg(l)dl B = s E(l) Sb(l)dl

28. Digital Audio, Image & Video Colors • Color matching function Colorimeter experiment

29. Digital Audio, Image & Video CIE Color Matching Functions • The amounts of R, G, B lighting sources to form single wavelength light • forms the color matching curves. CIE color matching curves CIE standard color matching functions CIE = International Commission on Illumination

30. Digital Audio, Image & Video Gamma Correction Linearly increasing intensity Without gamma correction Linearly increasing intensity with gamma correction

31. Digital Audio, Image & Video Video (Analog Video) Odd Frame Even frame DefinitionAnalog video is a video signal transferred by an analog signal.An analog color video signal contains luminance, brightness (Y) and chrominance (C) of an analog television image. When combined in to one channel, it is called composite video as is the case, among others with NTSC, PAL and SECAM. Analog video may be carried in separate channels, as in two channel S-Video (YC) and multi-channel component video formats.

32. Digital Audio, Image & Video Color System in Video • YUV was used in PAL (an analog video standard) and also used for digital video. • Y is the luminance component (brightness) • Y = 0.299 R + 0.587 G + 0.144 B • U and V are color components • U = B – Y • V = R - Y

33. Digital Audio, Image & Video PAL vs NTSC vs SECAM • PAL (Phase Alternating Line), is an analogue television colour encoding system used in broadcast television systems in many countries. • NTSC (National Television System Committee) receivers have a tint control to perform colour correction manually. If this is not adjusted correctly, the colours may be faulty. The PAL standard automatically cancels hue errors by phase reversal, so a tint control is unnecessary. • SECAM (Sequential Color with Memory) is an earlier attempt at compatible colour television which also tries to resolve the NTSC hue problem. It does so by applying a different method to colour transmission, namely alternate transmission of the U and V vectors and frequency modulation, while PAL attempts to improve on the NTSC method. SECAM transmissions are more robust over longer distances than NTSC or PAL.

34. Digital Audio, Image & Video PAL vs NTSC vs SECAM