Lecture 4: Lossless Compression(1) Hongli Luo Fall 2011

1. Lecture 4: Lossless Compression(1)Hongli LuoFall 2011

2. Lecture 4: Lossless Compression (1) • Topics (Chapter 7) • Introduction • Basics of Information Theory • Compression techniques • Lossless compression • Lossy compression

3. 4.1 Introduction • Compression: the process of coding that will effectively reduce the total number of bits needed to represent certain information.

4. Introduction • Compression ratio: • B0 - number of bits before compression • B1 - number of bits after compression

5. Types of Compression • Lossless compression • Does not lose information – the signal can be perfectly reconstructed after decompression • Produces a variable bit-rate • It is not guaranteed to actually reduce the data size • Depends on the characteristics of the data • Example : Winzip • Lossy compression • Loses some information – the signal is not perfectly reconstructed after decompression • Produces any desired constant bit-rate • Exampe: JPEG, MPEG

6. 4.2 Basics of Information Theory • Model Information at the Source • Model data at the Source as a Stream of Symbols –This defines the “Vocabulary” of the source. • Each symbol in the vocabulary is represented by bits • If your vocabulary has N symbols, each symbol represented with log2N bits. • Text by ASCII code – 8bits/code: N=28=256 symbols • Speech -16 bits/sample: N=216=65,536 symbols • Color Image: 3x8 bits/sample: N=224=17x106 symbols • 8x8 Image Blocks: 8x64 bits/block: N=2512=1077 symbols

7. Lossless Compression • Lossless compression techniques ensure no loss of data after compression/decompression. • Coding: “Translate” each symbol in the vocabulary into a “binary codeword”. Codewords may have different binary lengths. • Example: You have 4 symbols (a, b, c, d). Each in binary may be represented using 2 bits each, but coded using a different number of bits. • a(00) -> 000 • b(01) -> 001 • c(10) -> 01 • d(11) -> 1 • Goal of Coding is to minimize the average symbol length

8. Average Symbol Length • The vocabulary of the source has N symbols • l(i)– binary length of ith symbols • Symbol i has been emitted m(i) times • M = number of symbols that the source emits (on every T second) • Number of bits been emitted in T second • Probability P(i) of a symbol: number of times it occurs in the transmission and is defined as P(i) = m(i) /M

9. Average Symbol Length • Average length per symbol / average symbol length • Average bit rate

10. Minimum Average Symbol Length • Goal of compression • to minimize the number of bits being transmitted • Equivalent to minimize the average symbol length • How to reduce the average symbol length • Assign shorter codewords to symbols that appear more frequently, • Assign longer codewords to symbols that appear less frequently

11. Minimum Average Symbol Length • What is the lower bound of average symbol length? • Decided by the entropy • Shannon’s Information Theorem • The average binary length of the encoded symbol is always greater than or equal to the entropy H of the source

12. Entropy • The entropy ηof an information source with alphabet S = {s1, s2, …, sn} is: pi - probability that symbol si will occur in S. indicates the amount of information ( self- information as defined by Shannon) contained in si, which corresponds to the number of bits needed to encode si.

13. Entropy • The entropy is characteristics of a given source of symbols • Entropy is largest (equal to log2N) when all symbols are equally probable • Entropy is small (always >=0) when some symbols are much more likely to appear than other symbols • The chances that each symbol appear are similar, or symbols are uniformly distributed in the source

14. Entropy and code length • The entropy represents the average amount of information contained per symbol in the source S. • The entropy species the lower bound for the average number of bits to code each symbol in S, i.e., • - the average length (measured in bits) of the codewords produced by the encoder. • Efficiency of the Encoder

15. Distribution of Gray-Level Intensities Fig. 7.2(a) shows the histogram of an image with uniform distribution of gray-level intensities, i.e., pi = 1/256. Hence, the entropy of this image is: log2 256 = 8 (7.4) - No compression is possible for this image!

16. 4.3 Compression Techniques • Compression techniques are broadly classified into • Lossless compression • Run-length encoding • Variable length coding (entropy coding): • Shannon-fano algorithm, • Huffman coding, • adaptive Huffman coding • Arithmetic coding • LZW • Lossy compression

17. Run-length Encoding • Sequence of elements, c1, c2, …, ci,…, is mapped to a run (ci,li) • Ci = symbol • li = length of the symbol ci’s run • For example, given the sequence of symbols {1,1,1,1,3,3,4,4,4,3,3,5,2,2,2} The run-length encoding is (1,4),(3,2),(4,3),(3,2),(5,1),(2,3) • Apply run-length encoding to a bi-level image (with only 1-bit black and while pixels) • Assume the starting run is of a particular color (either black or white) • Code the length of each run

18. Variable Length Coding • VLC generates variable length codewords from fixed length • VLC is one of the best known entropy coding method • Methods of VLC • Shannon-Fano algorithm • Huffman coding • Adaptive Huffman coding

19. Shannon-Fano Algorithm A top-down approach, Steps: 1. Sort the symbols according to the frequency count of their occurrences. 2. Recursively divide the symbols into two parts, each with approximately the same number of counts, until all parts contain only one symbol. An Example: coding of “HELLO” Sort symbols according to their frequencies, LHEO

20. Assign bit 0 to its left branches and 1 to the right branches.

21. Coded bits: 10 bits • Raw datawords, 8 bits per character: 40 bits • Compression ratio : 10/40 = 25%