Chapter 1

1. Chapter 1 Introduction

2. Outline • 1.1 A Very Abstract Summary • 1.2 History • 1.3 Model of the Signaling System • 1.4 Information Source • 1.5 Encoding a Source Alphabet • 1.6 Some Particular Codes • 1.7 The ASCII Code • 1.8 Some other Codes • 1.9 Radix r Codes

3. 1.1 A Very Abstract Summary • Information source: • symbols s1, s2, . . . , sq • Symbol • They can be uniquely recognized. • The meanings of the word “Symbol” is intuitive, and can not be defined by Mathematics. • Because Define Symbol

4. symbols s1s2 . . . . sq probabilities p1p2 . . . . pq • One way to get estimates of them is to examine past usage of the symbol system and hope that the future is not significantly different from the past. • For any discrete probability distribution: • It is called the entropy function. • The entropy function measures the amount of uncertainty, surprise, or information.

5. Binary System: • It consists of the two symbols 0 and 1. • The problem of representing the source alphabet symbols si in terms of another system of symbols (usually the binary system) is the main topic of the course.

6. The two main problems of representation are: • 1. Channel encoding: How to represent the source symbols so that their representations are far apart in some suitable sense. • 2. Source encoding: How to represent the source symbols in a minimal form for purposes of efficiency. i.e. min L where , is the length of the representation of the i th symbol si. ＊Application: Storage, Communication.

7. 1.2 History • In 1948, C. E. Shannon published two papers on “A Mathematical Theory of Communication” in the Bell System Technical Journal. • Information theory sets bounds on what can be done, but does little to aid in the design of a particular system.

8. 1.3 Model of the Signaling System Noise (4) • The conventional signaling system is modeled by: (1) An information source. (2) An encoding of this source. (3) A channel over, or through, which the information is sent. (4) A noise (error) source that is added to the signal in the channel. (5) A decoding and hopefully a recovery of the original information from the contaminated received signal. (6) A sink for the information. Source Encode Channel Decode Sink (1) (2) (3) (5) (6)

9. Among Encode, it consists of: • (1) The more structure there is, the more, typically, we can compress the encoding. • (2) These encoded symbols are then further encoded to compensate for any known properties of the channel. Typically this second stage of encoding expands the representation of the message. • In other words, we first do source encoding, then do channel encoding.

10. 1.4 Information Source • A source of information • a sequence of symbols in a source alphabet s1, s2, . . . , sq. • The source of Information • They could be many things; for example, a book, a report, etc.

11. 1.5 Encoding a Source Alphabet • Currently, devices with two states, called binary devices are much more reliable than are multistate devices. As a result, binary system dominate all others. • 0 and 1 are used to represent two states of binary devices. And two binary devices can represent four states. (00, 01, 10, 11)

12. For a system having k binary digits – usually abbreviated as bits, the total number of distinct states is 2k. • If we have k different independent devices, the first having n1 states, the second n2 states, . . . ,the kth having nk states, then the total number of states is clearly the product • Typically, the channel encoding of the message increases the redundancy. The source encoding usually decreases the redundancy.

13. 1.6 Some Particular Codes • The octal representation (25)10 = (31)8 • The hexadecimal representation (25)10 = (19)16

14. 1.7 The ASCII code 7-bit (127)8 = 1 010 111 8-bit (127)8 = 11 010 111 even-parity check 01 010 111 odd-parity check

15. 1.8 Some Other Codes The “−” represents three units of “.” In fact, it a ternary code, having the symbols dot, dash, and space

16. 1.8 Some Other Codes • The space between dots and dashes in a single letter is 1 unit of time, between letters it is 3 time units, and between words it is 6 time units. • Morse code is a variable-length code that takes advantage of the high frequency of occurrence of some letters, such as “E,” by marking them short, and the very infrequent letters, such as “J,” relatively longer. • E: . • J: .−−−

17. It is not easy to decode with variable-length codes, so it is replaced by van Duuren. • Van Duuren code: uses three out of seven positions filled with 1’s and the other four with 0’s. (C(7.3)=35 possible words in this code ) • Van Duuren code is the same as 8 bit ASCII code, and also can detect errors.

18. Another widely used simple code is the 2-out-of-5 code. • two out of five positions are filled with1’s • C(5.2) = 10 possible symbols • One if the many ways of associating the code symbols with the numerical values of the decimal digits is the 01247 code, which is a weighted code. • All single errors can be detected.

19. The 2-Out-Of-5 code

20. 1.9 Radix r Codes • Two-state devices tend to be more reliable than multi-state devices. • Human begins clearly work better with multistate systems. • Such as alphabets or decimal numbers • It is necessary at times to consider codes with r symbols in their alphabets. • Such as Morse code, r = 3

21. For example, a source alphabet S consists of q symbols s1, s2, ..., sq. It can be represented by binary code model. • Such as ASCII codes having r = q = 27 = 128 symbols.