Lecture 6 Instantaneous Codes and Kraft’s Theorem (Section 1.4)

1. Lecture 6Instantaneous Codes and Kraft’s Theorem(Section 1.4) Theory of Information

2. Instantaneous Codes DEFINITION A code is said to be instantaneous if, whenever any sequence of codewords is transmitted, each codeword can be interpreted as soon as it is received. Example. {0, 01} {0, 10}

3. Prefix and Suffix Properties DEFINITION 1) A code is said to have the prefix property if no codeword is a prefix of any other codeword. 2) A code is said to have the suffix property if no code word is a suffix of any other codeword. Fact: Having the suffix or prefix property is sufficient to be uniquely Decipherable. But not vice versa. Does {0,01} have the prefix property? suffix property? Is it uniquely decipherable? Does {00, 001000, 001011, 11} have the prefix property? suffix property? Is it uniquely decipherable?

4. Prefix Property = Instantaneousity THEOREM 1.4.1 A code is instantaneous if and only if it has the prefix property. Comma code: {0, 10, 110, 1110} has the prefix property and is instantaneous. How about {0, 01, 011, 0111}: Prefix property? Instantaneous? Suffix property? Uniquely decipherable?

5. THEOREM 1.4.2 • There exists an instantaneous r-ary code with codeword lengths • a1,a2,…,aq if and only if Kraft’s inequality is satisfied: • 1/ra1 + … + 1/raq 1. • Let C be an instantaneous r-ary code. Then C is maximal • instantaneous, i.e. C is not contained in any strictly larger instantaneous • code, if and only if equality holds in Kraft’s inequality. • Suppose that C is an instantaneous code with maximum codeword • Length m. If C is not maximal, then it is possible to add a word of length • m to C without destroying its property of being instantaneous. Kraft’s Theorem

6. THEOREM 1.4.2 • There exists an instantaneous r-ary code with codeword lengths • a1,a2,…,aq if and only if Kraft’s inequality is satisfied: • 1/ra1 + … + 1/raq 1. Kraft’s Theorem Note: That a given code C satisfies Kraft’s inequality does not necessarily mean that C is instantaneous; maybe another code D --- with the same codeword lengths --- is instantaneous rather than C itself. Example: C={0, 11, 100, 110} Does C satisfy Kraft’s inequality? Is C instantaneous? However, D={0,11,101,100} has the same codeword lengths and Is instantaneous.

7. Kraft’s theorem allows us to find an instantaneous code (if such exists) with given codelengths a1a2… aq as follows: For i=1 to i=q do: Pick any codeword of length ai such that no earlier codeword is its prefix The Utility of Kraft’s Theorem Example: Construct a ternary code {c1,c2,c3,c4,c5,c6} with codelengths 1, 1, 2, 4, 4, 5 Kraft’s test to see if such a code exists: c1= c2= c3= c4= c5= c6=

8. Reasonable Uniquely Decipherable Codes Are Instantaneous THEOREM 1.4.3. If a uniquely decipherable code exists with codeword lengths l1,l2,…,ln, then an instantaneous code must also exist with these same codeword lengths. Proof: Suppose the above uniquely decipherable code exists. Then, by McMillan’s Theorem, Kraft’s inequality should be satisfied for those lengths. But then, by Kraft’s Theorem, there must also exist an Instantaneous code with the same codelengths.

9. Homework Exercises 2,3,4,5,6,7,8,9,10,11,12,13,14 of Section 1.4.