Introduction to Coding Theory

1. Introduction to Coding Theory Rong-Jaye Chen

2. Outline • [1] Introduction • [2] Basic assumptions • [3] Correcting and detecting error patterns • [4] Information rate • [5] The effects of error correction and detection • [6] Finding the most likely codeword transmitted • [7] Some basic algebra • [8] Weight and distance • [9] Maximum likelihood decoding • [10] Reliability of MLD • [11] Error-detecting codes • [12] Error-correcting codes

3. Information Source Transmitter(Encoder) CommunicationChannel Receiver(Decoder) Information Sink Noise Introduction to Coding Theory • [1] Introduction • Coding theory • The study of methods for efficient and accurate transfer of information • Detecting and correcting transmission errors • Information transmission system n-digit n-digit k-digit k-digit

4. Introduction to Coding Theory [2] Basic assumptions Definitions • Digit：0 or 1(binary digit) • Word：a sequence of digits • Example：0110101 • Binary code：a set of words • Example：1. {00,01,10,11} , 2. {0,01,001} • Block code ：a code having all its words of the same length • Example： {00,01,10,11}, 2 is its length • Codewords ：words belonging to a given code • |C| ： Size of a code C(#codewords in C)

5. Channel Channel 1. Receiving word by word Introduction to Coding Theory • Assumptions about channel 2. Identifying the beginning of 1st word 3. The probability of any digit being affected in transmission is the same as the other one.

6. reliability Introduction to Coding Theory Binary symmetric channel p: In many books, p denotes crossover probability. Here crossover probability(error prob.) is 1-p

7. correct source Channel correct source Channel Introduction to Coding Theory • [3] Correcting and detecting error patterns Any received word should be corrected to a codeword that requires as few changes as possible. Cannot detect any errors !!! parity-check digit

8. Introduction to Coding Theory • [4] Information rate • Definition: information rate of code C • Examples

9. Introduction to Coding Theory [5] The effects of error correction and detection 1. No error detection and correction Let C={0,1}11={0000000000, …, 11111111111} Transmission rate=107 digits/sec Reliability p=1-10-8 Then Pr(a word is transmitted incorrectly) = 1-p11 ≒11x10-8 11x10-8(wrong words/words)x107/11(words/sec)=0.1 wrong words/sec 1 wrong word / 10 sec 6 wrong words / min 360 wrong words / hr 8640 wrong words / day

10. Introduction to Coding Theory 2. parity-check digit added(Code length becomes 12 ) Any single error can be detected ! (3, 5, 7, ..errors can be detected too !) Pr(at least 2 errors in a word)=1-p12-12 x p11(1-p)≒66x10-16 So 66x10-16 x 107/12 ≒ 5.5 x 10-9 wrong words/sec one word error every 2000 days! The cost we pay is to reduce a little information rate + retransmission(after error detection!)

11. Introduction to Coding Theory 3. 3-repetition code Any single error can be corrected ! Code length becomes 33 and information rate becomes 1/3 Task：design codes with • reasonable information rates • low encoding and decoding costs • some error-correcting capabilities

12. BSC channel Introduction to Coding Theory • [6] finding the most likely codeword transmitted • Example： p ：reliability d ：#digits incorrectly transmitted n ：code length Code length = 5

13. Assume is sent when is received if Introduction to Coding Theory • Theorem 1.6.3 Suppose we have a BSC with ½ < p < 1. Let and be codewords and a word, each of length . Suppose that and disagree in positions and and disagree in positions. Then

14. channel Introduction to Coding Theory • Example

15. Multiplication： Addition： ：the set of all binary words of length n Addition： Scalar multiplication： ：zero word Introduction to Coding Theory • [7] Some basic algebra

16. Introduction to Coding Theory • Kn is a vector space words of length n scalar

17. Introduction to Coding Theory • [8] Weight and distance • Hamming weight： • the number of times the digit 1 occurs in • Example： • Hamming distance： • the number of positions in which and disagree • Example：

18. words of length n digit Introduction to Coding Theory • Some facts：

19. CMLDIMLD channel decode n k Introduction to Coding Theory • [9] Maximum likelihood decoding • CMLD：Complete Maximum Likelihood Decoding • If only one word v in C closer to w , decode it to v • If several words closest to w, select arbitrarily one of them • IMLD：Incomplete Maximum Likelihood Decoding • If only one word v in C closer to w, decode it to v • If several words closest to w, ask for retransmission w=v+u Source string x codeword Error pattern

20. error pattern Introduction to Coding Theory The most likely codeword sent is the one with the error pattern of smallest weight Example：Construct IMLD. |M|=3 , C={0000, 1010, 0111}

21. Introduction to Coding Theory • [10] Reliability of MLD • The probability that if v is sent over a BSC of probability p then IMLD correctly concludes that v was sent The higher the probability is, the more correctly the word can be decoded!

22. Can’t detect Can detect + Error pattern Introduction to Coding Theory • [11] Error-detecting codes Example： Can detect Can’t detect

23. Introduction to Coding Theory • the distance of the code C ： • the smallest of d(v,w) in C • Theorem 1.11.14 • A code C of distance d will at least detect all non-zero error patterns of weight less than or equal to d-1. Moreover, there is at least one error pattern of weight d which C will not detect. • t error-detecting code • It detects all error patterns of weight at most t and does not detect at least one error pattern of weight t+1 • A code with distance d is a d-1 error-detecting code.

24. For all v in C , if it is closer to v than any other word in C, a code C can correct u. + Error pattern Introduction to Coding Theory • [12] Error-correcting codes • Theorem 1.12.9 • A code of distance d will correct all error patterns of weight less than or equal to . Moreover, there is at least one error pattern of weight 1+ which C will not correct. • t error-correcting code • It corrects all error patterns of weight at most t and does not correct at least one error pattern of weight t+1 • A code of distance d is a error-correcting code.

25. Introduction to Coding Theory C corrects error patterns 000,100,010,001