Erasure Codes for Reliable Communication Protocols

1. Erasure Codes for Reliable Communication Protocols Presented by Sen Yang Based on L.Rizzo’spaper and online resource. Effective Erasure Codes for Reliable Computer Communication Protocols

2. Outline • Motivation • Math • Algorithm • Applications

3. Automatic Repeat reQuest (ARQ)

4. Automatic Repeat reQuest (ARQ) • ACK and timeouts • Receiver sends ACK when it receives packet • Sender waits for ACK and times out

5. Forward Error Correction

6. Example • Alice wants to send her phone number (555629) to Bob via internet.

7. Internet not good • However, the internet transmission are not good: • 1. Half of all the messages gets lost. • 2. Messages longer than 5 characters are illegal. • 3. It is very expensive .

8. Encode Process • Alice is very smart, she design the following rules to send the messages

9. Step 1 • Alice breaks her phone number up into two parts a = 555, b = 629 • sends 2 messages "A = 555" and "B = 629“ to Bob. • Bob might miss them!

10. Encode Process • Alice constructs a linear function • f(i)= a + (b-a)(i-1) • such that • f(1) =555 and f(2)=629.

11. Encode Process • How to construct it? • Solve the linear equations and find • a=555, b-a=74 • Therefore, a=555, b=629

12. Step 2 • Alice computes the values f(3), f(4), and f(5) and then also transmits three redundant messages: • "C = 703", "D = 777" and "E = 851".

13. Produce redundant

14. Receive & lose • Now suppose Bob only receives "D = 777" and "E = 851".

15. How recover a and b?

16. Bob’s homework • Before the transmission , Bob knows that the form of f(k) is • f(i)= a + (b-a)(i-1) , • where a and bare the two parts of the phone number.

17. Decode • Now he can use f(4) and f(5) to find a and b. • f(4) = a+ (b-a)(4-1)=777 • f(5)= a+ (b-a)(5-1)=851

18. Decode • Solve linear equations! Easy.

19. Advanced tools • In real world, we need advanced mathematical tools, such as matrices and finite field, to perform encode &decode • Some mathematics are in order.

20. Finite field • Finite Field is a field which only has finite elements . • Z/2Z={0,1} • Z/pZ={0,1,….p-1} , where p is a prime. • called Prime fields.

21. Classify • Each finite field is of size where p must be a prime. • In CS, p=2 is very useful. • Now, let p=2, k=2.

22. Classify • Can you construct a field of size 4? • Any idea? • How about Z/4Z? Why?

23. Two problems -1 • When considering software implementations, twominor difficulties in using a prime field: • 1.) with the exception of p = 2, field elements require > bits to be represented. • Causes a slight inefficiency in the encoding of data

24. Two problems -2 • 2.) modulo operation on sums and multiplications are expensive. • Why ? • Eg: In Z/5Z: • 2+4=2 • 2*4= 3

25. How to overcome ? • Any finite field has a generator, denoted g. • Eg. In Z/5Z, let’s consider the power of 2: • 1, 2, 4, 3,

26. generator • Express any non-zero element x as • X= , • g is the generator • some positive integer

27. generator • multiplication and division can be computed as follows: • xy = , • Where stands for "a modulo b“ • q is the size of the finite field.

28. generator • If the size of field not too large, we make tables to record • operations in extension fields with p = 2 can be extremely fast and simple to implement

29. Matrix

30. Vandermonde matrix

31. Vandermonde • The determinant of a square Vandermonde matrix (where m = n) can be expressed as

32. Erasure code

33. Encode • Source data: • G: n x k Vandermondematrix, • y= Gx

34. Decode • y’ : k components of y available at the receiver. • G’ : matrix corresponding to y’. • y’= G’x • x = Hy’ , H is the inverse.

35. forward error recovery on long delay paths

36. Mobile devices- power save

37. videoconferencing

38. one-to-many file transfer

39. XorbasHadoop

40. Big data • XORing Elephants: Novel Erasure Codes for Big Data. • Use erasure code to reduce the large storage overhead of three-replicated systems