On-The-Fly Verification of Rateless Erasure Codes

1. On-The-Fly Verification of Rateless Erasure Codes Max Krohn (MIT CSAIL) Michael Freedman and David Mazières (NYU)

2. Multicast Authentication: Dead/Exhausted On-The-Fly Verification of Rateless Erasure Codes Max Krohn (MIT CSAIL) Michael Freedman and David Mazières (NYU)

3. The Setting • A large file F • Linux ISO (650MB) • H(F) is available • signed by Publisher (RedHat) • A handful of untrusted sources/mirrors S1,…S8

4. A Handful of Senders

5. The Setting • A large file F • Linux ISO (650MB) • H(F) is available • signed by Publisher (RedHat) • A handful of untrusted sources S1,…S8 • Their aggregate BW is limited • A slew of receivers R1,...,R1,000,000 • Version 81.3 just released! Want it Now!

6. Three Desirable Properties Clients Get Fast Downloads Sources Can Multicast Clients Can Verify Blocks On-the-Fly

7. Receivers Get Fast, Verifiable Downloads • The trusted publisher (RedHat) • Splits up F into n blocks • Hashes all blocks • Signs all hashes (or hash tree) • Receivers: • Download and verify hashes • Download needed file blocks in parallel

8. Everyone for Themselves S3 S2 S4 S1 R7 R9 R2 R12 R3 R8 R10 R1 R13 R4 R11 R6 R5

9. Everyone For Themselves Clients Get Fast Downloads Sources Can Multicast Clients Can Verify Blocks On-the-Fly

10. Verifiable Multicast (BitTorrent) S3 S2 S4 S1 R12 R7 R10 R13 R5 R6 R9 R8 R3 R1 R2 R11 R4

11. Verifiable Multicast (BitTorrent) Clients Get Fast Downloads Sources Can Multicast Clients Can Verify Blocks On-the-Fly

12. … ? ? ? ? ? ? ? ? ? ? … ? ? ? ? ? ? ? ? ? ? … ? ? ? ? ? ? ? ? ? ? … ? ? ? ? ? ? ? ? ? ? … ? ? ? ? ? ? ? ? ? ? Multicast With Erasure Codes • Sources erasure encode the file F blocks F n blocks

13. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Multicast With Erasure Codes … … • Sources erasure encode the file F • Receivers collect blocks and decode … … … n blocks F blocks F n blocks 1.03n blocks

14. Multicast With Erasure Codes S2 S3 S1 S4 R8 R9 R10 R6 R5 R12 R4 R11 R1 R3 R13 R7 R2

15. Multicast With Erasure Codes • Bullet [SOSP 2003] • SplitStream [SOSP 2003] • Big Downloads [IPTPS 2003] • Informed Content Delivery [SIGCOMM 2002]

16. Receivers Cannot Verify Content S2 S3 S1 S4 ? ? ? ? ? ? ? ? ? ? ? ? ? ? R1

17. Receivers Cannot Verify Content S2 S3 S1 S4 ? ? ? ? ? ? ? ? ? ? R1

18. Multicast With Erasure Codes Clients Get Fast Downloads Sources Can Multicast Clients Can Verify Blocks On-the-Fly

19. Multicast With Erasure Codes Clients Get Fast Downloads Sources Can Multicast Clients Can Verify Blocks On-the-Fly

20. What is the Attack Goal? S2 S3 • To corrupt the file. • To waste bandwidth. S1 R

21. How To Attack? • Send correct blocks but with skewed distributions. • “Distribution Attack” • Send incorrect blocks • “Pollution Attack” • Karlof et al. [NDSS ’04] S2 S3 S1 R

22. Properties of a Solution to Pollution S2 S3 • OK: Receivers can tell good from bad. • Much better: Receivers can finger bad blocks as they arrive. S1 R CONTRIBUTION

23. Outline • Introduction • Review of LT Codes • Strawman #1 • Strawman #2 • Efficiently Catching Bad Blocks as They Arrive

24. LT-Codes [Luby, FOCS 2002] b1 F= b2 b3 b4 b5 n=5input blocks

25. LT-Codes – How To Encode • Pick degreed1 from a pre-specified distribution. (d1=2) • Select d1 input blocks uniformly at random. (Pick b1 and b4 ) • Compute their sum. • Output E(F)= c1 b1 F= b2 b3 b4 b5

26. LT-Codes – How To Encode (cont’d) E(F)= c1 c2 c3 c4 c5 c6 c7 b1 F= b2 b3 b4 b5

27. How To Decode E(F)= c1 c2 c3 c4 c5 c6 c7 b1 F= b2 b3 b4 b5

28. How To Decode E(F)= c1 c2 c3 c4 c5 c6 c7 b1 F= b2 b3 b4 b5

29. How To Decode E(F)= c1 c2 c3 c4 c5 c6 c7 b1 F= b2 b3 b4 b5

30. How To Decode E(F)= c1 c2 c3 c4 c5 c6 c7 b1 F= b2 b3 b4 b5

31. How To Decode E(F)= c1 c2 b5 c3 b5 c4 b5 c5 c6 c7 b1 F= b2 b3 b4 b5

32. c4 b5 How To Decode E(F)= c1 c2 b5 c3 b5 c5 c6 c7 b1 F= b2 b3 b4 b5

33. c4 b5 How To Decode E(F)= c1 c2 b5 c3 b5 c5 c6 c7 b1 F= b2 b3 b4 b5

34. c4 b5 How To Decode E(F)= c1 c2 b5 c3 b5 c5 c6 c7 b1 F= b2 b3 b4 b5

35. b5 c4 How To Decode E(F)= c1 c2 b2 c3 b5 b2 b5 c5 c6 c7 b1 F= b2 b3 b4 b5

36. b5 c4 How To Decode E(F)= c1 c2 b2 c3 b5 b2 b5 c5 c6 c7 b1 F= b2 b3 b4 b5

37. Outline • Introduction • Review of LT Codes • Strawman #1 • Simple Solution To Tell Good Blocks From Bad • Strawman #2 • Efficiently Catching Bad Blocks as They Arrive

38. “Smart Decoder” for LT-Codes E(F)= c1 c2 c3 c4 c5 c6 c7 b1 F= b2 b3 b4 b5

39. “Smart Decoder” for LT-Codes E(F)= c1 c2 c3 c4 c5 c6 c7 b1 F= b2 b3 b4 b5

40. “Smart Decoder” for LT-Codes E(F)= c1 c2 c3 c4 c5 c6 c7 b1 F= b2 b3 b4 b5

41. “Smart Decoder” for LT-Codes E(F)= c1 c2 c3 c4 c5 c6 c7 b1 F= b2 b3 b4 b5

42. “Smart Decoder” for LT-Codes E(F)= c1 c2 c3 c4 c5 c6 c7 b1 F= b2 b3 b4 b5

43. “Smart Decoder” for LT-Codes E(F)= c1 c2 b5 c3 b5 c4 b5 c5 c6 c7 b1 F= b2 b3 b4 b5

44. “Smart Decoder” for LT-Codes E(F)= c1 c2 b5 c3 b5 c4 b5 c5 c6 c7 b1 F= b2 b3 b4 b5

45. “Smart Decoder” for LT-Codes E(F)= c1 c2 b5 c3 b5 c4 c5 b5 c6 c7 X b1 F= b2 b3 b4 b5

46. “Smart Decoder” for LT-Codes X E(F)= c1 c2 b5 c3 c4 b5 b5 c5 c6 c7 b1 F= b2 b3 b4 b5

47. “Smart Decoder” for LT-Codes E(F)= c1 c2 b5 c3 b5 c4 b5 c5 c6 c7 b1 F= b2 b3 b4 b5

48. “Smart Decoder:” Problem • Data collected from 50 random Online encodings of a 10,000 block file.

49. Outline • Introduction • Review of LT Codes • Strawman #1 • Strawman #2 • Hashing/Signing Encoded Blocks • Efficiently Catching the Bad as They Arrive

50. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Hashing/Signing Encoded Blocks n blocks e·n blocks F • Trusted Publisher (RedHat) • Picks e, computes e·n encoded blocks • Hashes all encoded blocks • Signs the hashes.