Digital Fountains: Applications and Related Issues

1. Digital Fountains:Applications and Related Issues Michael Mitzenmacher

2. Goals • Explain the digital fountain paradigm for network communication. • Examine related advances in coding. • Summarize work on applications. • Speculate on what comes next. • For more, see • Digital Fountains: A Survey and Look Forward • www.eecs.harvard.edu/~michaelm/ListByYear.html

3. What is a Digital Fountain? • For this talk, a digital fountain is an ideal/paradigm for data transmission. • Vs. the standard (TCP) paradigm: data is an ordered finite sequence of bytes. • Instead, with a digital fountain, a k symbol file yields an infinite data stream; once you have received any k symbols from this stream, you can quickly reconstruct the original file.

4. How Do We Build a Digital Fountain? • We can construct (approximate) digital fountains using erasure codes. • Including Reed-Solomon, Tornado, LT, fountain codes. • Generally, we only come close to the ideal of the paradigm. • Streams not truly infinite; encoding or decoding times; coding overhead.

5. History • Reed-Solomon codes • Tornado Codes • Luby Transform • Rateless/Raptor codes

6. Raptor/Rateless Codes • Properties: • “Infinite” supply of packets possible. • Need k(1+e) symbols to decode, for some e > 0. • Decoding time proportional to k ln (1/e). • On average, ln (1/e) (constant)time to produce an encoding symbol. • Conclusion: these codes can be made very efficient and deliver on the promise of the digital fountain paradigm.

7. Applications • Reliable multicast • Parallel downloads • Long-distance transmission (avoiding TCP) • One-to-many TCP • Content distribution on overlay networks • Streaming video

8. Reliable Multicast • Many potential problems when multicasting to large audience. • Feedback explosion of lost packets. • Start time heterogeneity. • Loss/bandwidth heterogeneity. • A digital fountain solves these problems. • Each user gets what they can, and stops when they have enough.

9. Downloading in Parallel • Can collect data from multiple digital fountains for the same source seamlessly. • Since each fountain has an “infinite” collection of packets, no duplicates. • Relative fountain speeds unimportant; just need to get enough. • Combined multicast/multigather possible. • Can be used for BitTorrent-like applications.

10. Point-to-Point Data Transmission • TCP has problems over long-distance connections. • Packets must be acknowledged to increase sending window (packets in flight). • Long round-trip time leads to slow acks, bounding transmission window. • Any loss increases the problem. • Using digital fountain + TCP-friendly congestion control can greatly speed up connections. • Separates the “what you send” from “how much” you send. • Do not need to buffer for retransmission.

11. One-to-Many TCP • Setting: Web server with popular files, may have many open connections serving same file. • Problem: has to have a separate buffer, state for each connection to handle retransmissions. • Limits number of connections per server. • Instead, use a digital fountain to generate packets useful for all connections for that file. • Separates the “what you send” from “how much” you send. • Do not need to buffer for retransmission. • Keeps TCP semantics, congestion control.

12. Distribution on Overlay Networks • Encoded data make sense for overlay networks. • Changing, heterogeneous network conditions. • Allows multicast. • Allows downloading from multiple sources, as well as peers. • Problem: peers may be getting same encoded packets as you, via the multicast. • Not standard digital fountain paradigm. • Requires reconciliation techniques to find peers with useful packets.

13. Video Streaming • For “near-real-time” video: • Latency issue. • Solution: break into smaller blocks, and encode over these blocks. • Equal-size blocks. • Blocks increases in size geometrically, for only logarithmically many blocks. • Engineering to get right latency, ensure blocks arrive on time for display.

14. Other Applications • Other possible applications outside of networking • Storage systems • Digital fountain codes for errors • Others???

15. Putting Digital Fountains To Use • Digital fountains are out there. • Digital Fountain, Inc. sells them. • Limitations to their use: • Patent issues. • Perceived complexity. • Lack of reference implementation. • What is the killer app?

16. Patent Issues • Several patents / patents pending on irregular LDPC codes, LT codes, Raptor codes by Digital Fountain, Inc. • Supposition: the theory/practice of digital fountains was greatly developed by the company and its employees. • Supposition: but this stifles external innovation. • Potential threat of being sued. • Potential lack of commercial outlet for research. • Suggestion: need unpatented alternative that approximate a digital fountain. • There is work going on in this area, but more is needed to keep up with recent developments in rateless codes.

17. Perceived Complexity • Digital fountains are now not that hard… • …but networking people do not want to deal with developing codes. • A research need: • A publicly available, easy to use, reasonably good black box digital fountain implementation that can be plugged in to research prototypes. • Issue: patents. • Legal risk suggests such a black box would need to be based on unpatented codes.

18. What’s the Killer App? • Multicast was supposed to be the killer app. • But IP multicast was/is a disaster. • Distribution now handled by content distributions companies, e.g. Akamai. • Possibilities: • Overlay multicast. • General wireless: e.g. video phones. • Specialized wireless: e.g. automobiles. • Others???

19. Conclusions • Digital fountain paradigm and enabling codes have significant potential. • Many proposed applications. • More to come. • Applications helped push forward the technology. • Codes with better and better properties. • Challenge in moving from a “technology” to use in the real-world. • A simple, easy-to-use implementation based on non-proprietary might spur research community. • Need more potential killer apps to spur business community.