Capacity Upper Bounds for Deletion Channels

1. Capacity Upper Bounds for Deletion Channels Suhas Diggavi Michael Mitzenmacher Henry Pfister

2. The Most Basic Channels • Binary erasure channel. • Each bit is replaced by a ? with probability p. • Binary symmetric channel. • Each bit flipped with probability p. • Binary deletion channel. • Each bit deleted with probability p.

3. The Most Basic Channels • Binary erasure channel. • Each bit is replaced by a ? with probability p. • Very well understood. • Binary symmetric channel. • Each bit flipped with probability p. • Very well understood. • Binary deletion channel. • Each bit deleted with probability p. • We don’t even know the capacity!!!

4. The Challenge • Would like a single-letter characterization of capacity. • Or tight upper/lower bounds. • Or effective means of calculating capacity. • Such characterizations are difficult. • Deletion channels are channels with memory.

5. Recent Progress • Chain of results giving better lower bounds. • Shannon-style arguments. • Diggavi/Grossglauser, Drinea/Mitzenmacher, Drinea/Kirsch. • Global result: capacity is at least (1 – p)/9. • But essentially no work on upper bounds. • Ullman’s bound: zero-error decoding for worst-case synchronization errors. (Does not apply.) • Trivial bound of (1 – p). • Lower bound progress motivates need for progress in the other direction. • How close are we getting to capacity???

6. Our Results • An upper bound approach using side information that gives numerical bounds. • An upper bound approach using side information that gives asymptotic behavior as fraction of deletions p goes to 1, for a bound of c(1 – p).

7. Upper Bound via Run Lengths • Input can be thought of as runs of maximally contiguous 0s/1s. • Side information: Suppose an undeletable marker inserted every time a complete run is deleted. • Can only increase capacity. • Equivalent to a memoryless channel where symbols are run lengths. 0000110111000110000….

8. Example 0000110111000110000…. Sent: 0011*11000*00…. Received: 4 2 1 3 3 2 4…. Sent: 2 2 0 2 3 0 2…. Received:

9. Capacity Per Unit Cost • Associate cost of x with run of length x at input. • Capacity of binary channel with side info = Capacity per unit cost of run length channel. • Latter can be upper bounded numerically using Kuhn-Tucker conditions. • Challenging because of infinite alphabet.

10. Upper Bound Statement • For channel defined by pY | X and any output distribution qY let • Then for any non-negative cost function c(x), the capacity per unit cost C satisfies • [Abdel-Ghaffar 1993]

11. Upper Bound Calculation • Problem: Optimization over infinite alphabet. • Solution: Finitize the problem. • Find optimal input/output distribution for truncated alphabet. • Replace tail of finite output distribution with geometric distribution. • To allow analytic bound on for large x. • Bound performance of resulting distribution. • Optimize over truncated alphabet.

12. Bounds Achieved

13. Asymptotic Result • Motivation: • Previous upper bound approach breaks down for large p. • Not surprising; large p implies more completely deleted runs, so more side information released. • Want to find limits of possible global results. • The (1 – p)/9 lower bound argument seems tightest as p approaches 1. • Can we obtain an asymptotic c(1 – p) upper bound? • Build upon insights from global lower bound.

14. Upper Bound via Markers • Suppose an undeletable marker is inserted every 1/(1 – p) bits in the transmission. • Channel now memoryless. • Input symbols = 1/(1 – p) bits. • Output symbols = random subsequence, with expected length 1. • Capacity should scale with (1 – p). • Capacity bound: • How can we optimize over such a large dimensional space? • Symbols are big.

15. Upper Bound Calculation : Step 1, Output • Problem: Optimization over all output symbols. • Potentially infinitely many bit strings. • Solution: Finitize the problem. • At receiver, number of bits between markers converges to Poisson distribution. • For upper bounds, assume that if k > 6 bits received, then receiver obtained k bits of information. • Only affects bounds by a few percent. • Only need to consider outputs of 6 or fewer bits.

16. Upper Bound Calculation : Step 2, Input • Problem: Optimization over input strings. • Sequence of 1/(1 – p) bits. Potentially infinite alphabet. • Solution: Finitize the problem. • Key Lemma: if only considering up to 6 bits at output, need only consider sequences of up to 6 runs at input. • Same output distribution achieved by convex combination. • Upper bound achieved by optimizing over large number of finite-dimensional vectors representing up to 6 runs. • Heuristic/computational approach.

17. Bounds Achieved • As p goes to 1, cannot obtain capacity better than 0.7918(1 – p). • Gap between (asymptotic) upper/lower bounds now roughly (1 – p)/9 and 4(1 – p)/5. • Room for improvement, probably both sides.

18. Conclusions and Open Questions • What are the limitations of such side information arguments? • Are novel upper bound techniques required for these channels? • Is there a more purely information theoretic approach? • Can we characterize optimal input/output distributions? • Heuristic/other approaches to guide theory? • How tight can we make upper/lower bounds? • What is the right answer? • Extend upper bounds for insertion channels? • Many, many others…