Extremal Problems of Information Combining

1. Extremal Problems of Information Combining • Information Combining: formulation of the problem • Mutual Information Function for the Single Parity Check Codes • More Extremal Problems of Information Combining • Solutions (with the help of Tchebysheff Systems) for the Single Parity Check Codes Alexei Ashikhmin Joint work with Yibo Jiang, Ralf Koetter, Andrew Singer

2. Channel Encoder Channel Channel Information Transmission APP Decoder Density function of the channel is not known We only know

3. Optimization Problem We assume that and that the channel is symmetric Problem 1 Among all probability distributions such that determine the probability distribution that maximizes (minimizes) the mutual information at the output of the optimal decoder

4. To variable nodes Check nodes processing Variable nodes processing From variable nodes Decoder of single parity check code Input from channel Interleaver

5. Problem is Solved Already 1. I.Land, P. Hoeher, S.Huettinger, J. Huber, 2003 2. I.Sutskover, S. Shamai, J. Ziv, 2003

6. erasure Repetition code: The Binary Erasure Channel (BEC) is the best The Binary Symmetric Channel (BSC) is the worst Single Parity Check Code: is Dual ofRepetition Code The Binary Erasure Channel (BEC) is the worst The Binary Symmetric Channel (BSC) is the best

7. Our Goals • We would like to solve the optimization problem for the Single Parity Check Codes directly (without using duality) • Get some improvements

8. Channel Soft Bits We callsoft bit, it has support on

9. erasure

10. Binary symmetric channel, Gaussian Channel:

11. Decoder Single Parity Check Code Encoder Single Parity Check Code Channel Channel Channel E.Sharon, A. Ashikhmin, S. Litsyn Results:

12. Properties of the moments Lemma • is nonnegative and nonincreasing 2. The ratio sequence is nonincreasing Lemma In the Binary Erasure Channel all moments are the same

13. Problem 2 Among all T-consistent probability distributions on [0,1] such that determine the probability distribution that maximizes (minimizes) the second moment

14. Solution of Problem 2 Theorem Among all binary-input symmetric-output channel distributions with a fixed mutual information Binary Symmetric Channel maximizes and Binary Erasure Channel minimizes the second moment Proof: We use the theory of Tchebysheff Systems

15. Lemma Binary Symmetric, Binary Erasure and an arbitrary channel with the same mutual information have the following layout of

16. Lemma Let and 1) 2) if for and for then

17. satisfy conditions of the previous lemma This is exactly our case

18. Problem 1 on extremum of mutual information and Problem 2 on extremum of the second moment are equivalent

19. Channel Extrema of MMSE It is known that the channel soft bit is the MMSE estimator fo thechannel input Theorem Among all binary-input symmetric-output channels with fixed the Binary Symmetric Channel has the minimum MMSE: and the Binary Erasure Channel has the maximum MMSE:

20. How good the bounds are

21. Decoder Single Parity Check Code Encoder Single Parity Check Code Channel Channel Channel Problem 3 1) 2) Among all T-consistent channels find that maximizes (minimizes)

22. Problem 4 Among all T-consistent probability distributions on [0,1] such that 1) 2) determine the probability distribution that maximizes (minimizes) the fourth moment

23. Theorem The distribution with mass at , mass at and mass at 0 maximizes The distribution with mass at , mass at and mass at 1 minimizes

24. Extremum densities Maximizing Minimizing:

25. Lemma Channel with minimum and maximum and an arbitrary channel with the same mutual information have the followin layout of

26. Problem 3 on extremum of mutual information and Problem 4 on extremum of the fourth moment are equivalent

27. Assume that and is the same as in AWGN channel with this

28. Tchebysheff Systems Definition A set of real continues functions is called Tchebysheff system (T-system) if for any real the linear combination has at most distinct roots at Definition A distribution is a nondecreasing, right-continues function The moment space, defined by ( is the set of valid distributions), is a closed convex cone. For define

29. Problem For a given find that maximizes (minimizes)

30. Theorem If and are T-systems, and then the extrema are attained uniquely with distrtibutions and with finitely many mass points Lower principal representation Upper principal representation

31. Channel Soft Bits We callsoft bit, it has support on Lemma (Sharon, Ashikhmin, Litsyn) If then Random variables with this property are called T-consistent

32. Find extrema of Under constrains

33. Theorem Systems and are T-systems on [0,1]. --------------------------------------------------------------------------------- the distribution that maximizes has only one mass point at : has probability mass at and at This is exactly the Binary Symmetric Channel