Robust Entropy Rate for Uncertain Sources: Applications to Communication and Control Systems

1. Robust Entropy Rate for Uncertain Sources: Applications to Communication and Control Systems Charalambos D. Charalambous Dept. of Electrical and Computer Engineering University of Cyprus, Cyprus. Also with the School of Information Technology and Engineering University of Ottawa, Canada. and Alireza Farhadi School of Information Technology and Engineering University of Ottawa, Canada.

2. Overview • Robust Entropy • Solution and Relations to Renyi Entropy • Examples • Uncertainty in Distribution • Uncertainty in Frequency Response • Stabilization of Communication-Control Systems

3. Applications

4. Applications: Robust Lossless Coding Theorem [1] • Source words of block length produced by a discrete uncertain memory less source with Shannon entropy can be encoded into code words of block length from a coding alphabet of size with arbitrary small probability of error if Encoder Decoder Destination Uncertain Source Uniform Source Encoder

5. Applications: Observability and Stabilizability of Networked Control Systems [2] • Definition. The uncertain source is uniform asymptotically observable in probability if there exists an encoder and decoder such that where is the joint density of is the source density uncertainty set. and are fixed and .

6. Applications: Observability and Stabilizability of Networked Control Systems [2] • Definition. The uncertain controlled source is uniform asymptotically stabilizable in probability if there exists encoder, decoder and controller such that • For and small enough, a necessary condition for uniform observability and stabilizability in probability is

7. Problem Formulation, Solution and Relations

8. Problem Formulation • Let be the space of density functions defined on , be the true unknown source density, belonging to the uncertainty set . Let also be the fixed nominal source density. • Definition. The robust entropy of observed process having density function is defined by where is the Shannon entropy and

9. Problem Formulation • Definition. In Previous definition, if represent a sequence of R.V.’s with length of source symbols produced by the uncertain source with joint density , the robust entropy rate is defined by provided the limit exits.

10. Solution to the Robust Entropy • Let the uncertain set be defined by where is the relative entropy and is fixed. • Lemma. Given a fixed nominal density and the uncertainty defined above, the solution to the robust entropy is given by where the minimizing in above is the unique solution of .

11. Solution to the Robust Entropy • Corollary [1]. Suppose , is uniform Probability Mass Function (PMF), that is When and consequently Correspond to PMF’s, that is the previous solution is reduced to

12. Solution to the Robust Entropy Rate • Corollary. Let be a sequence with length of source symbols with uncertain joint density function and let we exchange with . Then, the robust entropy rate is given by where the minimizing in above is the unique solution of .

13. Robust Entropy Rate of Uncertain Sources with Nominal Gaussian Source Density • Example. From Previous result follows that if the nominal source density is -dimensional Gaussian density function with mean and covariance where is the unique solution of the following nonlinear equation

14. Relation among Renyi, Tsallis and Shannon Entropies • Definition. For and , the Renyi entropy [3] is defined by Moreover, the Tsallis entropy [4] is defined by • We have the following relation among Shannon, Renyi and Tsallis entropies.

15. Relation between Robust Entropy and Renyi Entropy • The robust entropy found previously is related to the Renyi entropy as follows. • Let ; then Moreover,

16. Examples

17. Examples: Partially Observed Gauss Markov Nominal Source • Assumptions. Let and are Probability Density Functions (PDF’s) corresponding to a sequence with length of symbols produced by uncertain and nominal sources respectively. Let also the uncertain source and nominal source are related by

18. Example: Partially Observed Gauss Markov Nominal Source • Nominal Source. The nominal density is induced by a partially observed Gauss Markov nominal source defined by • Assumptions. unobserved process, observed process, is iid , is iid , are mutually independent is detectable is stabilizable and

19. Partially Observed Gauss Markov Nominal Source: Lemmas • Lemma. Let be a stationary Gaussian process with power spectral density . Let and assume exists. Then

20. Partially Observed Gauss Markov Nominal Source: Lemmas • Lemma. For the partially observed Gauss Markov nominal source whereis the unique positive semi-definite solution of the following Algebraic Ricatti equation

21. Partially Observed Gauss Markov Nominal Source: Robust Entropy Rate • Proposition. The robust entropy rate of an uncertain source with corresponding partially observed Gauss Markov nominal source is is the Shannon entropy rate of the nominal source, is given in previously mentioned Lemma and is the unique solution of

22. Partially Observed Gauss Markov Nominal Source: Remarks • Remark. For the scalar case with , after solving , we obtain • Remark. The case corresponds to . Letting , we have

23. Example: Partially Observed Controlled Gauss Markov Nominal Source • The nominal source is defined via a partially observed controlled Gauss Markov system given by • Assumptions. The uncertain source and nominal source are related by is stabilizing matrix, is unobserved process, is observed process, is iid are mutually independent and

24. Partially Observed Controlled Gauss Markov Nominal Source: Robust Entropy Rate • Proposition. Using Body integral formula [5] (e.g., relation between the sensivity transfer function and the unstable eigenvalues of system), the robust entropy rate of family of uncertain sources with corresponding partially observed controlled Gauss Markov nominal source is are eigenvalues of the system matrix and is the unique solution of

25. Example: Uncertain Sources in Frequency Domain • Defining Spaces. Let and be the space of scalar bounded, analytic function of . This space is equipped with the norm defined by • Assumptions. Suppose the uncertain source is obtained by passing a stationary Gaussian random process , whit known power spectral density , through an uncertain linear filter where belongs to the following additive uncertainty model

26. Uncertain Sources in Frequency Domain: Robust Entropy Rate • Results. The observed process is Gaussian random process, and the Shannon entropy rate of observed process is • Subsequently, the robust entropy rate is defined via and the solution is given by [6]

27. Applications in Stabilizability of Networked Control Systems

28. Observability and Stabilizability of Networked Control Systems • Definition. The uncertain source is uniform asymptotically observable in probability if there exists an encode and decoder such that Moreover, a controlled uncertain source is uniform asymptotically stabilizable in probability if there exists an encoder, decoder and controller such that

29. Necessary Condition for Observability and Stabilizability of Networked Control Systems [2] • Proposition. Necessary condition for uniform asymptotically observability and stabilizability in probability is is the covariance matrix of Gaussian distribution that satisfies

30. References [1] C. D. Charalambous and Farzad Rezaei, Robust Coding for Uncertain Sources, Submitted to IEEE Trans. On Information Theory, 2004. [2] C. D. Charalambous and Alireza Farhadi, Mathematical Theory for Robust Control and Communication Subject to Uncertainty, Preprint, 2005. [3] A. Renyi, “On Measures of Entropy and Information”, in Proc. 4th Berkeley Symp. Mathematical Statistics and Probability, vol. I, Berkeley, CA, pp. 547-561, 1961.

31. References [4] C. Tsallis, Possible Generalization of Boltzmann-Gibbs Statistics, Journal of Statistics Physics, vol. 52, pp. 479, 1998. [5] B. F. Wu, and E. A. Jonckheere, A Simplified Approach to Bode’s Theorem for Continuous-Time and Discrete-Time Systems, IEEE Trans on AC, vol. 37, No. 11, pp. 1797-1802, Nov 1992. [6] C. D. Charalambous and Alireza Farhadi, Robust Entropy and Robust Entropy Rate for Uncertain Sources: Applications in Networked Control Systems, Preprint, 2005.