1 / 20

SHANNON LECTURE LIVING INFORMATION THEORY

SHANNON LECTURE LIVING INFORMATION THEORY. Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002. PART I. INTRODUCTION. MY BIO-IT COLLABORATORS. Prof. CHIP LEVY, UVA Med - Neuroscientist and my prime bio-collaborator. Much of this talk is our joint work.

bernadetted
Download Presentation

SHANNON LECTURE LIVING INFORMATION THEORY

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SHANNON LECTURELIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002

  2. PART I INTRODUCTION

  3. MY BIO-IT COLLABORATORS • Prof. CHIP LEVY, UVA Med - Neuroscientist and my prime bio-collaborator. Much of this talk is our joint work. • James W. Mandel, MD/PhD, UVA Med – Neuropathologist. • Bioscreening: J. Mandell, P. Subrahmanya, V. Levenshtein. • Info theory: Zhen Zhang, USC; Yuzheng Ying, PhD cand. MY THANKS • To all my students and colleagues over all the years. • To my wife, Florence, to our children and to our grandchildren, all of whom are here today!

  4. THE PRIMARY GOAL OF THIS TALK IS TO PROVIDE SUPPORT FOR THE FOLLOWING CONTENTION: TO MODEL AND ANALYZE CERTAIN BIOLOGICAL PHENOMENA VIA MATHEMATICAL INFORMATION THEORY IS BOTH GOOD BIOLOGY AND GOOD INFORMATION THEORY

  5. EXAMPLES OF ‘GOOD’ INFO THEORY DOUBLING AS ‘GOOD’ BIOLOGY • Intra-organism information transmission via chemical diffusion - no time to treat this today. • Efficient techniques for cDNA library screening and monoclonal antibody generation -no time for this. • Experimental work by W. Bialek et al. establishing Shannon mutual information rates between sensory stimuli and neural responses - no time for this either. • Perception of sensory information via neural nets with feedback – only this will be treated.

  6. Multiple Choice QuizQ: SHANNON’S SHANNON LECTURE WAS DEVOTED TO: • (a) Sources and Channels • (b) Information Measures • (c) Feedback • (d) Stock Market

  7. Answer:FEEDBACK!

  8. Facts and Theories re Feedback • Feedback sometimes increases capacity. • It can strengthen performance, reduce complexity • It often enhances stability. • Feedback is HOT! Baum-Welch/HMM/BCJR; Blahut-Arimoto, Csiszar-Tusnady; Turbo decoding, message passing, belief propagation, Tanner graphs, Gallager LDPC’s); DFE (Proakis…); ... . • Orlitsky’s award-winning theory of sequential communications: You talk, I talk, You talk, I talk,… .

  9. Example of an Orlitsky-type Comm Sequence Transcript of an ACTUAL radio conversation of a US naval ship with Canadian authorities off the coast of Newfoundland. (Released by Chief of Naval Operations 10-10-95)

  10. INFORMATION THEORISTS HAVE RESTRICTED THEIR ATTENTION ALMOST EXCLUSIVELY TO INTER-ORGANISM COMMUNICATION

  11. Nature of Inter-Organism Communication Source Source Encoder Channel Encoder • Channel is “given.” • Channel behavior is independent of source statistics. • Efficient use of channel usually requires long-delay codes and complex coding operations, especially source encoding and channel decoding. • Source and user must exchange coding rules a priori. This involves sharing a common “language,” namely at least that of the message indices. Channel User Source Decoder Channel Decoder

  12. INTRA-ORGANISM COMMUNICATION • Channel are not given. They adapts their transition probabilities over eons, or over milliseconds, in response to the empirical distribution of source. • Time-varying joint source-channel coding is efficiently performed by biological subsystems of appropriate chemotopology via simple probabilistic transformations. Little if any coding occurs in the classical sense of the term in inter-organism information theory.

  13. WAIT! What about DNA? Long block code, discrete alphabet, extensive redundancy, perhaps to control against the infiltration of errors. But DNA enables two organisms to communicate; it’s designed for inter-organism communication. DNA also controls gene expression, an intra-organism process, so a comprehensive theory of intra-organism communication needs to address it. We shall not treat it today, though.

  14. ROBUST SHANNON-OPTIMAL PERFORMANCE WITHOUT CODING Ex. 1: IID Source, MSE AWGN Channel Equating R(D) to C yields the Shannon-optimum mean distortion: But this minimum possible MSE per unit variance can be achieved simply by scaling the signal to the available channel input power level and then scaling the channel output to produce the MMSE estimate!! X Y User Source + Channel

  15. ROBUST SHANNON-OPTIMAL PERFORMANCE WITHOUT CODING Ex. 2: Bern-1/2 Source, Hamming Distance BSC(p) Channel Equating R(D) to C yields the Shannon-optimum Hamming distortion: This minimum possible Hamming distortion obviously can be achieved simply by feeding the source output directly into the channel and sending the channel output directly to the user – no delay, no coding!! X Y BSC(p) User Source, Bern-1/2

  16. SHANNON OPTIMALITY IS ACHIEVED WITHOUT CODING OR DELAY IN THESE TWO EXAMPES BECAUSE: Source is matched to the channel. Source outputs are distributed over channel input space in a way that maximizes the mutual info rate between the channel input and output subject to operative constraint(s), thereby achieving capacity. Channel is matched to the source. The channel transition probability structure is optimum for the source and distortion measure; i.e., it achieves the point on their rate-distortion function at which the rate equals the channel’s capacity.

  17. I CONTEND THAT MOST BIOLOGICAL SYSTEMS HAVE EVOLVED TO BE DOUBLY MATCHED LIKE THIS. THUS, THEY HANDLE DATA OPTIMALLY WITH MINIMAL IF ANY CODING AND NEGLIGIBLE DELAY. (Information theorists recently have come to appreciate that near-optimum performance can be obtained in many situations via relatively simple probabilistic methods that employ feedback in the source encoder and channel decoder. Biology has knows this for eons.)

  18. THERE’S MORE, THOUGH. LIVING ORGANISMS ARE INGENIOUSLY ENERGY-AWARE*. THEY’RE OPTIMALLY DOUBLY MATCHED OVER A WIDE RANGE OF POWER CONSUMPTION LEVELS. THEY HAVE EVOLVED THE ABILITY TO CHANGE THEIR INTERNAL CHANNEL TRANSITION FUNCTIONS TO MEET THE INFORMATION RATE NEEDS OF THE APPLICATION AT HAND. *The brain consumes 25-50% of the total metabolic energy budget of sedentary human. (L. Sokoloff (1989), “Circulation and energy metabolism of the brain,” in Basic Neurochemistry: Molecular, Cellular and Medical Aspects, 4th ed., G. Siegel et al., Eds.)

  19. C Slope = (bits/s)/(joules/s) = bits/joule Capacity – bits/s N.B. Increasing joules/s to get more bits/s requires expending more joules/bit !! S Average power – joules/s

  20. END OF PART I

More Related