1 / 49

Exam #4 …

Exam #4 …. III. Information Theory. Computers are informational tools Information (“I”) comes from Intelligence. A. Shannon (Info) Theory. Claude Shannon (1916-2001) American computer scientist of the 20th century. Two Statements:. “It is Thursday, December 8, 2005, 1:15pm, S118B, CUW …”

kara
Download Presentation

Exam #4 …

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Exam #4 … CSC 350 - Dr. Gary Locklair

  2. III. Information Theory Computers are informational tools Information (“I”) comes from Intelligence CSC 350 - Dr. Gary Locklair

  3. A. Shannon (Info) Theory Claude Shannon (1916-2001) American computer scientist of the 20th century CSC 350 - Dr. Gary Locklair

  4. Two Statements: “It is Thursday, December 8, 2005, 1:15pm, S118B, CUW …” “Plasma beings from the planet Threa have infiltrated CUW. One of them has taken the shape ofDr. Ferry and assumed his role as president.” CSC 350 - Dr. Gary Locklair

  5. Two Statements:One is dull: “It is Thursday, December 8, 2005, 1:15pm, S118B, CUW …” The other bizarre: “Plasma beings from the planet Threa have infiltrated CUW. One of them has taken the shape of Dr. Ferry and assumed his role as president.” CSC 350 - Dr. Gary Locklair

  6. Two Statements:One doesn’t convey any (Shannon) information “It is Thursday, December 8, 2005, 1:15pm, S118B, CUW …” Why? Because its probability is high - it is indeed Thursday, etc CSC 350 - Dr. Gary Locklair

  7. Two Statements:The other conveys lots of (Shannon) information: “Plasma beings from the planet Threa have infiltrated CUW. One of them has taken the shape of Dr. Ferry and assumed his role as president.” Why? Because its probability is low. CSC 350 - Dr. Gary Locklair

  8. A. Shannon (Info) Theory deals only with syntax; it does not deal with semantics. CSC 350 - Dr. Gary Locklair

  9. A. Shannon (Info) Theory As one example, the truth or falsity of the statement isn’t considered. For the moment, we won’t care if Plasma beings have really taken over Dr. Ferry or not! :-) CSC 350 - Dr. Gary Locklair

  10. Dr. A. E. Wilder-Smith called Information the “surprise effect” CSC 350 - Dr. Gary Locklair

  11. Shannon (Info) Theory attempts to: 1. Quantify, or measure, I 2. Set theoretical limits (define what’s possible) for conveying (transmitting) I CSC 350 - Dr. Gary Locklair

  12. Shannon was concerned with transmitting I over phone lines. He wanted to reliably convey I from source to destination. In other words, we don’t want the I corrupted during transmission. CSC 350 - Dr. Gary Locklair

  13. What would corrupt the I? [student responses here :-] How might you deal with a noisy channel? [student responses here :-] Notice that all require more effort CSC 350 - Dr. Gary Locklair

  14. Shannon asked 1. Is it possible to detect and correct a corrupted message (within what limits)? 2. How can a garbled message be recovered? CSC 350 - Dr. Gary Locklair

  15. Deal with noise by 1. Recognize the problem 2. Compensate for it to begin with CSC 350 - Dr. Gary Locklair

  16. Situation I must be conveyed from A  B over a noisy channel 1. Reliability – message at B should be identical to message at A CSC 350 - Dr. Gary Locklair

  17. Situation I must be conveyed from A  B over a noisy channel 2. Maximize Rate – effective transmission time {These are the real world tradeoffs} CSC 350 - Dr. Gary Locklair

  18. We don’t want to give up channel capacity …eg, we don’t want to repeat message 5 times to ensure it arrives since there is no new I conveyed during times 2-5 CSC 350 - Dr. Gary Locklair

  19. Shannon (Info) Theory shows that we don’t have to give up rate to gain reliability however, there is a price: delay – time to recover or decode the message increases due to (perhaps) longer messages CSC 350 - Dr. Gary Locklair

  20. Actually two subjects Info Theory – what is possible Coding Theory – how to do it CSC 350 - Dr. Gary Locklair

  21. CSC 350 - Dr. Gary Locklair

  22. CSC 350 - Dr. Gary Locklair

  23. Theory (fun!) Mutual Information – I provided about event X by occurrence of event Y I(X;Y) = LOG P(X|Y) / P(X) CSC 350 - Dr. Gary Locklair

  24. Theory (fun!) Self Info – I(X) = LOG 1 / P(X) Entropy – “average” I = H(X) =  P(X) * LOG 1 / P(X) CSC 350 - Dr. Gary Locklair

  25. remember 1. don’t consider if message is T or F (yet) 2. Shannon (Info) Theory depends upon probability of the message CSC 350 - Dr. Gary Locklair

  26. Example Self Info – I(X) = LOG 1 / P(X) CSC 350 - Dr. Gary Locklair

  27. If message is certain, Info should be …? I(X) = LOG 1 / P(X) … P(X) = 1 (100%) we usually use LOG2 and unit of I is the bit LOG2 1 = 0 … makes sense, there is no “info” in a certain message CSC 350 - Dr. Gary Locklair

  28. Practical Example Using Shannon (Info) Theory along with Coding Theory to see how to efficiently (and reliably) transmit I … CSC 350 - Dr. Gary Locklair

  29. What if we know some messages are more likely than others? Ex: a weather forecaster with 4 possible forecasts: CSC 350 - Dr. Gary Locklair

  30. Ex: a weather forecaster with 4 possible forecasts: Cloudy = 50% - ½ Sunny = 25% - ¼ Rainy = 12.5% - 1/8 Tornado = 12.5% - 1/8 CSC 350 - Dr. Gary Locklair

  31. Normally, for 4 different messages we’d need at least … how many bits? 2 bits (encode 4 possibilities as 00, 01, 10, 11) CSC 350 - Dr. Gary Locklair

  32. Shannon (Info) Theory says “average info” (entropy) of this “alphabet” (set of messages) is:  P(X) * LOG 1 / P(X) sum up (probability * self I) CSC 350 - Dr. Gary Locklair

  33. (½ * LOG2 1/(½)) + (¼ * LOG2 1/(¼)) + (1/8 * LOG2 1/(1/8)) + (1/8 * LOG2 1/(1/8)) = 1/2 + 2/4+ 3/8 + 3/8 = 1¾ CSC 350 - Dr. Gary Locklair

  34. What? How can we transmit in less than 2 bits?Shannon says only 1¾ bits! On average, one message is much more likely, therefore encode it in a shorter bit string than the others. CSC 350 - Dr. Gary Locklair

  35. Huffman encoding – variable length codes Coding Theory (prefix code)  0 = cloudy10 = sunny110 = rainy111 = tornado CSC 350 - Dr. Gary Locklair

  36. Zeb/Zeke Joke Zeb and Zeke were sadly returning from an expensive fishing trip which only produced one fish. CSC 350 - Dr. Gary Locklair

  37. Zeb/Zeke Joke “The way I figure it,” said Zeke, “that lousy fish cost us $400!” “Wow,” replied Zeb, “it’s a good thing we didn’t catch more!” CSC 350 - Dr. Gary Locklair

  38. B. Information Theory Gary Locklair - American computer scientist of the 21st century Who? Information only comes from Intelligence CSC 350 - Dr. Gary Locklair

  39. Dr A. E. Wilder-Smith British scientist of the 20th century Life is matter + teleonomy (Information Content) Ultimate source of teleonomy is an omnipotent God CSC 350 - Dr. Gary Locklair

  40. Dr. Werner Gitt German computer scientist of the 21st century “Laws of Information” Information consists of syntax and semantics CSC 350 - Dr. Gary Locklair

  41. Dr. Werner Gitt “According to Shannon’s Theory any random sequence of symbols is regarded as information, without regard to its origin, nor whether it is meaningful or not.” CSC 350 - Dr. Gary Locklair

  42. Gitt’s Levels of Information 1 - Statistics (transmitted signal) 2 - Syntax (coding method) 3 - Semantics (meaning) 4 - Pragmatics (action) 5 - Apobetics (purpose) CSC 350 - Dr. Gary Locklair

  43. Information Theory is more than just statistics (Shannon) There must be an associated meaning for information to be present Example: computer program … CSC 350 - Dr. Gary Locklair

  44. In Shannon (Info) Theory the truth or falsity of the statement isn’t considered! Although the “Plasma being” message has high Shannon (Info) Content It’s a bloomin’ lie! :-) CSC 350 - Dr. Gary Locklair

  45. In Shannon (Info) Theory the truth or falsity of the statement isn’t considered!Although the “Today is …” message has low Shannon (Info) Content It’s the truth! :-) CSC 350 - Dr. Gary Locklair

  46. Information Theory impliesa sender and a receiver Sender: I have a purpose in mind, that will require some action, so I will communicate my idea using a particular code and then transmit it. CSC 350 - Dr. Gary Locklair

  47. Information Theory impliesa sender and a receiver Receiver: I received a signal, now I must decode it. I can now understand the idea and implement some action to achieve the desired result. CSC 350 - Dr. Gary Locklair

  48. This is Information “For God so loved the world that he gave his one and only Son, that whoever believes in him shall not perish but have eternal life.” CSC 350 - Dr. Gary Locklair

  49. From Computer Scientist Don Knuth CSC 350 - Dr. Gary Locklair

More Related