1 / 53

Signals, Data, and Information

Representation of Information. Physical signalslight, sound, smelltransductionrepresentationPicturescapture relationships... how?Languagewide range of uses (Lucky). Storiesentertainment only?how do they relate to information?Metaphorscognitive reference pointsinference. Signals to Data.

nash
Download Presentation

Signals, Data, and Information

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

    1. Signals, Data, and Information Paul Munro (munro@sis.pitt.edu) Phone: 624-9427 SIS 752 Research area: Neural Networks Courses: Neural Networks (IS 2410) Information & Coding Theory (IS 2012)

    2. Representation of Information Physical signals light, sound, smell transduction representation Pictures capture relationships... how? Language wide range of uses (Lucky) Stories entertainment only? how do they relate to information? Metaphors cognitive reference points inference

    3. Signals to Data Physical stimuli (analog) to discrete/symbolic representation (digital) Sample continuous signals as a stream of numerical data

    4. Signals to Data Physical stimuli (analog) to discrete/symbolic representation (digital) Sample continuous signals as a stream of numerical data

    5. Signals to Data Physical stimuli (analog) to discrete/symbolic representation (digital) Sample continuous signals as a stream of numerical data

    6. Signals to Data Physical stimuli (analog) to discrete/symbolic representation (digital) Sample continuous signals as a stream of numerical data

    7. Signals to Data Physical stimuli (analog) to discrete/symbolic representation (digital) Sample continuous signals as a stream of numerical data

    8. What is a code? Representation of events by symbols “Events” are usually symbols (at another level) Two important qualities of a code Efficiency Robustness

    9. Model of the signaling system

    10. Model of the signaling system

    11. Model of the signaling system

    12. Model of the signaling system

    13. Model of the signaling system

    14. Model of the signaling system

    15. Model of the signaling system

    16. Model of the signaling system

    17. Encoding Source Alphabet A, B, C, D, ... Code Alphabet [0,1], [dot, dash], [flag positions] Codebook table that maps source symbols to codewords Numerical codes Binary, Octal, Hex, Decimal, ...

    18. A Codebook

    19. ASCII

    20. Radix r codes The radix is the number of symbols in the code alphabet For a fixed length (block) code of radix r with length n, there are rn possible symbols.

    21. Parity Checks (radix 2) M1 M2 M3 ... Mn-1 C n total bits/block n-1 message bits 1 check bit Check bit (C) equals 1 if number of 1s in message is odd 0 if number of 1s in message is even Redundancy = #total bits/#message bits r = n / (n-1)

    22. Error-Correcting Codes Use parity checks on overlapping subsets of bits Syndromes (patterns of errors) uniquely identify individual bits Rectangular Codes Triangular Codes

    23. Error Syndromes Error correction is accomplished by performing multiple simultaneous parity checks Each bit is included in a unique set of parity checks Error is identified by the syndrome, or pattern of errors

    24. Rectangular Codes Assume a single error in a block Place message bits in an m-by-n rectangle Parity checks down columns

    25. Rectangular Codes Assume a single error in a block Place message bits in an m-by-n rectangle Parity checks down columns

    26. Rectangular Code Example

    27. Rectangular Code Example

    28. Rectangular Error Correction

    29. Rectangular Error Correction

    30. Rectangular Error Correction

    31. Rectangular Error Correction

    32. Triangular Codes Assume a single error in a block Place message bits in a 1,2,3,...,n triangle (see figure below)

    33. Triangular Codes Assume a single error in a block Place message bits in a 1,2,3,...,n triangle (see figure below) Check bits computed at corners

    34. Triangular Codes Assume a single error in a block Place message bits in a 1,2,3,...,n triangle (see figure below) Check bits computed at corners

    35. Triangular Codes Assume a single error in a block Place message bits in a 1,2,3,...,n triangle (see figure below) Check bits computed at corners

    36. Triangular Codes Assume a single error in a block Place message bits in a 1,2,3,...,n triangle (see figure below) Check bits computed at corners

    37. Triangular Codes Assume a single error in a block Place message bits in a 1,2,3,...,n triangle (see figure below) Check bits computed at corners

    38. Triangular Codes Assume a single error in a block Place message bits in a 1,2,3,...,n triangle (see figure below) Check bits computed at corners

    39. Triangular Code Example

    40. Triangular Code Example

    41. Triangular Error Correction

    42. Triangular Error Correction

    43. Triangular Error Correction

    44. Triangular Error Correction

    45. Variable Length What’s the point? More frequent symbols should have shorter codes Think of symbols as random variables Formula for average length: Lavg = ??pili

    46. Example

    47. A Decoding Tree

    48. Two Decoding Trees

    49. Extensions S={a,b,c,...} S2 = {aa,ab,ac,...,ba,bb,bc,...,ca,cb,cc,...} if S has q symbols, n-th extension Sn has qn symbols Extensions example CODE: a = 0, b = 10, c = 11 S2={aa, ab, ac, ba, bb, bc, ca, cb, cc} 9 symbols could be assigned any code 2nd extension of CODE is... 00, 010, 011, 100, 1010, 1011, 110, 1110, 1111

    50. What would be the minimum average length for a code? code length <--> information content average code length <==> average information ENTROPY! H=? pi ln (1/pi) Skewed probability distributions give lower entropy and lower average length (recall example)

    51. How can the average length be less than 1 bit per symbol? consider encoding a message with just two symbols a and b, where p(a) = 2/3, and p(b)=1/3. encoding a as 1 and b as 0 ? use the second extension --> the new symbol set is aa, ab, ba, bb. probabilities are products of p(a) and p(b) p(aa) = 2/3 p(ab)=1/3 p(ba) = 2/3 p(bb)=1/3

    52. Huffman Encoding of S2

    53. Conditional probabilities of language The probability distribution is skewed by knowledge of the preceding characters For example: prob(“h” | “xylop”) The more you know, the less information you gain with each symbol... efficiency of “intelligent” (informed) communication

    54. Examples (from Silicon Dreams) Zero Order English: tsn rdjpdzfigypmawakno djwxltaqmucoxweoefi jyfpxsdawbircocrlut sfptroy efbubxiebbb First Order English: yos tmhota i n sshntletnt anootnrnoPtoeeIc kwe hehq ith t. oeitai s oclcinryctaet. risonalven atia worgumfi. wleos enn Second Order English: e obutant tnwe o Mar thionas pr is withious wid watout iofo inityrs ivasto tie w tapertibisthe te wiestime a Third Order English: Thea se thook, somly. Let ther of mory. A Romensmand codun shate sed be lat throphignis de thand sion le wentem macturt Fourth Order English: The generated job providual better trand the displayed code, abovery upondults well the coderst in thestical it do hock bothe merg. (Instates cons eration. Never any of puble and to theory. Evential callegand to elast benerated in with pies as is with

More Related