1 / 10

Kolmogorov Complexity

Kolmogorov Complexity. Presented by: Wei Peng. Introduction. Source: Alan Turing in 1936 of the universal Turing machine

reece-lucas
Download Presentation

Kolmogorov Complexity

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Kolmogorov Complexity Presented by: Wei Peng

  2. Introduction • Source: Alan Turing in 1936 of the universal Turing machine • Kolmogorov complexity can be measured by the length of shortest program for a universal Turing machine that correctly reproduces the observed data S KT(S) = |dT(S)| = |<T, w>|

  3. Relationship between two streams • Kolmogorov • Initiated by Kolmogorov with important later developments by Martin-Lof and Chaitin. The body of information is usually assumed to be a finite string of binary digits S. • Motivation: information theory • Solomonoff • Chronologically the first, springs from the work of Solomonoff. The intent is not to measure the complexity of S, but rather to develop a probability distribution over the set of finite binary strings. • Motivation: inductive inference and artificial intelligence

  4. Relationship between two streams Input tape: one-way, read-only Output tape: one-way, written, no over-written Input and output tapes: binary alphabet with no blank or delimiter symbol in addition to zero and one • Stream one: The turing machine is required to stop KT(S) = |dT(S)| • Stream two: Its further action is unspecified PT(S) =

  5. Definition of Information A = 0101010101010101010101010101 B = 1110010110100011101010000111 • The length of the description is the combined length of representing M and w • Definition 6.20 Let x be a binary string. The minimal description of x, written d(x), is the shortest string <M, w> where TM M on input w halts with x on its tape. If several such strings exist, select the lexicographically first among them. The descriptive complexity (Kolmogorov complexity) of x, written K(x), is K(x) = |d(x)|

  6. Some proofs of Kolmogorov complexity Proof: M is a Turing machine that halts as soon as it is started. This machine computes the identity function – its output is the same as its input. A description of x is simply <M>x. Letting c be the length of <M> completes the proof. Proof: M=“On input <N, w> where N is a TM and w is a string: 1. Run N on w until it halts and produces an output string s. 2. Output the string ss.” K(xx)= |<M>| + |d(x)| = c + K(x)

  7. Some proofs of Kolmogorov complexity Proof: We construct a TM M that breaks its input w into two separate descriptions. The bits of the first description d(x) are all doubled and terminated with string 01 before the second description d(y) appears. Once both descriptions are obtained, they are run to obtain the string X and y and the output xy is produced. 2log(K(x)) + K(x) + K(y) +c

  8. Incompressible strings and randomness • Let x be a string. Say that x is c-compressible if If x is not c-compressible, say that x is incompressible by c. If x is incompressible by 1, say that x is incompressible. • Incompressible strings of every length exist. Proof: The number of binary strings of length n is Each description is a nonempty binary string, so the number of descriptions of length less than n is at most the sum of the number of strings of each length up to n-1, or

  9. Incompressible strings and randomness • For some constant b, for every string x, the minimal description d(x) of x is incompressible by b. Proof: consider the following TM M: M = “On input <R, y>, where R is a TM and y is a string: 1. Run R on y and reject if its output is not of the form <S, z>. 2. Run S on z and halt with its output on the tape.” Let b be |<M>|+1. We show that b satisfies the theorem. Suppose to the contrary that d(x) is b-compressible for some string x. Then But then <M>d(d(x)) is a description of x whose length is at most This description of x is shorter that d(x), contradicting the latter’s minimality.

  10. Randomness • Random • Complexity theory: A string S is random for a UTM T if KT(S) ≥ |S|-c, where c is a small constant chosen to impose a ‘significance’ requirement on the concept of non-randomness • Work on applying the algorithmic theory of randomness to practical problems of computer learning is under way in the Computer Learning Research Center at Royal Holloway, University of London. The detail information can be accessed at http://www.clrc.rhbnc.ac.uk

More Related