CS 591 Complex Adaptive Systems Spring 2009 Measures of Complexity Melanie Moses 1/28/09

1. CS 591Complex Adaptive SystemsSpring 2009Measures of ComplexityMelanie Moses1/28/09

2. Topics • Why do we need formal definitions of complexity? • Complex Adaptive Systems • Definitions of Complexity • Flake & Gell Mann Computational Definitions of complexity • Running time complexity • Gell Mann’s Effective Complexity • Kolmogorov (or Algorithmic) complexity • Mutual Information • Logical Depth • Others • Per Bak • Stuart Kauffman • John Holland • Wolfram and Langton’s complexity of CAs

3. Why define complexity? • To estimate how long a particular system will take to solve a problem • To estimate difficulty in engineering complex systems • To understand the limits of prediction, approximation, and simulation • To answer fundamental scientific questions • Does (why does?) complexity increase through evolution? • How can complexity arise when entropy should always increase? “As evolution proceeded on the surface of the earth, there has been a progressive increase in size and complexity” J. T. Bonner Evolution of Complexity. “Evolution tends to push systems towards the edge of chaos, where complex, interesting behaviors such as life can occur “ Chris Langton “The universe, the biosphere, the econosphere have all become more complex” Stuart Kauffmann Are these statements correct? If so, how do we quantify increases in complexity over time?

4. What are Complex Adaptive Systems? • Collections of agents: • Molecules, cells, animals, planets, economic agents. • Agents interact (locally) with one another and with their environment: • No central controller. • Interaction rules may be trivial or nontrivial (i.e. nonlinear); deterministic or nondeterministic • Chemical reactions, cellular interactions, mating, buy/sell decisions. • Unanticipated properties often result from the interactions: • Immune system responses, flocks of animals, settlement patterns, earthquakes, speculative bubbles and crashes. • Agents adapt their behavior to other agents and environmental constraints: • Imitation, adaptation, learning. • System behavior evolves over time: • Rules change, unfolding of complex behavior.

5. Example Complex Adaptive Systems • Biological systems • Cells • Organs • Immune systems • Organisms • Ecosystems • Food webs • Symbioses • Colonies: ants, bees, termites, wasps; bacteria, corals… • Social/Economic Systems • Nations • Firms • Households • Cities • Universities • Technological Systems • The Internet • Peer to peer networks • Botnets • Computers? • Which is most complex?

6. Characteristics of Complex Systems • What makes a system complex? • Nonlinear interactions among components. • Multi-scale phenomena and hierarchy. • Evolution of underlying components and environments. • Is complexity inherent in the system or in our understanding of it? • Ex 1: quantum computation and factoring • Ex 2: Differential equations vs simulations to model disease spread • How to measure a system’s complexity? • By its unpredictability? • By how difficult it is to describe? • Length of most concise description. • No single model adequate to describe system---the more models that are required, the more complex the system. (Lee Segel) • The length of time before it halts or repeats itself, if ever? • Entropy? • Multiple levels of organization? • Number of interdependencies?

7. Caveat Some believe that there is no general science of complex systems: • “It’s becoming apparent that a theory of complexity in some sense of a great and transforming body of knowledge that applies to a whole range of casesmay be untenable.” Sir Robert May (2001)

8. G.W. Flake’s Definition of Complexity Flake’s scientific view • An intermediate view between Reductionism and Holism • The interactions of agents…bind one level of understanding to the next • Nature is Frugal: Occam’s Razor • CAS characterized by • Emergence, self-similarity, unpredictability, self-organizing • Collections/Multiplicity/Parallelism • Iteration, Recursion, Feedback • Adaptation, Learning, Evolution • Conversion of the sciences facilitated by computation

9. Representative Measures of Complexity • Computational complexity (Cook): • How long a program runs (or how much memory it uses). • Asymptotic. • Language complexity: • Classes of languages that can be computed (recognized) by different kinds of abstract machines. • Decidability, computability. • Information-theoretic approaches: • Algorithmic Complexity (Solomonoff, Komogorov, and Chaitin): • Length of the shortest program that can produce the phenomenon. • Mutual information (many authors) • Logical depth (Bennett) • Effective complexity (Gell-Mann and Lloyd)

10. Computational Complexity • Introduced by Steve Cook (~1970). • Asymptotic running time, and/or memory consumption of an algorithm. • Worst-case versus average-case. • Important computational complexity classes: • P problems can be solved in polynomial time on a deterministic Turing Machine • for some polynomial function p: Cnkwhere c and K are constants independent of n which is the size (length) of the input • Polynomial time algorithms P are O (p(n)) • NP (nondeterministic polynomial) problems can be verified in O(p(n)) or solved on a non-deterministic (parallel) Turing Machine: subset sum {6,-1,-4, -2, 5} • NP hard: an algorithm for solving it can be translated to solve any NP problem, e.g. traveling salesman • NP complete problems are both NP and NP hard

11. Turing Machines 1. a tape 2. a head that can r/w and move l/r 3. instruction table 4. state register tape is infinite, all else is finite and discrete

12. Topics • Why do we need formal definitions of complexity? • Complex Adaptive Systems • Definitions of Complexity • Flake & Gell Mann Computational Definitions of complexity • Running time complexity • Gell Mann’s Effective Complexity • Kolmogorov (or Algorithmic) complexity • Mutual Information • Logical Depth • Others • Per Bak • Stuart Kauffman • John Holland • Wolfram and Langton’s complexity of CAs

13. Quicksort (P) vs Traveling Salesman (NP hard) O(N2) or O(Nlog(N)) O(N!) or O(2N)

14. Computational Complexity • Drawbacks • Constants can matter • Special cases vs worst cases • Approximations may suffice • Context dependent: what machine, what is already known.. • Benefit: Allows objective comparison of how hard problems are as input size increases

15. Murray Gell-Mann: Effective Complexity Complexity is highest for things that are neither strictly regular nor strictly random Kolmogorov or Algorithmic Information Complexity is based on the length of a concise description of a set. AIC is a measure of randomness…NOT a good measure of complexity Effective complexity is the length of a concise description of a set’s regularities Regularities can be found using Mutual Information What are AIC, entropy and Mutual Information?

16. Algorithmic Complexity (AC)(also known as Kolmogorov-Chaitin complexity) • The Kolomogorov-Chaitin complexity K(x) is the length, in bits, of the smallest program that when run on a Universal Turing Machine outputs (prints) x and then halts. • Example: What is K(x) where x is the first 10 even natural numbers? Where x is the first 5 million even natural numbers? • Possible representations: • 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, … (2n - 2) • for (j = 0; j < n: j++) printf(“%d\n”, j * 2); • How many bits? • Alternative 1: O(n log n) • Alternative 2: K(x) = O(log n) • Two problems: • Calculation of K(x) depends on the machine we have available (e.g., what if we have a machine with an instruction “print the first 10 even natural numbers”?) • In general, it is an uncomputable problem to determine K(x) for arbitrary x. • How long is the program to do x?

17. Algorithmic Complexity cont. • AC formalizes what it means for a set of numbers to be compressible and incompressible. • Data that are redundant can be more easily described and have lower AC. • Data that have no clear pattern and no easy algorithmic description have high AC. • What about random numbers? If a string is random, then it possesses no regularities: • K(x) = | Print(x) | • The shortest program to produce x is to input to the computer a copy of x and say “print this.” • Implication: The more random a system, the greater its AC. • Gell Mann: Shakespeare vs monkeys • AC is related to entropy: • The entropy rate of a symbolic sequence measures the unpredictability (in bits per symbol) of the sequence. • For a sequence of n random variables, how does the entropy of the sequence grow with n?

18. Shannon Information • Shannon Entropy H to measure basic information capacity: • For a random variable X with a probability mass function p(x), the entropy of X is defined as: • Entropy is measured in bits. • H measures the average uncertainty in the random variable. • Example 1: • Consider a random variable with uniform distribution over 32 outcomes. • To identify an outcome, we need a label that takes on 32 different values, e.g., 5-bit strings.

19. Entropy is the uncertainty of a single random variable. • The entropy of a random variable is a lower bound on the average number of bits required to represent the variable. • The uncertainty (complexity) of a random variable can be extended to define the descriptive complexity of a single string. • How much entropy is in the string, or how much information is required to describe the string? • Kolmogorov (or algorithmic) complexity is the length of the shortest computer program that prints out the string. “Algorithmic information theory (AIT) is the result of putting Shannon's information theory and Turing's computability theory into a cocktail shaker and shaking vigorously. The basic idea is to measure the complexity of an object by the size in bits of the smallest program for computing it.” (G. J. Chaitin)

20. Murray Gell-Mann: Effective Complexity Effective complexity is the AIC of a set’s regularities Mutual Information can be used to determine regularities

21. Mutual Information • Measures the amount of information that one random variable contains about another random variable. • Mutual information is a measure of reduction of uncertainty due to another random variable. • That is, mutual information measures the dependence between two random variables. • It is symmetric in X and Y, and is always non-negative. • Recall: Entropy of a random variable X is H(X). • Conditional entropy of a random variable X given another random variable Y = H(X | Y). • The mutual information of two random variables X and Y is:

22. Recap: Effective Complexity • The shortest description of the regularities in an entity (and their probabilities) • Regularities are found by the mutual information • Effective complexity is NOT • AIC itself (random strings have high AIC but low complexity) • or Mutual information (0011001100110011 has high MI but low complexity) • but AIC and MI are used to determine Effective complexity • Exercise: regularities in spoken language • Give the most concise description you can of regularities in verb tenses and usage • English • Spanish • Hindi (?) • Chinese (dialect?) • Mutual Information to determine regularities? Context?

24. Other Points from Gell-Mann • Schema are representations of the regularities: a ‘package of information’ describing the regularities in a system, its environment and their interactions’ • The schemata evolve by competing to produce the most predictive description • How long does the schema take to make predictions? • Logical Depth • The Logical depth of x is the run time of the shortest program that will cause a UTM to produce x and then halt. • Must distinguish between effective complexity and logical depth • The universe has high effective complexity despite simple physical rules because of probabilities & alternative histories • Frozen accidents produce mutual information or regularities • Most of the effective complexity of the universe lies in the AIC of a description of those frozen accidents and their consequences

25. “an individual human being, owes its existence not only to the simple fundamental law of physics and the boundary condition on the early universe but also to the outcomes of an inconceivably long sequence of probabilistic events, each of which could have turned out differently”

26. Simplicity: algorithmic regularity, statistical regularity, compressibleFlake, chapter 9 AlgorithmicComplexity Randomness

27. Other Concepts of Complexity Per Bak: Sandpile model and the ‘edge of chaos’ 1. A powerlaw size distribution of extinction events: many small and few large 2. The powerlaw lifetime distribution of species and genera. 3. The powerlaw distribution of species per genus, genera per family, and so on. Stuart Kauffmann: The Adjacent possible • Autonomous agents have to live the most complex game that they can • There is a tendency for self-constructing biospheres to enlarge their workspace, the dimensionality of their adjacent possible John Holland CAS: multiple diverse agents, stimulus response rules and adaptation • Match environmental conditions at various time scales • Stimulus causes an immediate change in action (response) • Cumulative stimulus causes changes in rules (adaptation, learning, evolution)

28. Summary of Complexity Measures • Computational complexity: • How much time (how many resources) does it take to compute something? • The language/machine hierarchy: • How complex a machine is needed to compute a function? • Information-theoretic methods: • Entropy • Algorithmic complexity • Mutual information • Gell Mann’s Effective Complexity: The shortest description of the regularities • Logical depth: • Running time of the shortest program • Combines computational complexity and AIC • Asymptotic behavior of dynamical systems: • Fixed points, limit cycles, chaos. • Wolfram’s CA classification: the outcome of complex CA can not be predicted any faster than it can be simulated.

29. Neither random nor regular • Complex systems occur at a transition point between two extremes • Often the only way to predict a future state of a complex system is to simulate it • It cannot be described by a tractable formula or short program • It cannot be described statistically • Some cellular automata are complex: the results of running them are neither regular nor random • Langton’s lambda parameter and Wolfram’s CA classification schemes go from regular to chaotic, with ‘complex’ in the middle

30. Suggested References • Computational Complexity by Papadimitriou. Addison-Wesley (1994). • Elements of Information Theory by Cover and Thomas. Wiley (1991). • Kaufmann, At Home in the Universe (1996)and Investigations (2002). • Per Bak, How Nature Works: The Science of Self-Organized Criticality (1988) • Gell-Mann, The Quark and the Jaguar (1994) • Holland, Adaptation in Natural and Artificial Systems, MIT Press; 1992