Complexity

1. Complexity

2. organized complexity • From systems science to informatics • organized complexity • study of organization • whole is more than sum of parts • Systemhood properties • Holism vs. reductionism • Need for new mathematical and computational tools • Massive combinatorial searches • Problems that can only be tackled with computers • Computer as lab • Understanding function • Of wholes • Systems biology • Evolutionary thinking • Systems thinking • Emergence • How do elements combine to form new unities? Disorganized complexity Organized Complexity Randomness Organized simplicity Complexity

3. What is complexity? • Dictionary • Having many varied or interrelated parts, patterns or elements • Quantity of parts and extent of interrelations • Subjective or epistemic connotation • Ability to understand or cope • Complexity is in the eyes of the observer • Brain to a neuroscientist and to a butcher • Quantity of information required to describe a system

4. complexity and information Descriptive complexity: • Proportional to the amount of information required to describe the system • In a syntactic way • Measure number of entities (variables, states, components) and variety relationships among them • General requirements for indicators: • Nonnegative quantity • If system A is a homomorphic image of B, then the complexity of A should not be greater than B • If A and B are isomorphic, then their complexity should be the same • If system C consists of two non-interacting subsystems B and neither is a homomorphic image of the other, then the complexity of C should be equal to the sum of the complexities of A and B Size of shortest description or program in a standardlanguage or universal computer • Applicable to any system • Difficult to determine shortest description • A.K.A. Kolmogorov complexity

5. Uncertaintyy-based complexity and information • Proportional to the amount of information needed to resolve any uncertainty with the system involved • In a syntactic way • Related to number of alternatives left undecided to characterize a particular element • Examples • Hartley Measure • Shannon Entropy

6. Trade-off between descriptive and uncertainty-based complexity: • When one is reduced, the other is likely to increase • Trade certainty for acceptable descriptive complexity • Models of phenomena in the realm of organized complexity require large descriptive complexity • But to be manageable, we must simplify by accepting larger uncertainty (and smaller descriptive complexity)

7. Computational complexity of algorithms • Descriptive and uncertainty-based complexity pertain to systems • Characterized by information • Computational complexity pertains to systems problems • Characterization of the time or space (memory) requirements for solving a problem by a particular algorithm

8. Types of Problems • Algorithms are for Solving Problems • Types of Problems • Search • Find an X in input satisfying property Y • Find a prime number in a random sequence of numbers • Structuring Problems • Transform the input to satisfy property Y • Sort a random sequence of numbers • Construction Problems • Build an X satisfying property Y • Generate a random sequence of numbers with a given mean and standard deviation • Optimization Problems • Find the best X satisfying property Y • Find the largest prime number in a given sequence • Decision Problems • Decide whether the input satisfies property Y • Is the input number a prime number • Adaptive Problems • Maintain property Y over time • Grow a sequence of numbers such that there are always m prime numbers with a given mean and standard deviation

9. Artificial Intelligence Algorithmic Complexity Theory Computability Theory Problem Difficulty • Conceptually Hard Problem • No algorithm exists to solve the problem • Analytically Hard Problem • An algorithm exists to solve the problem, but we don’t know how long it will take to solve every instance of the problem • Computationally Hard Problem • An algorithm exists to solve the problem, but relatively few instances take millions of years to solve • Problems we know to be • problems we suspect to be • Computationally unsolvable Problem • No algorithm can exist to solve the problem

10. Hanoi Problem • Invented by French Mathematician Édouard Lucas in 1883 • At the Tower of Brahma in India, there are three diamond pegs and sixty-four gold disks. When the temple priests have moved all the disks, one at a time preserving size order, to another peg the world will come to an end. • If the priests can move a disk from one peg to another in one second, how long does the World have to exist?

11. 0 3 4 7 Solving the Hanoi Problem • Solve for the smallest instances and then try to generalize • N=2 • N=3 Use Hanoi_2 (H2) as building block (of 3 moves) H3 uses H2 twice, plus 1 move of the largest disk

12. Hanoi Problem for n disks Use Hanoi_2 (H2) as building block (of 3 moves) H3 uses H2 twice, plus 1 move of the largest disk • Algorithm to move n disks from A to C • Move top n-1 disks from A to B • Move biggest disk to C • Move n-1 disks on B to C • Recursion • Until H2 An Algorithm that uses itself to solve a problem

13. Temp End Start Pseudocode for Hanoi Problem • Hanoi (Start, Temp, End, n) • If n = 1 then • Move Start’s top disk to End • Else • Hanoi (Start, End, Temp, n-1) • Move Start’s top disk to End • Hanoi (Temp, Start, End, n-1)

14. Computational Complexity "FLOPS" (FLoating Point Operations Per Second) • Resources required during computation of an algorithm to solve a given problem • Time • how many steps does it take to solve a problem? • Space • how much memory does it take to solve a problem? • The Hanoi Towers Problem • f(n)is the number of times the HANOI algorithm moves a disk for a problem of n disks • f(1)=1, f(2)=3, f(3)=7 • f(n)= f(n-1) + 1 + f(n-1) = 2 f(n-1) + 1 • Every time we add a disk, the time to compute is at least double • f(n) = 2n - 1 585 billion years in seconds!!!!!!!! Earth: 5 billion years Universe: 15 billion years Fastest Computer: 1petaflops - 1015(aprox250 instructions a second) 214s needed = 4.6 hours

15. Fastest Computers IBM Roadrunner MDGRAPE-3: Not a general-purpose computer IBM BlueGene/L Fastest general-purpose computer (may 2008): 1 petaflop !!! – 1 quadrillion calculations per second --- Roadrunner @ Los Alamos--- aprox 214 s needed = 4.6 hours for Hanoi problem (assuming one disk change per operation) Fastest Computer (june 2006): 1 petaflop !!! – 1 quadrillion calculations per second --- MDGRAPE-3 @ Riken, Japan Fastest Computer (late 2005): 280.6 teraflops - 280.6 trillion calculations a second --- Approaching petaflops: 3 petaflops in late 2006???? Fastest Computer (early 2005): 135.5 teraflops - 135.5 trillion calculations a second --- Approaching petaflops: 250

16. Bremermann's Limit • Physical Limit of Computation • Hans Bremmermann in 1962 • “no data processing system, whether artificial or living, can process more than 2 × 1047 bits per second per gram of its mass.” • Based on the idea that information could be stored in the energy levels of matter • Calculated using Heisenberg's uncertainty principle, the Hartley measure, Planck's constant, and Einstein's famous E = mc2 formula • A computer with the mass of the entire Earth and a time period equal to the estimated age of the Earth • would not be able to process more than about 1093 bits = transcomputationalproblems

17. Transcomputational Problems • A system of n variables, each of which can take k different states • kn possible system states • When is it larger than 1093? • Pattern Recognition • Grid of n = q2 squares of k colors • Blackbox: 10100 possible states! • The human retina contains a million light-sensitive cells • Large scale integrated digital circuits • K= 2 (bits): a circuit with 308 inputs and one output! • Complex Problems need simplification!

18. What happens to Moore’s law ?

19. Is the singularity near? • Ray Kurtzweil, VernorVinge • Technological progress grows exponentially and reaches infinity in finite time • Clay Shirky’s gin analogy http://www.shirky.com/herecomeseverybody/2008/04/looking-for-the-mouse.html Also, noteworthy: metasystem transition (Turchin/heylighen)

20. computational complexity (algorithms) • Computational complexity pertains to systems problems • Characterization of the time or space (memory) requirements for solving a problem by a particular algorithm • Time complexity function • f(n) is the largest amount of time needed for an algorithm to solve a problem instance of size n. • Polynomial time algorithms • Complexity :O(nk) • can be computed in a polynomial time function of order k. • Efficient or tractable algorithms • P-Problems • Exponential time algorithms • Exponential time function, e.g. O(2n), O(10n), O(nn), O(n!) • Inefficient or intractable algorithms

21. Computational complexity Growth rates for time complexity functions Assuming a million operations per second

22. Computational complexity Effect of increasing computing speed

23. computational complexity (algorithms) • NP-Problems • Solved in polynomial time by a nondeterministic Turing machine • Can have more than one rule applied to the same situation (or input state and symbol read) • Solved by deterministic turing machine in O(2p(n)) • NP contains P • NP-Complete problems • Every other NP-Problem can be converted to them in polynomial time