EME: Information Theoretic views of emergence and self-organisation

1. EME: Information Theoretic views of emergence and self-organisation Continuing the search for useful definitions of emergence and self organisation

2. The plot so far … • Researchers agree on just two things • We need a consistent definition of emergence • We don’t have one • Statistical complexity and mutual information help us to explore definitions in information theoretic terms • So long as we can extract a Shannon-compliant representation of information in a system • So, now we can give formal definitions of emergence and self-organisation • And to explore the relationships among them…

3. Entropy: recap and use • Joint entropy • Measure of information in joint systems • Conditional entropy (mutual information) • Measure of information in a system relative to that in another • Evolution increases mutual information between a system and its environment • Entropy can be compared • between systems, or, for one system: • over space, or • over time • Entropy measures to find how much emergent information is a direct consequence of low-level information • If we can encode the information appropriately • But that does not help to define emergence

4. Phase transitions: recap and use • System behaviour is most complex at phase transitions • Many emergent and self-organising phenomena are associated with complexity • First order transitions generate latent heat, entropy, and complex behaviour • Turbulence and related behaviours • Second order transitions are not abrupt changes in entropy • But important in identifying complex behaviour (Ising model) • Phase transitions are related to emergence • but do not help define it

5. Recognising emergence and self-organisation • Emergence is essentially about the appearance of patterns at larger/higher scales than components • Self-organisation is essentially about the “spontaneous” appearance of structure over time • To define emergence and self-organisation, we need to identify pattern or structure, and compare it across space or time • Complexity measures can be used in their definition • So long as the measures distinguish complexity from randomness

6. Emergence and complexity • Emergence and self-organisation can be defined using statistical complexity • Used by Crutchfield (1994), and subsequently by Shalizi (2001) • Based on the ability of a measurable system (e.g., an ε-machine) to reproduce the statistical information characteristics of an actual system • Essentially, looking for a robust way to identify and measure structure, or pattern Crutchfield, The Calculi of Emergence, Physica D, 75, 1994 Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf Prokopenko et al, An information-theoretic primer…, Advances in Complex Systems, 2006

7. Argument underlying definitions • Exact models of the structure inherent in systems are uncomputable in general, and uninformative • Needs a UTM, or a simulated universe, and precise measurement… • Much low-level detail is unnecessary to approximate higher-level behaviour • As in thermodynamics • Approximations improve when statistical analysis of repeated measurement and recalculation is used • Best representation of average properties • “Given the ubiquity of noise in nature, this is a small price to pay” Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

8. Shalizi’s definition of emergence A derived process is emergent if it has a greater predictive efficiency than the process it derives from • Predictive efficiency e relates excess entropy,E and statistical complexity, Cμ : e = E / Cμ • Cμ is amount of memory of past stored in a process or system • E is mutual information between system’s past and future • amount of apparent information about past stored in observed behaviour • E ≤ Cμ ,so 0≤ e < 1 • e = the fraction of historical memory stored in the process which does “useful work” in telling us about the future • Cμ = 0if no complexity, so no predictive interest, so set e = 0 • perfect predictive efficiency “unlikely” Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

9. Intuitive interpretation of Shalizi’s definition A derived process is emergent if it has a greater predictive efficiency than the process it derives from • For process X that emerges from process Y , ex > ey • e = E / Cμ • So an emergent system has • either lower statistical complexity • or greater excess entropy • Compared to system from which it emerges, an emergent system: • either has fewer irreducible complex or random components • or its past determines its future more completely

10. A informal check of Shalizi’s definition • GoL glider vs CA with GoL rules • Glider’s past predicts its future absolutely • CA rule prediction depends on attractor space • Slime mould: Dicty slug vs Dicty amoebae • Slug moves to favourable site and fruits • Amoeba may stay alive, become spore/cyst on its own, may become a slug pre-stalk or a slug pre-spore • In these simple emergent behaviours predictive efficiency seems greater than the underlying system • … but what about all the others? • Exhaustive studies are not yet being done

11. Excess entropy calculation • How do we calculate predictive efficiency, e = E / Cμ? • Excess entropy (E) is mutual information between a system’s pasts and future • A system cannot have more mutual information than that in either the past or future states • System’s pasts are represented by causal states • Equivalence class of input states that all have same conditional probability distribution of outputs Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

12. Intuition for Causal States • Causal states are produced by the modelling process • They are observations of the “state machine”, not the state machine itself • Recreates a minimal model with equivalent statistical behaviour • Uses a series of spatial or temporal measurements • Causal states are not states of the actual system • Recall lecture 12: logistic process • 47 deduced causal states Crutchfield, The Calculi of Emergence, Physica D, 75, 1994

13. ε-machines and causal states • From Crutchfield, ε-machines used to extract causal states from discrete time series • Discrete measurements are approximate indicators of a hidden environment with finite accuracy, ε • An ε-machinedetects causal states by identifying pasts that predict the future • Based on computation theory and prediction of bit-strings • Various algorithms for reconstructing an ε-machine • Shalizi and Crutchfield give example calculations Crutchfield, The Calculi of Emergence, Physica D, 75, 1994 Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

14. ε-machine: optimal model of complexity • We cannot measure complexity directly • An ε-machine approximates the system’s information processing • An ε-machine is the smallest possible explanatory model • Ockham’s razor – include only what is needed • Maximal accounting for structure • Basic tenet of science – obtain prediction of nature • Achieve appropriate balance between order and randomness • Compromise between: • Smallest model with huge error ε and little prediction • Model with minimal error that differs minimally from system Crutchfield, The Calculi of Emergence, Physica D, 75, 1994

15. Can we “calculate” emergence? • It is hard to calculate causal states in general • In Markovian behaviour (e.g. thermodynamics) any pre-state is a causal state • Standard entropy formulae in thermodynamics and statistical mechanics allow calculation of predictive efficiency • Not surprisingly, the predictive efficiency of macro-scale is considerably better than that of the micro-scale • 1020 independent gas particles with almost no predictive power give rise to predictable emergent behaviour expressible in a small number of macro-variables • So, by this measure, thermodynamics is emergent Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

16. A note on calculations for thermodynamics • Shalizi’s predictive efficiency calculation uses recognised facts and formulae of thermodynamics and statistical mechanics • Many assumptions about quantities, accuracy, etc. • e.g. assumes macro-measurement error factor <10-15 • Finds sub-nano-second predictive efficiency of micro-scale is high, but rapidly reduces over time • Most information in statistical mechanics is irrelevant to thermodynamic macro-state • But the cumulated assumptions and errors make the figures at best debatable

17. Emergent structures and ε-machines • An ε-machine summarises the dynamics of a process • The ε-machine could be divided in to sub-machines and transitions among them • At each time step, causal and previous state may or may not be in same sub-machine • If successive states are in one sub-machine, this is an emergent process of the process approximated by the ε-machine • Because knowing sub-machine reduces statistical complexity Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

18. Shalizi’s definition of self-organisation • An increase in statistical complexity is a necessary condition for self-organisation • A more realistic definition, relying only on statistical complexity, not excess entropy, causal states, etc. • Successive states in different sub-machines of an ε-machine • Optimal prediction requires more information • There are more irreducible complex or random components • For non-stationary processes, distribution of causal states changes over time • Statistical complexity is measured as a function of time, Cμ(t) • A system that spontaneously moves from uniform to periodic behaviour exhibits an increase in Cμ(t) Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

19. Thermodynamics: a box of gas • In the micro-state, all particles are independent, and all behaviours equally likely • What happens in one time or spatial unit has no later effect • Statistical complexity remains constant, and low • Thermodynamic micro-process is not self-organising • Note that, at least in this case, the definitions allow emergence without self-organisation • But thermodynamics is perhaps an extreme case Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

20. Summarising the definitions • Emergence • Informally: behaviour observed at one scale is not apparent at other scales • Formally: processes have better predictive efficiency than those from which they emerge • Lower statistical complexity or greater explanatory power of the past • Self-organisation • Informally: structures that emerge without systematic external stimuli • Formally: processes with an increase in statistical complexity over time

21. Intuitions on flocking • Individual birds are probably not independent, but follow simple local rules • For a collection of birds, statistical complexity > 0 but no where near 1 • If a bird could see the whole flock it would see complex dynamics • For a flock of birds, statistical complexity is closer to 1 • The flock is aself-organised collection of birds • When we recognise self-organisation we label it as an emergent behaviour at a higher scale • “Flock” denotes an emergent pattern of behaviour • Predictive efficiency of flock is better than that of a group of birds

22. Reconciling the definitions • Natural systems can self-organise • Independent of observation • Systems that self-organise are studied by “cognitively-limited observers” • Seeking descriptions that have good predictive ability • Patterns are at a more abstract level than self-organising system elements • An abstract description with enhanced prediction becomes an emergent process of the original behaviour

23. So does self-organisation imply emergence? • Shalizi states that he knows of no reason against self-organising non-emergent systems, but … • Emergence may be a precondition of detectable self-organisation • In practice, when humans recognise self-organisation, they identify the abstract result at an emergent process • At least some of what humans call “noise” may be unrecognisable complex self-organisation • We know that chaos has complex structure … Shalizi, May 2001, http://cse.ucdavis.edu/~cmg/compmech/pubs/CRS-thesis.pdf

24. Self-organisation and emergence Not emergent Emergent Not self- organising Thermodynamics Not interesting Possibly very complex self-organisation EMER interest: Biology Interesting physics … Self- organising Improved modeling and understanding

25. So where are we now? • Statistical complexity gives us a nice definition of self-organisation • And an intuition for how self-organisation and emergence are related • Statistical complexity gives us a possible definition of emergence • In reality, predictive efficiency is hard to estimate with any confidence • The definition does clarify features of emergence • The definitions make assumptions about measurement and discrete spatial or temporal series

26. A note on discrete measurement • All work on information theoretic definitions of emergence and self-organisation assumes discrete temporal or spatial observation • Shalizi discusses using causal states with continuous trajectories: • No current mathematics of continuous conditional probability • Continuous entropy exists but depends on co-ordinates • Entropy changes if e.g., distance is measured in inches or metres • Reconstruction from data is hard, and seriously affected by measurement problems • Not unlike problems of using PDEs for explanatory models of biological systems

27. Open questions Observers and intrinsic emergence: • If emergence is a precondition of detectable self-organisation, then it must be possible to observe the system • In intrinsic emergence, the observer is a sub-process of the system • monitors environment through sensors to construct an imperfect behavioural model • Observer makes predictions of future • which, because internal, can interfere with that future • But, are intrinsic observation and self-organisation compatible? • does observer have to be outside a self-organising system?

28. Open questions How to determine agency in self-organisation: • Organisation assumes that structure emerges over time • Self-organisation assumes that there is no consistent external agency in the appearance of structure • But, how can we distinguish self-organisation from organisation by external agency? • A complex system in a complex environment, where external inputs may be stochastic, but might also be very complex • Biological systems are usually in this category

29. And one last open question • Informally, research has identified level, scope and resolution as relevant to emergence • Statistical complexity and mutual information assume that processes or systems are well-defined • Statistical analysis (e.g., for causal states or PDEs) also assumes that scope is known • What happens if the chosen bounds exclude a key component of the emergence or self-organisation? • What happens if slightly widening the bounds would reveal that emergence or self-organisation was an artifice of the scope? • We have raised as many questions as we have solved