Causal Models as Minimal Descriptions of Multivariate Systems

1. Causal Models as Minimal Descriptions of Multivariate Systems Jan Lemeire June 15th 2006 Causality & MDL

2. What can be learnt about the world from observations? • We have to look for regularities • & model them Causality & MDL

3. MDL-approach to Learning • Occam’s Razor “Among equivalent models choose the simplest one.” • Minimum Description Length (MDL) “Select model that describes data with minimal #bits.” model = shortest program that outputs data length of program = Kolmogorov Complexity Learning = finding regularities = compression Causality & MDL

4. Randomness vs. Regularity • 0110001101011010101 random string=incompressible=maximal information • 010101010101010101 regularity of repetitionallows compression Separation by the Two-part code Causality & MDL

5. Model of Multivariate Systems • Variables • Experimental data Probabilistic model of joint distribution with minimal description length? Causality & MDL

6. 1 variable • Average code length = Shannon entropy of P(x) • Multiple variables • With help of other, P(E| A…D) (CPD) • Factorization • Mutual information decreases entropy of variable Causality & MDL

7. I. Conditional Independencies • Reduction of factorization complexity • Bayesian Network Ordering 1 Ordering 2 Causality & MDL

8. II. Faithfulness Joint Distribution Directed Acyclic Graph Conditional independencies  d-separation Theorem: if a faithful graph exists, it is the minimal factorization. Causality & MDL

9. III. Causal Interpretation • Definition through interventions Causality & MDL

10. Reductionism • Causality = reductionism • Canonical representation: unique, minimal, independent • Building block = P(Xi|parentsi) • Whole theory is based on modularity like asymmetry of causality • Intervention • = change of block Causality & MDL

11. Ultimate motivation for causality Model = canonical representation able to explain all regularities • close to reality Reality Learnt Example taken from Spirtes, Glymour and Scheines 1993, Fig. 3-23 Causality & MDL

12. Causal model is MDL of joint distribution if Incompressible Incompressible (random distribution) Causality & MDL

13. A Bayesian network with unrelated, random CPDs is faithful • d-separation tells what we can expect from a causal model • Eg. D depends on C, unless a dependency in P(D|C,E) P(d1|c0,e0).P(e0)+ P(d1|c0,e1).P(e1) = P(d1|c1,e0).P(e0)+ P(d1|c1,e1).P(e1) Causality & MDL

14. When do causal models become incorrect? • Other regularities! Causality & MDL

15. A. Lower-level regularities • Compression of the distributions Causality & MDL

16. B. Better description form • Pattern • in figure random patterns -> distribution Causal model?? • Other models are better • Why? Complete symmetry among the variables Causality & MDL

17. C. Interference with independencies X and Y independent by cancellation of X→U → Y and X → V → Y • dependency of both paths • = regularity Causality & MDL

18. Violation of weak transitivity condition One of the necessary conditions for faithfulness Causality & MDL

19. Deterministic relations • Y=f(X1, X2) • Y becomes (unexpectedly) independent from Z conditioned on X1 and X2 • ~ violation of the intersection condition Solution: augmented model - add regularity to model - adapt inference algorithms • Learning algorithm: • variables possibly contain equivalent information about another • Choose simplest relation Causality & MDL

20. Conclusions • Interpretation of causality by the regularities • Canonical, faithful representation • ‘Describe all regularities’ • Causality is just one type of regularity? • Occam’s Razor works • Choice of simplest model • models close to ‘reality’ • but what is reality? • Atomic description of regularities that we observe? Papers, references and demos: http://parallel.vub.ac.be Causality & MDL