Symbolic dynamics of M arkov chains

1. Symbolic dynamics of Markov chains P S Thiagarajan School of Computing National University of Singapore Joint work with: ManindraAgrawal, S Akshay, BlaiseGenest

2. Acknowledgements • Samarjit and Javier • IAS • TÜVSÜD Foundation

3. Agenda • Study complexdynamical systems • verification of temporal logic specification • High dimensional • Continuous time; continuous value domains. • Differential equations • Hybrid automata • Networks of such dynamical systems

4. Strategy • Approximate the dynamics. • Discretize the time and value domains • Today’s first talk will have this flavor. • Sample the dynamics and do a statistical analysis. • Today’s second talk • To handle large networks: Deterministic synchronizations • Tomorrow’s talk

5. Probabilistic dynamics • Illustrate these ideas in the setting of probabilistic dynamical systems. • Yield useful approximations in the presence intractability and lack of knowledge. • Scalability can be achieved through sampling (simulations) based statistical analysis techniques. • Statistical model checking • Key application: (Networks) of biochemical networks.

6. Study I: Symbolic dynamics of Markov chains • Well-known probabilistic dynamical systems. • Rich theory • Widely applied. • Our focus (finite state, discrete time): • Symbolic dynamics • Discretize the probability value space [0, 1].

7. Symbolic dynamics of Markov chains • Analyze the symbolic dynamics via: • classical(linear) temporal logic specifications. • Probabilities are sneaked in through the atomic propositions. • Model checking methods • Details in: • LICS’2012 • Journal paper under review.

8. Two views of Markov chains • View I • a finite state probabilistic transition system. • View II • a linear transform of probability distributions over the states.

9. The transition system view 1 3 2 1 3/5 1 1 4 2/5

10. The transition system view 0 1 1 3 2 0 0 1 3/5 1 1 4 2/5

11. The transition system view 3 1 1 3 2 1 4 1 3/5 1 2/5 3/5 1 4 2 1 2/5 3 3 4 4 3/5 2/5 2/5 1

12. The transition system view • Outcomes: Infinite paths • Basic cylinder: The set of infinite paths that have a common finite prefix. • Path space: -algebra generated by the basic cylinders. • Probabilities are assigned to basic cylinders in a natural way. • Extends canonically to a probability measure over the path space.

13. The transition system view 3 1 1 1 3 2 4 1 3/5 1 2/5 3/5 1 4 2 1 2/5 1 1 B – The set of all paths that have the prefix 3 4 1 3 4 3 3 1 1 Pr(B) = 1  2/5  1  1 = 2/5 4 4 B 1

14. Measurable properties • The state 3 will be visited infinitely often. PCTL, CTL, …. Baier, C. and Katoen, J.-P. Principles of Model Checking

15. VIEW II • The Markov chain transforms (linearly) a given distribution over its nodes to a new one. • The graph of the chain is represented as a stochastic matrix • (all row sums are 1) • The Perron-Frobinius theory (for non-negative matrices) applies.

16. VIEW II • Leads to natural sub-classes: • Irreducible and aperiodic, • periodic, … • transient states, recurrent states…. • Stationary distributions….

17. VIEW II 1 3 2 1 3/5 1 1 4 2/5

18. The second view 0 0 1 3 2 1 3/5 0 1 1 1 4 2/5

19. Trajectory of distributions • Markov chain as (linear)transformer of probability distributions 1 0 3 2 1 1 3/5 1 1 4 Transition matrix M 2/5 0 0

20. Trajectory of distributions 0 0 1 1 3 2 0 • Each initial distribution generates a trajectory of distributions: 1 3/5 1 1 4 2/5 Transition matrix M

21. Trajectory of distributions 0 0 1 1 3 2 0 • Each initial distribution generates a trajectory of distributions: 1 3/5 1 1 4 2/5 Transition matrix M

22. Trajectory of distributions • Dynamics: • The set of trajectories generated from an initial set of distributions • This set can be infinite • We can ask whether all (none) of the trajectories satisfy some desired dynamical property • What we can ask in this setting is incomparable with View I specifiable properties

23. Trajectory of distributions • Eventually the probability of being in 1 is always greater than being in 3. • There is a future time at which 90% of the probability mass lies on {1, 4}.

24. Symbolic dynamics • A trajectory: • an infinite sequence of probability distributions (over a finite set of nodes). • The alphabet of probability distributions -over a finite set of nodes- is (uncountably) infinite. • Hence a trajectory is an infinite sequence over an infinite alphabet, • But exactly tracking probability distributions may be neither necessary nor possible. • Bio-chemical networks • Sensor networks…

25. Symbolic dynamics • We propose to reason about sequences of distributions in a symbolic way. • Using a finite alphabet of discretized distributions.

26. Symbolic dynamics • A classical tradition dynamical systems theory. • Jacques Hadamard(1898), Morse and Hedlund (1921), …. • Shannon, Smale… • Significant applications in data storage and transmission (via coding theory), linear algebra.

27. Symbolic dynamics • We know how to do this for: • Timed automata • In some restricted settings for hybrid automata. • For finite state Markov chains this has been done under very restricted setting and unnatural restrictions (Chada et.al 2011, Korthikantiet.all 2010.) • We shall instead give up on bisimulation-based equivalence classes: • Instead, fix a discretization of [0, 1] • With no restrictions handle all finite state Markov chains .

28. Symbolic dynamics c d F2(x) F(x) F3(x) e a x b Finite alphabet!

29. Symbolic dynamics • Each block is a letter. • Each trajectory is now recorded as a sequence of letters taken from a finite alphabet. • If each block bisimualtion equivalence class it is called a Markov partition! • Study the system dynamics in terms of these sequences. • Sophic shift sequences • Shift sequences of finite type.

30. Symbolic dynamics • We know how to get “Markov partitions”) do this for: • Timed automata • For hybrid automata, in a few settings. • For finite state Markov chains this has been done under very restricted setting and unnatural restrictions (Chada et.al 2011, Korthikantiet.all 2010.) • We do not look for bisimulations of finite index.: • Instead, we fixa discretization of [0, 1] • Handle all finite state Markov chains ; no restrictions.

31. Symbolic dynamics of Markov chains • Discretization • We partition [0,1] into a finite set of intervals, a b c

32. Symbolic dynamics of Markov chains a b c Each (probability) value is mapped to (identified with) the interval it falls in.

33. Symbolic dynamics of Markov chains 

34. Symbolic dynamics of Markov chains  = D Each distribution𝜇 maps to D, a unique n-tuple of intervals. 𝜇 ∈DmeansΓ(𝜇) = D

35. Symbolic dynamics of Markov chains • Each distribution𝜇 maps to D, a unique n-tuple of intervals. • Several distributions can map to the same D, • in fact Γ-1(D) can be infinite  = D 

36. Symbolic dynamics of Markov chains • Discretization • We partition [0,1] into a finite set of intervals, • Γ-1(D) can be empty  ? = D

37. Symbolic dynamics of Markov chains  = D  = D ø × • Discretized distribution • A tuple of intervals D is discretized distribution ( 𝒟- distribution for short ) ifΓ-1(D) ≠ ø

38. Symbolic dynamics of Markov chains • The set of discretized distributions 𝒟 is finite

39. Symbolic dynamics of Markov chains • A Trajectory of M starting from a distribution 𝜇 : • Induces a symbolic trajectory • a word over 𝒟ω • ξ 𝜇 = (Γ(𝜇) Γ(𝜇1) Γ(𝜇2) ….)

40. Symbolic dynamics of Markov chains 1 3 2 . . 0.45 0.12  1 3/5 1 1 4 2/5 0.08 0.35

41. Symbolic dynamics of Markov chains • Fix a set of initial distributions IN. • We use a 𝒟-distribution Din to specify IN. That is IN = Din This can be an infinite set . • IN = = , ……….. • We can fix the set of initial distributions in many other ways.

42. Symbolic dynamics of Markov chains • The symbolic dynamics of (M,Din ) is: • We wish to reason about this -language.

43. Symbolic dynamics of Markov chains • The discretization need not be uniform • Can be different for each dimension (node). • {[0, 1]} can be used to mask out “don’t care” nodes. • Dimension-reduction.

44. A temporal logic to reason about the symbolic dynamics • I, a discretization of [0, 1] : • < i , d > is an atomic proposition • nodeiinterval d • “In the current distribution 𝜇, 𝜇(i)falls in the interval d ”. • In the current discretized distribution D, D(i) = d • Probabilistic linear-time temporal logic LTL I

45. The verification problem • Given, • a Markov chain M, • a discretization I, • an initial set of distributions represented as Din , • and a specification φ as an LTLIformula, • Determine if M, Din ⊨ φ. In other words, • (0) ⊨  for every  in LM • Does every symbolic trajectory of M satisfy φ, i.e., • is it the case LM ⊆ L φ?

46. Example formulas • Whenever probability of nodeiis “high”, the probability of nodej is “low” : • The 𝒟 -distribution (d1, d2, . . . , dn) appears infinitely often:

47. Example formulas • Extending with FO theory of reals, we can express much more: • e.g., Infinitely often, the probability of node i is at least twice the sum of probabilities of all other nodes. • Logics based on path spaces (PCTL etc.) and logics based sequencesof probability distributions are incomparable

48. Examples • We have modeled a simple version of the Google pageranking algorithms. • We have also modeled a small pharmacokinetics system for drug delivery.

49. The verification problem • Given, M, I, Din , φ, determine if M, Din ⊨ φ, i.e., LM ⊆ L φ • If LMis ω-regular and effectively computable then we can use standard model checking techniques. • But LMis not always ω-regular !

50. The verification problem LMis not always ω-regular.