1 / 41

Discrete time Markov Chain

Discrete time Markov Chain. G.U. Hwang Next Generation Communication Networks Lab. Department of Mathematical Sciences KAIST. Definition.

kayo
Download Presentation

Discrete time Markov Chain

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Discrete time Markov Chain G.U. Hwang Next Generation Communication Networks Lab. Department of Mathematical Sciences KAIST

  2. Definition • The sequence of R.V.s X0, X1, X2,  with a countable state space S is said to be a discrete time Markov chain (DTMC) if it satisfies the Markov Property: for any ik2 S, k=0,1,,n-1 and i, j 2 S. • Time homogeneous DTMC : P{Xn+1 = j | Xn = i} is independent of n. Next Generation Communication NetworksLab.

  3. Transition Probability Matrices • One step transition probability matrix P • P = (pij) where pij = P{Xn+1 = j | Xn = i } • The matrix P is nonnegative and stochastic, i.e., pij¸ 0 and j2 S pij = 1 • n step transition probability matrixP(n) = (p(n)ij) • pij(n) = P{Xn = j | X0 = i} Next Generation Communication NetworksLab.

  4. For a DTMC, the initial distribution and the matrix P uniquely determine the future behavior of the DTMC because Next Generation Communication NetworksLab.

  5. Example: A general random walk • Let Xi be i.i.d. R.V.s with P{X1 = j} = aj, j=0,1,. • Let S0 = 0, Sn = k=1n Xk. Then {Sn, n¸ 1} is a DTMC because Next Generation Communication NetworksLab.

  6. Chapman - Kolmogorov's equation • Chapman - Kolmogorov's theorem pij(n+m) = k2 S pik(n) pkj(m) proof: Next Generation Communication NetworksLab.

  7. Using the chapman-Kolmogorov’s theorem we get i.e., the n-th power of the one step transition matrix P is, in fact, the n step transition matrix. Next Generation Communication NetworksLab.

  8. Example • Find the distribution of X4 where {Xn} (Xn2 S = {1,2}) forms a DTMC with initial distribution P{X0 = 1}=1 and one step transition probability P as follows: sol: Next Generation Communication NetworksLab.

  9. Analysis of a DTMC • When a communication system can be modeled by a DTMC with P and S = {0,1,2}, what happens? A sample path of a DTMC transient period stationary period Next Generation Communication NetworksLab.

  10. The stationary probabilities • The state space S ={0,1, } • The stationary probability vector (distribution)  • a row vector  = (0, 1, ) is called a stationary probability vector of a DTMC with transition matrix P if it satisfies Next Generation Communication NetworksLab.

  11. Does the stationary distribution always exist? 0 1 2 Next Generation Communication NetworksLab.

  12. Does the stationary distribution always exist? 0 1 2 3 ….. Next Generation Communication NetworksLab.

  13. The key question: • When does the stationary vector exist? • to answer the question, we need to classify DTMCs according to its probabilistic properties as • irreducibility • recurrence • positive recurrence and null recurrence • periodic and aperiodic Next Generation Communication NetworksLab.

  14. Irreducible DTMC • state i can reach state j if there exists n¸ 0 s.t. pij(n) > 0. • In this case we write i ! j. • If i! j and j! i, then we say i and j communicate and write i $ j. i k h j ….. g f ….. Next Generation Communication NetworksLab.

  15. $ is an equivalent relation, that is, • i $ i • i $ j iff j $ i • if i $ j and j $ k, then i $ k • Definition of irreducibility • A DTMC is irreducible if its state space consists of a single equivalent class, i.e., for any i, j 2 S we have i $ j. Next Generation Communication NetworksLab.

  16. A closed set • a set A of states is closed if no one step transition is possible from any state in A to any state in AC, i.e., for every pair of states i 2 A and j 2 AC, pij = 0 • An absorbing state • A single state which alone form a closed set is called an absorbing state • if state i is an absorbing state, pii = 1. Next Generation Communication NetworksLab.

  17. Recurrence • The hitting time (i) of state i: • For a state i 2 S, (i) = inf {n¸ 1| Xn = i}, i.e., (i) is the first visiting time of the DTMC {Xn} to state i. • When no such n exists, (i) = 1 by convention. • The number Ni of visits to state i: • Ni = n=11I{Xn=i} where IA is an indicator function which is defined by 1 if the event A occurs and by 0 otherwise. • Clearly, {Ni > 0} = {(i) < 1}. Next Generation Communication NetworksLab.

  18. Probability mass function of (i) • Define fji (n) = P{ (i)=n | X0 = j } • Then fji = P{the DTMC ever visits state i | X0 = j } = n = 1 1 fji (n) • Definition of recurrence of state i • state i is said to be recurrent if P{(i) < 1 |X0 = i} = 1, and transient otherwise. • that is, state i is recurrent if fii =1, and transient if fii <1. Next Generation Communication NetworksLab.

  19. Observations • Once the DTMC revisits a recurrent state i starting from state i, by the Markov property it has the same probabilistic behavior as before. Hence, state i is visited infinitely. • Further, for a recurrent state i and X0 = i a.s. successive visits to state i can be viewed as renewals and {fii (n) | n¸ 1} is the p.m.f. of the inter-renewal times. Next Generation Communication NetworksLab.

  20. For a transient state i, starting from state i the DTMC revisits state i with probability fii (< 1) and it never enters state i with probability 1- fii. Therefore, by Markov property we have P{the DTMC visits state i n times} = (fii)n (1- fii ), n¸ 1 i.e., the number of visits to state i is according to a geometric distribution. Next Generation Communication NetworksLab.

  21. The following are equivalent: • state i is recurrent • Ni = 1 with probability 1 provided that X0 = i • E[Ni|X0 = i] = E[n=11 I{Xn = i}|X0 = i] = n=11 pii(n) = 1 • The following are equivalent: • state i is transient • Ni < 1 with probability 1 provided that X0 = i • E[Ni|X0 = i] = n=11 pii(n) < 1 Next Generation Communication NetworksLab.

  22. If i $ j and i is recurrent, then j is also recurrent. Proof: Since i $ j, there exist n, m ¸ 0 s.t. pij(n) > 0 and pji(m) > 0. Then, which comes from the recurrence of state i. This completes the proof. Next Generation Communication NetworksLab.

  23. Positive recurrence • For a recurrent state i, • if E[(i) | X0 = i] < 1, state i is called positive recurrent. • if E[(i) | X0 = i] = 1, then state i is called null recurrent. • Note that, E[(i) | X0 = i] = 1 for a transient state i. Next Generation Communication NetworksLab.

  24. A stationary measure • a vector q = (q0, q1,) is called a stationary measure of a M.C. with transition matrix P if • q 0 • all qi are finite, i.e., qi < 1 • q P = q Next Generation Communication NetworksLab.

  25. Let i2 S be a recurrent state. Then a stationary measure (q0, q1,) can be defined by Next Generation Communication NetworksLab.

  26. The stationary vector q defined above depends on the chosen state i. • However, it can be shown that the stationary measure (q0, q1,) is unique up to a constant multiplication. Next Generation Communication NetworksLab.

  27. Note that • So, if state i is positive recurrent, by normalizing the stationary measure (q0, q1,) , we have a stationary distribution (p0, p1,) for the DTMC {Xn} Next Generation Communication NetworksLab.

  28. The stationary distribution • The existence of the stationary distribution • If the DTMC is irreducible and positive recurrent, the stationary distribution exists and is given by • Note that the above equation is not used in numerical computation. In fact, we use • For numerical algorithms, we will see them shortly. Next Generation Communication NetworksLab.

  29. If i $ j and i is ㅔpositive recurrent, then j is also positive recurrent. proof: Let (i) and (j) be two stationary measures by using state i and j, respectively. Since both are stationary measures, there exists a constant c (< 1) such that (j) = c (i). So, By summing over all elements in both sides, we get (j) e = c (i) e < 1. Hence, state j is also positive recurrent. Next Generation Communication NetworksLab.

  30. An irreducible DTMC with a finite state space S is always positive recurrent. proof. Let Ni be the total number of visits to state i. Since i Ni should be 1 and the state space S is finite, for at least one state, say k, we have Nk = 1, which means state k is recurrent. Consequently, all states are recurrent because of the irreducibility of the DTMC. Now, since the stationary measure has a vector  of finite size, the sum ii should be finite, i.e., all states are positive recurrent. Next Generation Communication NetworksLab.

  31. Criterion for recurrence • Suppose that the DTMC is irreducible and let i be some fixed state. • Then the chain is transient if and only if there is a bounded non-zero real valued function h:S-{i} ! R satisfying h(j) = k i pjk h(k), j i. Next Generation Communication NetworksLab.

  32. Criteria for positive recurrence • (Pakes' lemma) Let a DTMC {Xn} be irreducible and aperiodic with state space S={0,1,}. Then {Xn} is positive recurrent if the following are satisfied: • |E[Xn+1-Xn|Xn=i]| < 1 for i=0,1,2, • limsupi!1 E[Xn+1-Xn|Xn = i] < 0, i.e., there exist positive numbers  and N such that E[Xn+1-Xn|Xn = i] < - for all i¸ N Next Generation Communication NetworksLab.

  33. Limiting distribution • The definition of the limiting probabilities of {Xn} i = limn!1 P{Xn = i|X0 = j} • when does the limiting probabilities exist? • consider a DTMC with transition matrix P • For state 0, {Xn} can visits state 0 only at slots with even numbers, i.e., P{X2k+1 = 0} = 0 which means that no limiting probability exists. Next Generation Communication NetworksLab.

  34. Periodicity • Definition of the periodicity • For state i, the span d(i) of state i is defined by d(i) = g.c.d. {n¸ 1 | pii(n) > 0}. • If d(i) = 1, we say state i is an aperiodic state. • If i $ j, then d(i) = d(j). • If pii > 0 for some state i in an irreducible DTMC, the chain is aperiodic. Next Generation Communication NetworksLab.

  35. Ergodicity of a DTMC • The concept of ergodicity Time average T Ensemble average = E[f(X(T))] T Next Generation Communication NetworksLab.

  36. Time average of a DTMC • For a state i, recall that fii(n), n¸ 1 form the p.m.f. for the length between consecutive visits to state i (i.e., renewals), and (i) is the length of a renewal. • When X0 =i, Ni (n) denotes the number of visits to state i (i.e., renewals) in [1,n]. Then by the elementary renewal theorem, Next Generation Communication NetworksLab.

  37. A DTMC {Xn} is said to be ergodic if it is irreducible and all the states are positive recurrent and aperiodic. • In an irreducible and aperiodic Markov chain, there always exist the limits which are the limiting probabilities of the DTMC. Next Generation Communication NetworksLab.

  38. The properties of an irreducible and aperiodic DTMC • When state i is transient or null recurrent, the limit i = 0. • When state i is positive recurrent (and hence all the states are positive recurrent), the limiting distribution is, in fact, the stationary distribution of the DTMC. Therefore, the limiting distribution also satisfies  =  P, i2 Si = 1 (or  e = 1), where e is a column vector all of whose elements are equal to 1. • Since i = limn!1 P{Xn = i|X0 = j} = limn!1 pji(n), we have Pn! e. Next Generation Communication NetworksLab.

  39. stationary vs. limiting Distribution • consider a DTMC with transition matrix P the DTMC is periodic with period 2, but it has the stationary distribution Next Generation Communication NetworksLab.

  40. Computation of the limiting distribution • Iterative algorithm • (0) be a given initial distribution • (n) = (0) Pn, the distribution of Xn. • Then ¼(n) if |(n) - (n-1)| <  for a sufficiently small >0. • Pn! e • Eigenvector of P •  is, in fact, the eigenvector of the matrix P corresponding to an eigenvalue 1. Next Generation Communication NetworksLab.

  41. When the transition matrix P is of dimension k. • Let E be a square matrix of dimension k with all the elements equal to 1. Note that  E = et. (et = the transpose of e) • from  =  P we have  (P+E-I) = et where I denotes the identity matrix of dimension k. • since the matrix P+E-I is invertible,  = et(P+E-I)-1. • Note that the solution of the above equation automatically satisfies the normalizing condition  e = 1. Next Generation Communication NetworksLab.

More Related