410 likes | 607 Views
Discrete time Markov Chain. G.U. Hwang Next Generation Communication Networks Lab. Department of Mathematical Sciences KAIST. Definition.
E N D
Discrete time Markov Chain G.U. Hwang Next Generation Communication Networks Lab. Department of Mathematical Sciences KAIST
Definition • The sequence of R.V.s X0, X1, X2, with a countable state space S is said to be a discrete time Markov chain (DTMC) if it satisfies the Markov Property: for any ik2 S, k=0,1,,n-1 and i, j 2 S. • Time homogeneous DTMC : P{Xn+1 = j | Xn = i} is independent of n. Next Generation Communication NetworksLab.
Transition Probability Matrices • One step transition probability matrix P • P = (pij) where pij = P{Xn+1 = j | Xn = i } • The matrix P is nonnegative and stochastic, i.e., pij¸ 0 and j2 S pij = 1 • n step transition probability matrixP(n) = (p(n)ij) • pij(n) = P{Xn = j | X0 = i} Next Generation Communication NetworksLab.
For a DTMC, the initial distribution and the matrix P uniquely determine the future behavior of the DTMC because Next Generation Communication NetworksLab.
Example: A general random walk • Let Xi be i.i.d. R.V.s with P{X1 = j} = aj, j=0,1,. • Let S0 = 0, Sn = k=1n Xk. Then {Sn, n¸ 1} is a DTMC because Next Generation Communication NetworksLab.
Chapman - Kolmogorov's equation • Chapman - Kolmogorov's theorem pij(n+m) = k2 S pik(n) pkj(m) proof: Next Generation Communication NetworksLab.
Using the chapman-Kolmogorov’s theorem we get i.e., the n-th power of the one step transition matrix P is, in fact, the n step transition matrix. Next Generation Communication NetworksLab.
Example • Find the distribution of X4 where {Xn} (Xn2 S = {1,2}) forms a DTMC with initial distribution P{X0 = 1}=1 and one step transition probability P as follows: sol: Next Generation Communication NetworksLab.
Analysis of a DTMC • When a communication system can be modeled by a DTMC with P and S = {0,1,2}, what happens? A sample path of a DTMC transient period stationary period Next Generation Communication NetworksLab.
The stationary probabilities • The state space S ={0,1, } • The stationary probability vector (distribution) • a row vector = (0, 1, ) is called a stationary probability vector of a DTMC with transition matrix P if it satisfies Next Generation Communication NetworksLab.
Does the stationary distribution always exist? 0 1 2 Next Generation Communication NetworksLab.
Does the stationary distribution always exist? 0 1 2 3 ….. Next Generation Communication NetworksLab.
The key question: • When does the stationary vector exist? • to answer the question, we need to classify DTMCs according to its probabilistic properties as • irreducibility • recurrence • positive recurrence and null recurrence • periodic and aperiodic Next Generation Communication NetworksLab.
Irreducible DTMC • state i can reach state j if there exists n¸ 0 s.t. pij(n) > 0. • In this case we write i ! j. • If i! j and j! i, then we say i and j communicate and write i $ j. i k h j ….. g f ….. Next Generation Communication NetworksLab.
$ is an equivalent relation, that is, • i $ i • i $ j iff j $ i • if i $ j and j $ k, then i $ k • Definition of irreducibility • A DTMC is irreducible if its state space consists of a single equivalent class, i.e., for any i, j 2 S we have i $ j. Next Generation Communication NetworksLab.
A closed set • a set A of states is closed if no one step transition is possible from any state in A to any state in AC, i.e., for every pair of states i 2 A and j 2 AC, pij = 0 • An absorbing state • A single state which alone form a closed set is called an absorbing state • if state i is an absorbing state, pii = 1. Next Generation Communication NetworksLab.
Recurrence • The hitting time (i) of state i: • For a state i 2 S, (i) = inf {n¸ 1| Xn = i}, i.e., (i) is the first visiting time of the DTMC {Xn} to state i. • When no such n exists, (i) = 1 by convention. • The number Ni of visits to state i: • Ni = n=11I{Xn=i} where IA is an indicator function which is defined by 1 if the event A occurs and by 0 otherwise. • Clearly, {Ni > 0} = {(i) < 1}. Next Generation Communication NetworksLab.
Probability mass function of (i) • Define fji (n) = P{ (i)=n | X0 = j } • Then fji = P{the DTMC ever visits state i | X0 = j } = n = 1 1 fji (n) • Definition of recurrence of state i • state i is said to be recurrent if P{(i) < 1 |X0 = i} = 1, and transient otherwise. • that is, state i is recurrent if fii =1, and transient if fii <1. Next Generation Communication NetworksLab.
Observations • Once the DTMC revisits a recurrent state i starting from state i, by the Markov property it has the same probabilistic behavior as before. Hence, state i is visited infinitely. • Further, for a recurrent state i and X0 = i a.s. successive visits to state i can be viewed as renewals and {fii (n) | n¸ 1} is the p.m.f. of the inter-renewal times. Next Generation Communication NetworksLab.
For a transient state i, starting from state i the DTMC revisits state i with probability fii (< 1) and it never enters state i with probability 1- fii. Therefore, by Markov property we have P{the DTMC visits state i n times} = (fii)n (1- fii ), n¸ 1 i.e., the number of visits to state i is according to a geometric distribution. Next Generation Communication NetworksLab.
The following are equivalent: • state i is recurrent • Ni = 1 with probability 1 provided that X0 = i • E[Ni|X0 = i] = E[n=11 I{Xn = i}|X0 = i] = n=11 pii(n) = 1 • The following are equivalent: • state i is transient • Ni < 1 with probability 1 provided that X0 = i • E[Ni|X0 = i] = n=11 pii(n) < 1 Next Generation Communication NetworksLab.
If i $ j and i is recurrent, then j is also recurrent. Proof: Since i $ j, there exist n, m ¸ 0 s.t. pij(n) > 0 and pji(m) > 0. Then, which comes from the recurrence of state i. This completes the proof. Next Generation Communication NetworksLab.
Positive recurrence • For a recurrent state i, • if E[(i) | X0 = i] < 1, state i is called positive recurrent. • if E[(i) | X0 = i] = 1, then state i is called null recurrent. • Note that, E[(i) | X0 = i] = 1 for a transient state i. Next Generation Communication NetworksLab.
A stationary measure • a vector q = (q0, q1,) is called a stationary measure of a M.C. with transition matrix P if • q 0 • all qi are finite, i.e., qi < 1 • q P = q Next Generation Communication NetworksLab.
Let i2 S be a recurrent state. Then a stationary measure (q0, q1,) can be defined by Next Generation Communication NetworksLab.
The stationary vector q defined above depends on the chosen state i. • However, it can be shown that the stationary measure (q0, q1,) is unique up to a constant multiplication. Next Generation Communication NetworksLab.
Note that • So, if state i is positive recurrent, by normalizing the stationary measure (q0, q1,) , we have a stationary distribution (p0, p1,) for the DTMC {Xn} Next Generation Communication NetworksLab.
The stationary distribution • The existence of the stationary distribution • If the DTMC is irreducible and positive recurrent, the stationary distribution exists and is given by • Note that the above equation is not used in numerical computation. In fact, we use • For numerical algorithms, we will see them shortly. Next Generation Communication NetworksLab.
If i $ j and i is ㅔpositive recurrent, then j is also positive recurrent. proof: Let (i) and (j) be two stationary measures by using state i and j, respectively. Since both are stationary measures, there exists a constant c (< 1) such that (j) = c (i). So, By summing over all elements in both sides, we get (j) e = c (i) e < 1. Hence, state j is also positive recurrent. Next Generation Communication NetworksLab.
An irreducible DTMC with a finite state space S is always positive recurrent. proof. Let Ni be the total number of visits to state i. Since i Ni should be 1 and the state space S is finite, for at least one state, say k, we have Nk = 1, which means state k is recurrent. Consequently, all states are recurrent because of the irreducibility of the DTMC. Now, since the stationary measure has a vector of finite size, the sum ii should be finite, i.e., all states are positive recurrent. Next Generation Communication NetworksLab.
Criterion for recurrence • Suppose that the DTMC is irreducible and let i be some fixed state. • Then the chain is transient if and only if there is a bounded non-zero real valued function h:S-{i} ! R satisfying h(j) = k i pjk h(k), j i. Next Generation Communication NetworksLab.
Criteria for positive recurrence • (Pakes' lemma) Let a DTMC {Xn} be irreducible and aperiodic with state space S={0,1,}. Then {Xn} is positive recurrent if the following are satisfied: • |E[Xn+1-Xn|Xn=i]| < 1 for i=0,1,2, • limsupi!1 E[Xn+1-Xn|Xn = i] < 0, i.e., there exist positive numbers and N such that E[Xn+1-Xn|Xn = i] < - for all i¸ N Next Generation Communication NetworksLab.
Limiting distribution • The definition of the limiting probabilities of {Xn} i = limn!1 P{Xn = i|X0 = j} • when does the limiting probabilities exist? • consider a DTMC with transition matrix P • For state 0, {Xn} can visits state 0 only at slots with even numbers, i.e., P{X2k+1 = 0} = 0 which means that no limiting probability exists. Next Generation Communication NetworksLab.
Periodicity • Definition of the periodicity • For state i, the span d(i) of state i is defined by d(i) = g.c.d. {n¸ 1 | pii(n) > 0}. • If d(i) = 1, we say state i is an aperiodic state. • If i $ j, then d(i) = d(j). • If pii > 0 for some state i in an irreducible DTMC, the chain is aperiodic. Next Generation Communication NetworksLab.
Ergodicity of a DTMC • The concept of ergodicity Time average T Ensemble average = E[f(X(T))] T Next Generation Communication NetworksLab.
Time average of a DTMC • For a state i, recall that fii(n), n¸ 1 form the p.m.f. for the length between consecutive visits to state i (i.e., renewals), and (i) is the length of a renewal. • When X0 =i, Ni (n) denotes the number of visits to state i (i.e., renewals) in [1,n]. Then by the elementary renewal theorem, Next Generation Communication NetworksLab.
A DTMC {Xn} is said to be ergodic if it is irreducible and all the states are positive recurrent and aperiodic. • In an irreducible and aperiodic Markov chain, there always exist the limits which are the limiting probabilities of the DTMC. Next Generation Communication NetworksLab.
The properties of an irreducible and aperiodic DTMC • When state i is transient or null recurrent, the limit i = 0. • When state i is positive recurrent (and hence all the states are positive recurrent), the limiting distribution is, in fact, the stationary distribution of the DTMC. Therefore, the limiting distribution also satisfies = P, i2 Si = 1 (or e = 1), where e is a column vector all of whose elements are equal to 1. • Since i = limn!1 P{Xn = i|X0 = j} = limn!1 pji(n), we have Pn! e. Next Generation Communication NetworksLab.
stationary vs. limiting Distribution • consider a DTMC with transition matrix P the DTMC is periodic with period 2, but it has the stationary distribution Next Generation Communication NetworksLab.
Computation of the limiting distribution • Iterative algorithm • (0) be a given initial distribution • (n) = (0) Pn, the distribution of Xn. • Then ¼(n) if |(n) - (n-1)| < for a sufficiently small >0. • Pn! e • Eigenvector of P • is, in fact, the eigenvector of the matrix P corresponding to an eigenvalue 1. Next Generation Communication NetworksLab.
When the transition matrix P is of dimension k. • Let E be a square matrix of dimension k with all the elements equal to 1. Note that E = et. (et = the transpose of e) • from = P we have (P+E-I) = et where I denotes the identity matrix of dimension k. • since the matrix P+E-I is invertible, = et(P+E-I)-1. • Note that the solution of the above equation automatically satisfies the normalizing condition e = 1. Next Generation Communication NetworksLab.