1 / 27

Continuous Time Markov Chain

Continuous Time Markov Chain. G.U. Hwang Next Generation Communication Networks Lab. Department of Mathematical Sciences KAIST. Continuous time Markov chains.

seven
Download Presentation

Continuous Time Markov Chain

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Continuous Time Markov Chain G.U. Hwang Next Generation Communication Networks Lab. Department of Mathematical Sciences KAIST

  2. Continuous time Markov chains • A continuous time stochastic process {Xt}t¸ 0 with state space S is said to be a continuous time Markov Chain (CTMC) if it satisfies the Markov property: for all s,t¸ 0 and i,j,iu2 S, 0· u < s. • Time homogeneous DTMC : P{Xt+s = j | Xs = i} is independent of s. Next Generation Communication Networks Lab.

  3. Let P(t) = (pij(t)) where pij(t) = P{Xt+s = j | Xs = i}, which is called the transition probability function from state i to state j. • As for a DTMC, the initial distribution and the matrix P uniquely determine the future behavior of the CTMC because, for 0· t1< t2<<tn Next Generation Communication Networks Lab.

  4. Notations for a CTMC • the times of jumps (state transitions) : S0 = 0 < S1< S2, • the sojourn times : Tn = Sn+1 - Sn, n¸ 0 • the sequence of states visited : Y0 = X0, Y1= XS1, Y2= XS2,  ….. We assume that the sample path of a CTMC is right continuous. time Next Generation Communication Networks Lab.

  5. Properties of a CTMC • Note that by the Markov property, we have P{T0 > t+s| T0 > s, X0 = i} = P{T0 > t| X0 = i}, which shows that the memoryless property of the sojourn times in CTMC. • Let zi(t) = P{T0 > t| X0 = i}. Then • since zi(t) is right continuous, the only solution of the above equation is zi(t) = e-qi t for some constant qi > 0. Next Generation Communication Networks Lab.

  6. Furthermore, we can show that, for i  j where rij = P{Y1 = j | Y0 = i}. c.f. Y0 = X0 • The matrix R=(rij) is called the first jump probability matrix and, in fact, the one step transition probability matrix of the embedded DTMC {Yn}. • Note that the diagonal elements of R are all 0. Next Generation Communication Networks Lab.

  7. A regular CTMC • A CTMC is said to be regular (or non-explosive) if the number of jumps in a finite time interval is finite. • In an explosive CTMC, we have sample paths that stop at some time point and never evolve further. So, an explosive CTMC is obviously out of our interest. • A sufficient condition for a regular CTMC is that there is a finite upper bound on qi, i2 S , i.e., supi2 S qi < 1. Next Generation Communication Networks Lab.

  8. Define and Q = (qij) : the transition rate matrix • Note that j i, j2 S qij = qi for j i, j2 S rij=1. That is, Qe = 0. Next Generation Communication Networks Lab.

  9. Properties of qij Next Generation Communication Networks Lab.

  10. Next Generation Communication Networks Lab.

  11. We showed that • Given that Xt = i • qij is the transition rate from state i to state j • qi is the transition rate of leaving state i Next Generation Communication Networks Lab.

  12. Basic equations for a CTMC • Chapman - Kolmogorov's equation pij(t+s) = k2 S pik(t) pkj(s) • Kolmogorov's backward differential equation • Kolmogorov's forward differential equation provided that supi2 S qi < 1 Next Generation Communication Networks Lab.

  13. Next Generation Communication Networks Lab.

  14. Example of a CTMC • On and Off source with transition rate matrix Q • an on period is exponentially distributed with  • an off period is exponentially distributed with  • Now we want to compute p00(t) for the ON and OFF source. 0 (silent) 1 (active) Next Generation Communication Networks Lab.

  15. From Kolmogorov's forward equation, we get where we use p01(t) = 1-p00(t). Then we have Thus Next Generation Communication Networks Lab.

  16. Since p00(0) = 1, we see that c = /(+) and finally • In addition, we get Next Generation Communication Networks Lab.

  17. Time dependent probability • From Kolmogorov's forward differential equation, the vector P(t) is given by P(t) = P(0) eQt = eQt (because P(0) = I), where the matrix exponent function eQ is defined by eQ = I + Q + Q2/(2!) + = n=01 Qn/(n!) Next Generation Communication Networks Lab.

  18. Irreducibility and recurrence • We define a CTMC {Xt} is irreducible if any one in the following holds: • The embedded DTMC {Yn} is irreducible • For any i,j 2 S, we have pij(t) > 0 for some t>0. • For any i,j 2 S, we have pij(t) > 0 for all t>0. • Let (i) = inf{t>0| Xt =i, Xt-= lims! t Xs i}. • We say state i is recurrent (transient) if P{(i) < 1|X0=i} = 1 (< 1). Next Generation Communication Networks Lab.

  19. Stationary distribution • A measure  = (0 , 1,…)( 0) is stationary if 0·i <1 and  P(t) = for all t ¸ 0. Next Generation Communication Networks Lab.

  20. The stationary measure  is, in fact, a solution of  Q = 0. Next Generation Communication Networks Lab.

  21. Positive recurrence • A recurrent state i is called positive recurrent (null recurrent) if E[(i)|X0 = i] <1 (=1). • A positive probability vector  =(0,1,)(> 0) is stationary if 0 < i <1 and  P(t) =  for all t¸ 0. • Therefore, an irreducible and positive recurrent CTMC has the unique stationary distribution by normalizing the stationary measure. Next Generation Communication Networks Lab.

  22. Remarks • There is NO relation for positive recurrence between a CTMC and its embedded DTMC. • Example: consider an irreducible and null recurrent DTMC having stationary measure . If we take qi such that then the resulting CTMC is positive recurrent. • In the study of a CTMC, we don't need to pay attention to periodicity because all the sojourn times are exponentially distributed. Next Generation Communication Networks Lab.

  23. In the Kolmogorov’s forward differential equation, i.e., d/dt P(t) = P(t) Q, if we set P(t) = e  and d/dt P(t) = 0, the stationary probabilities  satisfies  Q =0. Next Generation Communication Networks Lab.

  24. Global balance equation : rate out = rate in From  Q = 0 we have i qi = j2 S, j ijqji, i 2 S • i qi : rate out • j2 S, j ijqji : rate in Next Generation Communication Networks Lab.

  25. Ergodicity • An irreducible positive recurrent CTMC is called ergodic. • If an irreducible regular CTMC {Xt} has a nonnegative numbers i satisfying  Q = 0 and  e = 1, then {Xt} is ergodic and hence it has the unique stationary distribution . • If a CTMC {Xt} is ergodic and  is the stationary distribution, then for all i,j2 S we have pij(t)!j as t!1. Next Generation Communication Networks Lab.

  26. Computational Algorithms • When the transition rate matrix Q is a square matrix of dimension k. • Let E be a square matrix of dimension k with all the elements equal to 1. •  E = et. • from  Q=0 we have  (Q+E) = et. We can show the matrix Q+E is invertible, and consequently  = et(Q+E)-1. • Note that the solution of the above equation automatically satisfies the normalizing condition  e = 1. Next Generation Communication Networks Lab.

  27. Let q = sup(qi). Then P = I + (1/q) Q is a stochastic matrix and it satisfies  =  P. Using the same methods for a DTMC, we can compute . Next Generation Communication Networks Lab.

More Related