EE255/CPS226 Discrete Time Markov Chain (DTMC)

1. EE255/CPS226Discrete Time Markov Chain (DTMC) Dept. of Electrical & Computer engineering Duke University Email: bbm@ee.duke.edu, kst@ee.duke.edu

2. Discrete Time Markov Chain • Markov process: dynamic evolution is such that future state depends only on the present (past is irrelevant). • Markov Chain  Discrete state (or sample) space. • DTMC : time (index) is also discrete i.e. system is observed only at discrete intervals of time. • X0, X1, .., Xn, .. :observed state (of a particular ensemble} member (of the sample space) at discrete times, t0, t1,..,tn, .. • {X0, X1, .., Xn ,..} describes the states of a DTMC • Xn= j  system state at time step n is j. Then for a DTMC, • P(Xn= in| X0= i0, X1= i1, …, Xn-1 = in-1) = P(Xn= in| Xn-1 = in-1) • pj(n)  P(Xn = j) (pmf), or, • pjk(m,n)  P(Xn = k | Xm = j ), Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

3. Transition Probability • pjk(m,n): probability transition function of a DTMC. • Homogeneous DTMC: pjk(m,n) = pjk(m-n) i.e., transition probabilities exhibit stationary property. For such a DTMC, • 1-step transition prob, pjk = pjk(1) = P(Xn= k| Xn-1 = j) , • Assuming 0-step transition prob as: • Joint pmf is given by, P(X0= i0, X1= i1, …, Xn = in) = P(X0= i0, X1= i1, …, Xn-1 = in-1). P(Xn = in|X0= i0, X1= i1, …, Xn-1 = in-1) = P(X0= i0, X1= i1, …, Xn-1 = in-1). P(Xn = in|Xn-1 = in-1) (due to Markov prop) = P(X0= i0, X1= i1, …, Xn-1 = in-1).pin-1, in : = pi0(0)pi0, i1 (1)…pin-1, in (1) = pi0(0)pi0, i1 …pin-1, in Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

4. Transition Probability Matrix • The initial prob. is, pi0(0) = P(X0 = i0 ). In general, • p0(0) = P(X0 = 0), …, pk(0) = P(X0 = k) etc, or, • p(0) = [p0(0), p1(0), … ,pk(0), ….] (initial prob. vector) • This allows us to define transition prob. matrix as, • Sum of ith row elements pi,0(0)+ pi,1(0)+ … ? • Any such sq. matrix with non-negative entries whose row sum =1 is called a stochastic matrix. Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

5. State Transition Diagram • pij : describes random state value evolution from i to j • Node with labels i, j etc. and an arc labeled pij • Concept of ri reward (cost or penalty) for each state I allows evaluation of various interesting performance measures. • Example: 2-state DTMC for a cascade binary comm. channel. Signal values: ‘0’ or ‘1’ form the state values. i j pij Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

6. Total Probability • Finding total pmf: Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

7. n-Step Transition Probability • For a DTMC, find • Events: state reaches k (from i) & reaches j (from k) are independent due to the Markov property (i.e. no history) • Invoking total probability theorem: • Let P(n) : n-step prob. transition matrix (i,j) entry is pij(n). Making m=1, n=n-1 in the above equation, Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

8. Marginal pmf • j, in general can assume countable values, 0,1,2, …. Defining, • pj(n) for j=0,1,2,..,j,… can be written in the vector form as, • Or, • Pn can be easily computed if n is finite. However, if n is countably infinite, it may not be possible to compute Pn (and p(n) ). Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

9. Marginal pmf Example • For a 2-state DTMC described by its 1-step transition prob. matrix, the n-step transition prob. Matrix is given by, • Proof follows easily by using induction, that is, assuming that the above is true for Pn-1. Then, Pn = P. Pn-1 Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

10. Computing Marginal pmf • Previous example of a cascade digital comm channels: each stage described by a 2-state DTMC, We want to find p(n) (a=0.25 & b=0.5), • The ’11’ element for n=2 and n=3 are, • Assuming initial pmf as, p(0) = [p0(0) p1(0)] = [1/3 2/3] gives, • What happens to Pn as n becomes very large ( infinity)? Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

11. DTMC State Classification • From the previous example, as n becomes infinity, pij(n) becomes independent of n and i! Specifically, • Not all Markov chains may exhibit this behavior. • State classification may be based on the distinction that asymtotically: • some states may be visited infinitely many times. Whereas, some other states may be visited only a small number of times • Transient state: iff there is non-zero probability that the system will NOT return to this state. • Define Xji to be the # of visits to state i, starting from state j, then, • For a transient state (i), visit count needs to finite, which requirespji(n) 0 as n infinity. Eventually, the system will always leave state i. Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

12. DTMC State Classification (contd.) • State i is a said to be recurrentiff, starting from state i, the process eventually returns to the state i with probability 1. • For a recurrent state, time-to-return is a relevant measure. Define fij(n) as the cond. prob. that the first visit to j from i occurs in exactly n steps. • If j = i, then fii(n) denotes the prob. of returning to i in exactly n steps. • Known result: • Let, • Mean recurrence time for state i is Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

13. Recurrent state • Let i be recurrent and pii(n) > 0, for some n > 0. • For state i, define period di as GCD of all such +ve n’s that result in pii(n) > 0 • If di=1, aperiodic and if di>1, then periodic. • Absorbing state: state i absorbing iffpii=1. • Communicating states: i and j are said to be communicating if there exits directed paths from i and j and from j and i. • Closed set of states: A commutating set of states C forms a closed set, if no state outside of C can be reached from any state in C. Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

14. Irreducible Markov Chains • Markov chain states partitioned into two distinct subsets: c1, c2, .., ck-1, ck , such that • ci, i=1,2,..k-1are closed set of recurrent nun-null. • ck transient states. • If ci contain only one state, then ci’s form a set absorbing states • If k=2 and ck empty, then c1 forms an irreducible Markov chain • Irreducible Markov chain: is one in which every state can be reached from every other state in a finite no. of steps, i.e., for all i,j ε I, for some integer n > 0, pij(n) > 0.Examples: • Cascade of digital comm. channels is 0 1 Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

15. Irreducible Markov Chains (contd.) • If one state is recurrent aperiodic, then so are all the other states. Same result if periodic or transient. • For a finite aperiodic irreducible Markov chain, pij(n) becomes independent of i and n as n goes to infinity. • All rows of Pn become identical Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

16. Irreducible Markov Chains (contd.) • Law of total probability gives, • Therefore, 1st eq. can be rewritten as, • In the matrix form, • v is a probability vector, therefore, • Self reading exercise (theorems on pp. 351) • For an aperiodic, irreducible, finite state DTMC, Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

17. Irreducible Markov Chain Example • Typical computer program: continuous cycle of compute & I/O • The resulting DTMC is irreducible with period =1. Therefore, Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

18. Sojourn Time • If Xn = i, then Xn+1 = j should depend only on the current state i, and not on the time spent in state i. • Let Tibe the time spent in state i, before moving to state j • DTMC will remain in state i in the next step with prob. pii and, • Next step (n+1), toss a coin, ‘H’Xn+1 = i, ‘T’Xn+1# i • At each step, we perform a Bernoulli trial. Then, Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

19. Markov Modulated Bernoulli Process • Generalization of a Bernoulli process: the Bernoulli process parameter is controlled by a DTMC. • Simplest case is Binary state (on-off) modulation • ‘On’ Bernoulli param = c1; ‘Off’  c2’ (or =0) • Controlling process is an irreducible DTMC, and, • Reward assignment, r0 = c1 and r1=c2. This gives, Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

20. Examples of Irreducible DTMCs • Example 7.14: non-homogeneous DTMC for s/w reliability • Slotted ALOHA wireless multi-access protocol • Advantages: • 2X more efficient than pure Aloha. • Automatically adapts to changes in station population • Disadvantages: • Throughput maximum of 36.8% theoretical limit. • Requires queuing (buffering) for re-transmission • Synchronization. Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

21. Slotted ALOHA DTMC • New and backlogged requests • Successful channel access iff : • Exactly one new req. and no backlogged req. • Exactly one backlogged req. and no new req. • DTMC state: # of backlogged requests. backlogged new n + + x m-n + + x x Σ Channel Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

22. Slloted Aloha contd. • In a particular state n, successful contention occurs with prob. rn • rn may be assigned as a reward for state n. Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

23. Discrete-time Birth-Death Processes • Special type of DTMC in which P has a tri-diagonal form, Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

24. DTMC solution steps • Solving for v = vP, gives the steady state probabilities. Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

25. Finite DTMCs with Absorbing States • Example: Program having a set of interacting modules. Absorbing state: failure state ( ps5 : unreliability) Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

26. Finite DTMCs, Absorbing States (contd.) • M contains useful information. • Xij : rv denoting random number to visits to j starting from i • E[Xij] = mij (for i, j = 1,2,…, n-1) . Need to prove this statement. • There are three distinct situations that can be enumerated • Let rv Y denote the state at step #2 (initial state: i) • E[Xij| y = n] = δij • E[Xij| y = k] = E[Xkj + δij]= E[Xkj]+ δij { δij , occurs with prob. pij Xkj + δij, occurs with prob. Pik k=1,2,..n (δij : term accounts for i=j case) Xij = si sk sj sn i Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

27. Finite DTMCs, Absorbing States (contd.) • Since, P(Y=k) = pik , k=1,2,..n, total expectation rule gives, • Over all (i,j) values, we need to work with the matrix, • Therefore, fundamental matrix M elements give the expected # of visit to state j (from i) before absorption. • If the process starts in state “1”, then m1jgives the average # of visits to state j (from the start state) before absorption. Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

28. Performance Analysis, Absorbing States • By assigning rewards values to different state, a variety of performance measures may be computed. • Average time to execute a program • s1 is the start state; rjk : (fractional) execution time/visit for sj • Vj = m1j is the average # times statement block sjis executed • We need to calculate total expected execution time, I.e. until the process gets absorbed into stop state (s5) • Software reliability: jth reward = Rj: Reliability of sj .Then, Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University