Chapter 4 Discrete time Markov Chain

1. Chapter 4 Discrete time Markov Chain • Learning objectives : • Introduce discrete time Markov Chain • Model manufacturing systems using Markov Chain • Able to evaluate the steady-state performances • Textbook : • C. Cassandras and S. Lafortune, Introduction to Discrete Event Systems, Springer, 2007

2. Plan • Basic definitions of discrete time Markov Chains • Classification of Discrete Time Markov Chains • Analysis of Discrete Time Markov Chains

3. Basic definitions of discrete time Markov chains

4. Discrete Time Markov Chain (DTMC) Definition : a stochastic process with discrete state space and discrete time {Xn, n > 0} is a discrete time Markov Chain (DTMC) iff P[Xn+1 = j  Xn = in, ..., X0 = i0] = P[Xn+1 = j  Xn = in] = pij(n) In a DTMC, the past history impacts on the future evolution of the system via the current state of the system pij(n) is called transition probability from state i to state j at time n.

5. Discrete Time Markov Chain (DTMC) Stochastic process Continuous event Discrete events Continuous time A DTMC is a discrete time and memoriless discrete event stochastic process. Discrete time Memoryless

6. Example: a mouse in a maze (老鼠在迷宫) 1 2 start 5 3 4 exit 0 Which stochastic process can be used to represent the position of the mouse at time t? Under which assumptions, the system can be represented by a discrete time Markov chain?

7. Example: a mouse in a maze 1/2 1/2 1 2 start 1/2 1/4 1/2 1/4 1 1/4 5 3 4 1 1/4 exit 0 1 • Let {Xn}n=0, 1, 2, ... the position of the mouse after n rooms visited • Assume that the mouse does not have any memory of rooms visited previously and that she chooses any corridor equi-probably.

8. Homogenuous DTMC • A DTMC is said homogenuous iff its transitions probabilities do not depend on the time n, i.e. • P[Xn+1 = j  Xn = i] = P[X1 = j  X0 = i] = pij • A homogenuous DTMC is then defined by its transition matrix P =[pij]i,jE

9. What is the transition matrix of the process? 1/2 1/2 1 2 start 1/2 1/4 1/2 1/4 1 1/4 5 3 4 1 1/4 exit 0 1 • Let {Xn}n=0, 1, 2, ... the position of the mouse after n rooms visited • Assume that the mouse does not have any memory of rooms visited previously and that she chooses any corridor equi-probably.

10. Stochastic Matrix • A square matrix is said stochastic iff • all entries are non negative • each line sums to 1 • Properties: • A transition matrix is a stochastic matrix • If P is stochastic, then Pn is stochastic • The eigenvalues of P are all smaller than 1, i.e. |l| ≤1

11. Assumptions • In the remaining of the chapter, we limit ourselves to Markov chain • of discrete time • defined on a finite state space E • homogeneous in time. • Note that most results extend to countable state space.

12. Graphic representation of a DTMC

13. Classification of Discrete Time Markov Chains

14. Classification of states • Let fjj be the probability of returning to state j after leaving j. • A state j is said transient if fjj < 1 • A state j is said recurrent if fjj = 1 • A state j is said absorbing if pjj = 1. • Let Tjj be the average reccurn time, i.e. time of returning to j • A recurrent state j is positive recurrent if E[Tjj] is finite. • A recurrent state j is null recurrent if E[Tjj] = .

15. Classify the states of the example 1/2 1/2 1 2 start 1/2 1/4 1/2 1/4 1 1/4 5 3 4 1 1/4 exit 0 1

16. Irreducible Markov chain • A DTMC is said irreducible iff a state j can be reached in a finite number of steps from any other state i. • An irreducible DTMC is a strongly connected graph.

17. Irreducble Markov chain

18. Periodic Markov chain • A state j is said periodic if it is visited only in a number of steps which is multiple of an integer d > 1, called period. • A state j is said aperiodic otherwise • A state with a self-loop transition (i.e. pii > 0) is always aperiodic. • All states of an irreducible Markov chain have the same period.

19. Partitionning a DTMC into irreducible sub-chains • A DTMC can be partitionned into strongly connected components, each corresponding to an irreducible sub-chain.

20. Classification of irreducible sub-chains • A sub-chain is said absorbing if there is no arc going out of it. • Otherwise, the sub-chain is transient. transcient sub-chain absorbing sub-chain absorbing sub-chain

21. Canonic form of transition matrix • Q : transitions of transient sub-chains • Pi : transititions between states of aborbing sub-chain i • Ri: Transitions toward absorbing sub-chain i

22. Formal definitions • A state j is said reachable from a state i if there is a path from i to j in the state transition diagram. • A subset S of states is said closed if there is no transition leaving S. • A closed set S is said irreducible if all states in S are mutually reachable. • A Markov chain is said irreducible if its state space is irreducible.

23. Theorems • Th1. If a Markov chain has a finite state space, then at least one state is recurrent. • Th2. If i is a recurrent state and j is reachable from i, then state j is recurrent. • Th3. If S is a finite closed irreducible set of states, then every state in S is recurrent. • Th4. If i is a positive recurrent state and j is reachable from i, then state j is positive recurrent. • Th5. If S is a closed irreducible set of states, then every state in S is positive recurrent or every state in S is null recurrent or every state in S is transient. • Th6. If S is a finite closed irreducible set of states, then every state in S is positive recurrent.

24. Analysis of DTMC

25. Sojourn time in a state • Let Ti be the temps spent in state i before jumping to other states. • Ti is a random variable of geometric distribution.

26. Properties of geometric distribution • Let X be a random variable of geometric distribution with parameter p, i.e. P{X = n} = (1-p)n-1p. • E[X] = 1/p • Var(X) = 1/p2 • sX = 1/p • Coefficient of variation = sX / E[X] = 1 • Memoryless (only discrete distribution of this property):

27. m-step transition probabilities • The probability of going from i to j in m steps is • pij(m) = P{Xn+m = j|Xn=i} = P{Xm = j|X0=i}. • Let P(m) = [pij(m)] be the m-step transition matrix • Properties (to prove): • P(m) = Pm • Chapman-Kolmogorov equation: • P(l+m) = P(l)P(m) • or

28. Example 1/2 1/2 1 2 start 1/2 1/4 1/2 1/4 1 1/4 5 3 4 1 1/4 exit 0 1 • What is the probability that the mouse is still in room 2 at time 4? (p22(4))

29. Probability of going from i to j in exactly n steps • fij(n) : probability of going from i to j in exactly n steps (without passing j before) • fij: probability of going from i to j in a finite number of steps • Similar approach can be used to determine the average time Tij it takes for going from i to j

30. Probability distribution of states • pi(n) : probability of being in state i at time n • pi(n) = P{Xn = i} • p(n) = (p1(n), p2(n), ...) : vector of probability distribution over the state spaceat time n • The probability distribution p(n) depends on • the transition matrix P • the initial distribution p(0) • Remark: if the system isat state i for certainty, thenpi(0) = 1 and pj(n) = 0, for j ≠i • Whatis the relation betweenp(n), p(0), and P?

31. Transient state equations • By conditioning on the state at time n, • Property: • Let P be the transition matrix of a markovchain and p(0) the initial distribution, then over the state spaceat time n • p(n+1) = p(n)P • p(n)= p(0)Pn

32. Steady-state distribution • Key questions : • Is the distribution p(n) converges when n goes to infinity? • If the distribution converges, does its limit p = (p1, p2, ...) depend on the initial distribution p(0)? • If a state is recurrent, what is the percentage of time spent in this state and what is the number of transitions between two successive visits to the state? • If a state is absorbing, what is the probability of ending at this state? What is the average time to this state?

33. Steady state distribution • Theorem: For a irreducible and aperiodicDTMCwith positive recurrent states, the distribution p(n) converges to a limitvectorpwhichisindependent of p(0) and is the unique solution of the system: • pi are alsocalledstationaryprobabilities (alsocalledsteady state or equilibrium distribution). • For an irreducible and periodicDTMC, pi are the percentage of time spent in state i Normalization equation balance equation equilibrium equation

34. Flow balance equation • Equation canbeinterpretated as balance equation of probability flow. • A probability flow pipijisassociated to each transition (i, j). • is the sum of probability flow intonode j • is the sum of flow out of node j • The flow balance equation : Outgoing flow = Incoming flow

35. A manufaturing system • Consider a machine which can be either UP or DOWN. • The state of the machine is checked every day. • The average time to failure of an UP machine is 10 days. • The average time for repair of a DOWN machine is 1.5 days. • Determine the conditions for the state of the machine {Xn} at the begining of each day to be a Markov chain. • Draw the Markov chain model. • Find the transient distribution by starting from state UP and DOWN. • Check whether the Markov chain is recurrent and aperiodic. • Determine the steady state distribution. • Determine the availability of the machine.

36. A telephone call process • Discrete time model with time slots indexed by k = 0, 1, 2, ... • At most one telephone call can occur in a single time slot, and there is a probability a that a call occurs in any slot • If the phone is busy, the call is lost; otherwise, the call is processed. • There is a probability b that a call in process completes in any time slot • If both a call arrival and a call completion occur in the same time slot, the new call will be processed. • Issues to solve: • Markov chain model • Loss probability