Continuous Time Markov Chains

1. Chapter 8 Continuous Time Markov Chains

2. Definition • A discrete-state continuous-time stochastic process is called a Markov chain if for t0 < t1 < t2 < …. < tn < t , the conditional pmf satisfies the relation • A CTMC is characterized by state changes that can occur at any arbitrary time • Index space is continuous. • The state space is discrete valued.

3. Continuous Time Markov Chain (CTMC) • A CTMC can be completely described by: • Initial state probability vector for X(t0): • Transition probabilities. • Also,

4. Homogenous CTMCs • is a time-homogenous CTMC iff Or, the conditional pmf satisfies: • A CTMC is said to be irreducible if every state can be reached from every other state, with a non-zero probability. • A state is said to be absorbing if no other state can be reached from it with non-zero probability.

5. CTMC Chapman-Kolmogorov Equation • It can also be written as : • In the matrix form, (Matrix Q is called the infinitesimal generator matrix (or simply Generator Matrix)

6. CTMC Steady-state Solution • Steady state solution of CTMC • Irreducible CTMCs having +ve steady-state {πj} values are called recurrent non-null. • Performance measures may be computed by assigning reward rates to states and computing expected steady state reward rates • Accumulated reward (over an interval of time)

7. Continuous Time Birth-Death Process • The CTMC and i={0,1,2,…} forms a B-D process, if λi, i={0,1,2,..} and μi, i={1,2,..}exists, and λi: Birth rate (>= 0) and μi: Death rate (>= 0)

8. Continuous Time Birth-Death Process (contd.) In Steady-state,

9. Steady State Equations These are called balance eqs. Re-arranging above, = 0

10. M/M/1 Queue Poisson arrival Process with rate λ • Arrivals follow Poisson distribution, i.e., inter-arrival times are all i.i.d, EXP(λ). • Inter-departuretimes are i.i.d, EXP(μ). • N(t): birth-death proc., λk=λ; μk=μ. • Define, ρ=λ/μ (traffic intensity, in Erlangs)

11. M/M/1 queue (contd.) • From the balance flow equations, we get • ρ < 1 (for reasons of stability). • Expected # of customers,

12. M/M/1 queue (contd.) • This measure can be viewed as a weighted average, . • By choosing suitable weights to the states of a CTMC, we can get most measures of interest and the resulting model is known as the MRM(Markov Reward Model). • Other measures: • Average queue length (E[n]) • Average (expected) response time • Average (expected) wait time etc.

13. M/M/1 queue: Little’s formula • Let the random variable R denote the response time (defined as the time elapsed from the instant of job arrival until its completion) Little’s law states E[R] = E[N]/λ • Here • Response time (R) = wait time (W) + service time (S) E[W] = E[R] – E[S] = 1/μ(1-ρ) - 1/ μ .

14. Response time distribution (tagged job approach) • Assuming FCFS and steady-state conditions • If there are already n jobs in the system, the next job (N+1)st will experience a response time =R= S*+S’1+S2+..+SN • S* : service time for the (N+1)st job; S’1+: residual service time for job currently undergoing service (#1). • Because of the memory-less property, these times are EXP( ). • Hence, for some N=n, the LST of R is, • Therefore,

15. M/M/m queue • m-servers service the queue. μ Poisson arrivals (λ)

16. M/M/m Queue Solution

17. M/M/m Queue performance measures • Average queue length E[N]: rk= k

18. M/M/m Queue performance measures • Server utilization: rv M - number of busy servers. For number of customers 0 <= k <= m, the number of busy servers = k. Beyond that the number of busy servers = m. • A customer may have to join the queue.

19. Poisson stream behavior • M/M/m: input/output both form Poisson streams. • m=2 case • Case 1: Two independent queues • Case 2: M/M/2 case Two separate Poisson streams  2 separate M/M/1 queues Two separate Poisson streams Combined Poisson steams

20. Comparative performance • Case 1: For each M/M/1 queue, • Case 2: Common queue M/M/2

21. M/M/1/n Queue • Finite queue size, finite buffer space  finite state space. • Steady State Solution:

22. M/M/1/n Queue Performance Measures • Mean queue length (expected # of jobs in the system). • rk = k, • Loss probability • rn = 1, rk = 0, k=0,1,..,n-1 • Throughput • rk =m , k=1,2, ..,n; r0 = 0 (or, rk =l , k=0,1,2, ..,n-1; rn = 0)

23. M/M/1/n: Response time distribution • Response time distribution: Job may be rejected (or accepted) • Unconditional • Conditional (conditioned on the job being accepted): • Reward assignment: for the kth state, response time experienced by the tagged task is sum of k-service times, each of which is EXP(μ), i.e., k-stage Erlang. • Unconditional • Conditional

24. Special cases of Birth-Death Process • Pure birth processes • Poisson process • Software Reliability Growth Model: NHPP • Number of software failures occurring in (0, t] is N(t), and N(t) is Poisson with, λ(t) = abe-btand m(t) = E[N(t)] = a(1- e-bt) • Instantaneous failure intensity, λ(t) = b[a-m(t)] • Transient solution may be found using Laplace transforms • Pure death processes • No-repairs

25. Markov Availability Model

26. UP 1 DN 0 2-State Markov Availability Model 1) Steady-state balance equations for each state: • Rate of flow IN = rate of flow OUT • State1: • State0: 2 unknowns, 2 equations, but there is only one independent equation.

27. 2-State Markov Availability Model(Continued) Need an additional equation: Downtime in minutes per year = * 8760*60

28. 2-State Markov Availability Model(Continued) 2) Transient Availability for each state: • Rate of buildup = rate of flow IN - rate of flow OUT This equation can be solved to obtain assuming P1(0)=1

29. 2-State Markov Availability Model(Continued) 3) 4) Steady State Availability:

30. Using SHARPE to Solve the models

48. Markov availability model • Assume we have a two-component parallel redundant system with repair rate . • Assume that the failure rate of both the components is . • When both the components have failed, the system is considered to have failed.

49. Markov availability model(Continued) • Let the number of properly functioning components be the state of the system. The state space is {0,1,2} where 0 is the system down state. • We wish to examine effects of shared vs. non-shared repair.

50. Markov availability model(Continued) 2 1 0 Non-shared (independent) repair 2 1 0 Shared repair