1 / 53

EE255/CPS226 Continuous Time Markov Chain (CTMC)

EE255/CPS226 Continuous Time Markov Chain (CTMC). Dept. of Electrical & Computer engineering Duke University Email: bbm@ee.duke.edu , kst@ee.duke.edu. Discrete State-Continuous Time Stochastic Process.

alexis
Download Presentation

EE255/CPS226 Continuous Time Markov Chain (CTMC)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EE255/CPS226Continuous Time Markov Chain (CTMC) Dept. of Electrical & Computer engineering Duke University Email: bbm@ee.duke.edu, kst@ee.duke.edu

  2. Discrete State-Continuous Time Stochastic Process • A CTMC is characterized by state changes that can occur at any arbitrary time (in contrast to a DTMC, where state changes can occur only at well defined times). • Index space is un-countable. • The state space continues to a discrete valued. • I={0,1,2,..} denotes the process state space • T=[0, ) is the index space. • This forms discrete space, cont. time stochastic process: Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  3. Continuous Time Markov Chain (CTMC) • The process qualifies to be Markov chain, if for t0 < t1 < t2 < …. < tn < t , the conditional pmf satisfies: • Process can then be completely described by: • Initial state probability vector for X(t0): • Transition probabilities. • Also, Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  4. Homogenous CTMCs • is a time-homogenous CTMC iff , • Or, the conditional pmf satisfies: • Marginal pmf (state probability) is given by: • State prob. May be written as, Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  5. CTMC Chapman-Kolmogorov Equation • Proof: using total prob. Law • Since v < u and invoking Markov property proves it. • The final goal of obtaining πj(t) using Chapman-Kolmogorov eq. is cumbersome. • Instead we are forced to prob. transition rates Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  6. Prob. Transition Rates • Lot of math/calculus follows:. Define, • Relating transition probs. & rates : Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  7. Calculus steps • In general, • Therefore, (1) can be rewritten as, Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  8. Prob. Transition Matrix Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  9. Transition Prob. (Homogenous case) • Transition rates qij(t) and qj(t) are independent of t. The Kolmogorov forward equation reduces to, • In the matrix form, (Matrix Q is called the infinitesimal generator matrix (or simply Generator Matrix) • Defining, Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  10. Classification • Classification of states of a CTMC is identical to DTMC • i is anabsorbing state if qij = 0 for j = I. Once the process enters this state, it then never leaves this state. • Steady-state behavior when there are absorbing states: • j is aReachable state from i, if for some t>0, pij(t) > 0. • Irreducible CTMC: iff every state is reachable from every other state. For an irreducible CTMC, the following is true: always exists and are independent of the initial state i. In case the limiting probabilities πj exist, then, Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  11. CTMC Steady-state Solution • Irreducible CTMCs having +ve steady-state {πj} values are called recurrent non-null. • Performance measures may not be computed by assigning rewards to all states and computing E[reward]: • Accumulated reward (over an interval of time) • Markov chain exhibits memory-less property • Hj(t) follows EXP( ) distribution. Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  12. Continuous Time Birth-Death Process • The CTMC and i={0,1,2,…} forms a B-D process, if λi, i={0,1,2,..} and μi, i={1,2,..}exists, and, • λi,: Birth rate (>= 0) and μi,: Death rate (>= 0) Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  13. Continuous Time Birth-Death Process (contd.) Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  14. Steady State Equations These are called balance eqs. Re-arranging above, = 0 Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  15. M/M/1 Queue • Arrivals follow Poisson distribution, i.e., inter-arrival times are all i.i.d, EXP(λ). • Departures are also Poissonian i.e., inter-departure times are all i.i.d, EXP(μ). • Models systems such as, • Packet arrivals at a router port (router switch is the server) • Arrival of tasks at a computer system or a scheduler (server computer) • Customer queues (clerk/cashier is the server) • Repair workshops, etc. (Repair station or the technician is the server) Poisson arrival Process with rate λ Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  16. M/M/1 queue (contd.) • N(t): birth-death proc., λk=λ; μk=μ • Define, ρ=λ/μ (traffic intensity, in Erlangs) • ρ < 1 (for reasons of stability). Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  17. M/M/1 queue performance measures • Server utilization = 1- π0 • Expected # of customers, • Above measures can be viewed as expected reward, • Resulting model is known as the MRM. • Other measures: • Average queue length (E[n]) • Average (expected) response time • Average (expected) wait time et. Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  18. M/M/1 queue: Little’s formula • Response time (R) = wait time (W) + service time (S) • E[R] = E[N]/λ (Little’s formula) N = (1(t2-t1)+2(t3-t2)+1(t4-t3)+1(t6-t5)+2(t7-t6)+3(t8-t7)+2(t9-t8)+1(T- t9 ))/T = (area under the curve)/T = (T+t9 + t8 – t7 – t6 – t5 + t4 + t3 – t2 – t1)/T R = ((t3-t1)+ (t4-t2)+ (t8-t5)+ (t9-t6)+(T- t7))/5 = (T+t9 + t8 – t7 – t6 – t5 + t4 + t3 – t2 – t1)/5 = (area under the curve)/K R.K = N.T . Note that, λ = K/T . This yields the Little’s formula. Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  19. Expected response time • Analysis has tacitly assumed the FCFS scheduling. • Other scheduling policies may be pre-emptive, e.g., RR. • Round Robin (RR)  time-slice, then back to the end of queue. • Scope for more than one RR queue. • The E[R] formula holds for any scheduling policy provided, some conditions are met. Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  20. Response time distribution • Assuming FCFS and steady-state conditions • If there are already n jobs in the system, the next job (N+1)st will experience a response time =R= S*+S’1+S2+..+SN • S* : service time for the (N+1)st job; S’1+: residual service time for job currently undergoing service (#1). • Because of the memory-less property, these times are EXP( ). • Hence, for some N=n, the LST of R is, • Therefore, Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  21. M/M/k queue • m-servers together service the queue. μ Poisson arrivals (λ) μ Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  22. M/M/m Queue Solution Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  23. M/M/m Queue performance measures • Average queue length E[N]: rk= k Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  24. M/M/m Queue performance measures • Server utilization: rv M - number of busy servers. M may be defined in terms of the number of items N in the queue. • A customer may have to join the queue. Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  25. Poisson stream behavior • M/M/m: input/output both form Poisson streams. • m=2 case • Case 1: Two independent queues • Case 2: M/M/2 case Two separate Poisson streams  2 separate M/M/1 queues Two separate Poisson streams Combined Poisson steams Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  26. Comparative performance • Case 1: For each M/M/1 queue, • Case 2: Common queue M/M/2 Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  27. M/M/1/n Queue • Finite queue size, finite buffer space  finite state space. • Transient soln? • Steady State Solution: Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  28. M/M/1/n Queue Performance Measures • Mean queue length (expected # of jobs in the system). • rk = k, • Loss probability • rn = 1, rk = 0, k=0,1,..,n-1 • Throughput • rk =m , k=1,2, ..,n; r0 = 0 (or, rk =l , k=0,1,2, ..,n-1; rn = 0) Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  29. M/M/1/n: Response time distribution • Response time distribution: Job may be rejected (or accepted) • Unconditional (rejected): • Conditional (accepted): • Reward assignment: for the kth state, response time experienced by the tagged task is sum of k-service times, each of which is EXP(μ), i.e., k-stage Erlang. • Unconditional • Conditional Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  30. M/M/1/n Example • Machine failure/repair model • Machine fails with MTTF = 1/λ and MTTR = 1/μ • Transient solution: • pUU(t) can also be computed. • A(t) = pUU(t) • Interval Availability: • Read examples λ U D μ Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  31. M/M/1/n Example : varying birth rate • Birth rate in state j is λj= (M-j) λ and μj = μ. • Multiple clients generating service requests and single server. After requesting, client waits for response. • M- parallel component - single repair station Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  32. M/M/1/n: M-components Example • Various repair possibilities exist • Each component may have its own repair facility (U/D model) • Single repair station (to reduce cost) • Single repair station solution may be slow, increase the repair rate from μ  M μ Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  33. Special cases of Birth-Death Process • Pure birth processes • Poisson process • Software Reliability Growth Model: NHPP • Number of software failures occurring in (0, t] is N(t), and N(t) is Poisson with, λ(t) = abe-btand m(t) = E[N(t)] = a(1- e-bt) • Instantaneous failure intensity, λ(t) = b[a-m(t)] • Transient solution may be found using Laplace transforms • Pure death processes • No-repairs Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  34. Non Birth-Death processes • Not all CTMC may exhibit nearest neighbor transittions only. • Markov chains with arbitrary transition pattern • Availability modeling • Performance modeling • Performability modeling • Example based Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  35. Availability model using CTMCs • Failure/repair model • repair = failure detection/location + actual repair • Life time: EXP(λ ); Detection/location: EXP(μ1); Repair: EXP(μ2) • The flow equations are: λ μ1 0 1 2 μ2 Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  36. 2-component Availability model • 2-component availability model • Ass = 1-π0 • Failures detection stage takes random time, EXP(δ) • Down states are ‘0’ and ‘1D’  Ass = 1- π0- π1D Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  37. 2-components : finite coverage • Coverage factor = c • ‘1C’ state is a re-boot (down) state. Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  38. 2-components : delay+finite coverage • Model has detection delay+coverage factor • Down states are ‘0’, ‘1C’ and ‘1D’. Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  39. Preventive Maintenance example • Prolonged usage of a component may lead to increased failure rate (i.e. IFR situation) • Hence, life time may be modeled as HypoEXP() distribution, say 2-stage Hypo. • Component is inspected randomly. Time between inspections is a random, following EXP(λi). Inspection completion time is EXP(μi). • What does inspection do? • First stage of life – no action • Second stage of life – repair • That is, preventive maintenance • State = <#stage, faulty> Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  40. Performance Models • Example: 2-servers with different service times. • State = <n1, n2> • Performance: Average no. of jobs in the system, E[n1+n2] • Reward rn1, n2 = n1+n2 • Except for the <0,0>, in all other states, viz., <k,0> and <k,1>, there are k jobs in the system. Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  41. Markov Modulated Poisson Process (MMPP) • MMPP is a doubly stochastic Poisson process, such that the arrival rate is dependent on the state of another CTMC. • The second CTMC (call it MM) is the modulating process having m states in general, Q = [qij]. • If MM is in state I, then the (first) Poisson process has λi as its arrival rate. • The Poisson process is a counting process as the overall modulated process. • 2-d state <Poisson (counting) proc, Modulating proc> . Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  42. MMPP Counting process λ1 λ2 λ3 λ1 λ2 λ3 λ3 λ2 Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  43. CTMCs with Absorbing states • Example: 2-component parallel system (perfect package) • Steady-state solution- not meaningful. We need to find transient solution. Initial starting state i.e., π2(0)=1 Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  44. 2-components, finite coverage • Faults are covered with coverage factor = c. Initial starting state i.e., π2(0)=1 Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  45. Petri Net • Is a modeling tool used for modeling issues, such as, concurrency, synchronization, mutual exclusion etc. Conflict/concurrency PN Producer/Consumer PN PN Markings p1 p2 p1 p1 p2 p3 p4 p5 t1 C 11000 t1 p3 t1 t2 p2 01100 p5 B t2 t3 t3 p3 11001 p4 t2 10010 p5 p4 t4 t3 t5 t4 t4 n : Denotes a place with finite capacity for tokens (=n) n Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  46. PN Definitions • A PN is a 5-tuple: (P, T, A, X, M) . PN is a bi-partite graph. • Arcs: TP or PT • An arc may have multiplicity X. • Rules for enabling a transition t : when all places p (ε P) such pt exists have one or more tokens. • Firing a transitions: an enabled transition may fire, if all the places p’ that have non-null tp’ arcs have no tokens. • Markings, aka reachability Graph • Inhibitor arcs t1 Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  47. Stochastic PNs • Random delay (firing time) between enabling and firing a transition. Such a transition: stochastic transition. • Frequently, firing time: EXP( ) distributed. • Resulting reachability graph of an SPN : Markov chain. • Example: Poisson process Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  48. Stochastic PN: M/M/1 Queue • Arrival and service transitions • M/M/1/n queue • Reachability graph Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  49. Generalized SPN (GSPN) • GSPN is an an enhancement to the SPN. • GSPN has both immediate and timed transitions • Firing probabilities: GSPN also admits the possibility of firing more than one immediate transitions. • Transition priorities: by assigning an integer priority level to each transition. • GSPN vanishing and tangible markings • A vanishing marking involves atleast one immediate transition. • Tangible markings involve only timed transitions. Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

  50. GSPN examples: • Example: M/Em/1/n+1 queue • Analysis of a GSPN has 4-steps • Generate reachability graphs • Eliminate vanishing markings  CTMC of tangible markings • Analyze steady state or transient behavior of the CTMC • Evaluate any measures Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

More Related