Markov Processes and Birth-Death Processes

1. Markov ProcessesandBirth-Death Processes J. M. Akinpelu

2. Exponential Distribution • Definition. A continuous random variable X has an exponential distribution with parameter  > 0 if its probability density function is given by • Its distribution function is given by

3. Exponential Distribution • Theorem 1. A continuous R.V. X is exponentially distributed if and only if for • or equivalently, • A random variable with this property is said to be memoryless.

4. Exponential Distribution • Proof: If X is exponentially distributed, (1) follows readily. Now assume (1). Define F(x) = P{X ≤ x}, f(x) = F(x), and, G(x) = P{X > x}. It follows that G(x) = ‒ f(x). Now fix x. For h 0, • This implies that, taking the derivative wrt x,

5. Exponential Distribution • Letting x = 0 and integrating both sides from 0 to t gives

6. Exponential Distribution • Theorem 2. A R.V. X is exponentially distributed if and only if for h 0,

7. Exponential Distribution • Proof: Let X be exponentially distributed, then for h 0, • The converse is left as an exercise.

8. Exponential Distribution slope (rate) ≈ 

9. Markov Process • A continuous time stochastic process {Xt, t 0} with state space E is called a Markov process provided that • for all states i, j E and all s, t  0. known 0 s s+t

10. Markov Process • We restrict ourselves to Markov processes for which the state space E = {0, 1, 2, …}, and such that the conditional probabilities • are independent of s. Such a Markov process is called time-homogeneous. • Pij(t) is called the transition function of the Markov process X.

11. Markov Process - Example • Let X be a Markov process with • where • for some > 0. X is a Poisson process. 0

12. Chapman-Kolmogorov Equations • Theorem 3. For i, j E, t, s  0,

13. Xt() 7 S4 6 S2 5 4 S3 3 S1 S5 2 S0 1 0 t T0 T2 T3 T4 T5 T1 Realization of a Markov Process

14. Wt t Tn+1 Tn t+u Time Spent in a State • Theorem 4. Let t  0, and n satisfy Tn ≤ t < Tn+1, and let Wt = Tn+1 – t. Let i E, u  0, and define • Then • Note: This implies that the distribution oftime remaining in a state is exponentially distributed, regardless of the time already spent in that state.

15. Time Spent in a State • Proof: We first note that due to the time homogeneity of X, G(u) is independent of t. If we fix i, then we have

16. An Alternative Characterization of a Markov Process • Theorem 5. Let X ={Xt, t 0} be a Markov process. Let T0, T1, …, be the successive state transition times and let S0, S1, …, be the successive states visited by X. There exists some number i such that for any non-negative integer n, for any j E, and t > 0, • where

17. An Alternative Characterization of a Markov Process • This implies that the successive states visited by a Markov process form a Markov chain with transition matrix Q. • A Markov process is irreducible recurrent if its underlying Markov chain is irreducible recurrent.

18. Kolmogorov Equations • Theorem 6. • and, under suitable regularity conditions, • These are Kolmogorov’s Backward and Forward Equations.

19. Kolmogorov Equations • Proof (Forward Equation): For t, h 0, • Hence • Taking the limit as h 0, we get our result.

20. Limiting Probabilities • Theorem 7. If a Markov process is irreducible recurrent, then limiting probabilities • exist independent of i, and satisfy • for all j. These are referred to as “balance equations”. Together with the condition • they uniquely determine the limiting distribution.

21. Birth-Death Processes • Definition. A birth-death process {X(t), t 0} is a Markov process such that, if the process is in state j, then the only transitions allowed are to state j + 1 or to state j – 1 (if j > 0). • It follows that there exist non-negative values jand j, • j = 0, 1, 2, …, (called the birth rates and death rates) so that,

22. j-1 j j j+1 j-1 j j+1 Birth and Death Rates • Note: • The expected time in state j before entering state j+1 is 1/j; the expected time in state j before entering state j‒1 is 1/j. • The rate corresponding to state j is vj = j + j.

23. Differential-Difference Equations for a Birth-Death Process • It follows that, if , then • Together with the state distribution at time 0, this completely describes the behavior of the birth-death process.

24. Birth-Death Processes - Example • Pure birth process with constant birth rate j =  > 0, j = 0 for all j. Assume that Then solving the difference-differential equations for this process gives

25. Birth-Death Processes - Example • Pure death process with proportional death rate j = 0 for all j, j = j > 0 for 1 ≤ j ≤ N, j = 0 otherwise, and Then solving the difference-differential equations for this process gives

26. Limiting Probabilities • Now assume that limiting probabilities Pjexist. They must satisfy: • or

27. Limiting Probabilities • These are the balance equations for a birth-death process. Together with the condition • they uniquely define the limiting probabilities.

28. Limiting Probabilities • From (*), one can prove by induction that

29. When Do Limiting Probabilities Exist? • Define • It is easy to show that • if S < . (This is equivalent to the condition P0 > 0.) Furthermore, all of the states are recurrent positive, i.e., ergodic. If S = , then either all of the states are recurrent null or all of the states are transient, and limiting probabilities do not exist.

30. j-1 j j j+1 j-1 j j+1 Flow Balance Method • Draw a closed boundary around state j: • “flow in = flow out” Global balance equation:

31. Flow Balance Method • Draw a closed boundary between state j and state j–1: j-1 j j j+1 j-1 j j+1 Detailed balance equation:

32. Example • Machine repair problem. Suppose there are m machines serviced by one repairman. Each machine runs without failure, independent of all others, an exponential time with mean 1/. When it fails, it waits until the repairman can come to repair it, and the repair itself takes an exponentially distributed amount of time with mean 1/. Once repaired, the machine is as good as new. • What is the probability that j machines are failed?

33. j-1=(m‒j+1) j=(m‒j) j j+1 j‒1 j= j+1= Example • Let Pj be the steady-state probability of j failed machines.

34. j-1=(m‒j+1) j=(m‒j) j j+1 j‒1 j= j+1= Example

35. Example • How would this example change if there were m (or more) repairmen?

36. Homework • No homework this week due to test next week.

37. References • Erhan Cinlar, Introduction to Stochastic Processes, Prentice-Hall, Inc., 1975. • Leonard Kleinrock, Queueing Systems, Volume I: Theory, John Wiley & Sons, 1975. • Sheldon M. Ross, Introduction to Probability Models, Ninth Edition, Elsevier Inc., 2007.