1 / 29

Flows and Networks Plan for today (lecture 2):

Flows and Networks Plan for today (lecture 2):. Questions? Continuous time Markov chain Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria

darius
Download Presentation

Flows and Networks Plan for today (lecture 2):

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Flows and NetworksPlan for today (lecture 2): • Questions? • Continuous time Markov chain • Birth-death process • Example: pure birth process • Example: pure death process • Simple queue • General birth-death process: equilibrium • Reversibility, stationarity • Truncation • Kolmogorov’s criteria • Summary / Next • Exercises

  2. Discrete time Markov chain: summary • stochastic process X(t) countable or finite state space SMarkov propertytime homogeneous independent tirreducible: each state in S reachable from any other state in Stransition probabilities Assume ergodic (irreducible, aperiodic) global balance equations (equilibrium eqns) solution that can be normalised is equilibrium distributionif equilibrium distribution exists, then it is unique and is limiting distribution

  3. Random walk http://www.math.uah.edu/stat/ • Gambling game over infinite time horizon: on any turn • Win €1 w.p. p • Lose €1 w.p. 1-p • Continue to play • Xn= amount after n plays • State space S = {…,-2,-1,0,1,2,…} • Time homogeneous Markov chain • For each finite time n: • But equilibrium?

  4. Continuous time Markov chain • stochastic process X(t) countable or finite state space SMarkov propertytransition probabilityirreducible: each state in S reachable from any other state in SChapman-Kolmogorov equationtransition rates or jump rates

  5. Continuous time Markov chain • Chapman-Kolmogorov equationtransition rates or jump rates • Kolmogorov forward equations: (REGULAR)Global balance equations

  6. Continuous time Markov chain: summary • stochastic process X(t) countable or finite state space SMarkov propertytransition rates independent tirreducible: each state in S reachable from any other state in SAssume ergodic and regular global balance equations (equilibrium eqns) π is stationary distribution solution that can be normalised is equilibrium distributionif equilibrium distribution exists, then it is unique and is limiting distribution

  7. Flows and NetworksPlan for today (lecture 2): • Questions? • Birth-death process • Example: pure birth process • Example: pure death process • Simple queue • General birth-death process: equilibrium • Reversibility, stationarity • Truncation • Kolmogorov’s criteria • Summary / Next • Exercises

  8. Birth-death process • State space • Markov chain, transition rates • Bounded state space:q(J,J+1)=0 then states space bounded above at Jq(I,I-1)=0 then state space bounded below at I • Kolmogorov forward equations • Global balance equations

  9. Flows and NetworksPlan for today (lecture 2): • Questions? • Birth-death process • Example: pure birth process • Example: pure death process • Simple queue • General birth-death process: equilibrium • Reversibility, stationarity • Truncation • Kolmogorov’s criteria • Summary / Next • Exercises

  10. Example: pure birth process • Exponential interarrival times, mean 1/ • Arrival process is Poisson process • Markov chain? • Transition rates : let t0<t1<…<tn<t • Kolmogorov forward equations for P(X(0)=0)=1 • Solution for P(X(0)=0)=1

  11. Flows and NetworksPlan for today (lecture 2): • Questions? • Birth-death process • Example: pure birth process • Example: pure death process • Simple queue • General birth-death process: equilibrium • Reversibility, stationarity • Truncation • Kolmogorov’s criteria • Summary / Next • Exercises

  12. Example: pure death process • Exponential holding times, mean 1/ • P(X(0)=N)=1, S={0,1,…,N} • Markov chain? • Transition rates : let t0<t1<…<tn<t • Kolmogorov forward equations for P(X(0)=N)=1 • Solution for P(X(0)=N)=1

  13. Flows and NetworksPlan for today (lecture 2): • Questions? • Birth-death process • Example: pure birth process • Example: pure death process • Simple queue • General birth-death process: equilibrium • Reversibility, stationarity • Truncation • Kolmogorov’s criteria • Summary / Next • Exercises

  14. Simple queue • Poisson arrival proces rate , single server exponential service times, mean 1/ • Assume initially empty:P(X(0)=0)=1, S={0,1,2,…,} • Markov chain? • Transition rates :

  15. Simple queue • Poisson arrival proces rate , single server exponential service times, mean 1/ • Kolmogorov forward equations, j>0 • Global balance equations, j>0

  16. Simple queue (ctd)   j j+1  Equilibrium distribution: < Stationary measure; summable  eq. distrib. Proof: Insert into global balance Detailed balance!

  17. Flows and NetworksPlan for today (lecture 2): • Questions? • Birth-death process • Example: pure birth process • Example: pure death process • Simple queue • General birth-death process: equilibrium • Reversibility, stationarity • Truncation • Kolmogorov’s criteria • Summary / Next • Exercises

  18. Birth-death process • State space • Markov chain, transition rates • Definition: Detailed balance equations • Theorem: A distribution that satisfies detailed balance is a stationary distribution • Theorem: Assume that then is the equilibrium distrubution of the birth-death prcess X.

  19. Flows and NetworksPlan for today (lecture 2): • Questions? • Birth-death process • Example: pure birth process • Example: pure death process • Simple queue • General birth-death process: equilibrium • Reversibility, stationarity • Truncation • Kolmogorov’s criteria • Summary / Next • Exercises

  20. Reversibility; stationarity • Stationary process: A stochastic process is stationary if for all t1,…,tn, • Theorem: If the initial distribution is a stationary distribution, then the process is stationary • Reversible process: A stochastic process is reversible if for all t1,…,tn, NOTE: labelling of states only gives suggestion of one dimensional state space; this is not required

  21. Reversibility; stationarity • Lemma: A reversible process is stationary. • Theorem: A stationary Markov chain is reversible if and only if there exists a collection of positive numbers π(j), jS, summing to unity that satisfy the detailed balance equationsWhen there exists such a collection π(j), jS, it is the equilibrium distribution • Proof

  22. Flows and NetworksPlan for today (lecture 2): • Questions? • Birth-death process • Example: pure birth process • Example: pure death process • Simple queue • General birth-death process: equilibrium • Reversibility, stationarity • Truncation • Kolmogorov’s criteria • Summary / Next • Exercises

  23. 10 S\A A Truncation of reversible processes Lemma 1.9 / Corollary 1.10: If the transition rates of a reversible Markov process with state space S and equilibrium distribution are altered by changing q(j,k) to cq(j,k) for where c>0 then the resulting Markov process is reversible in equilibrium and has equilibrium distribution where B is the normalizing constant. If c=0 then the reversible Markov process is truncated to A and the resulting Markov process is reversible with equilibrium distribution

  24. Time reversed process X(t) reversible Markov process  X(-t) also, but Lemma 1.11: tijdshomogeneity not inherited for non-stationary process Theorem 1.12 : If X(t) is a stationary Markov process with transition rates q(j,k), and equilibrium distribution π(j), jS, then the reversed processX(-t) is a stationary Markov process with transition ratesand the same equilibrium distribution Theorem 1.13: Kelly’s lemmaLet X(t) be a stationary Markov processwith transition rates q(j,k). If we can find a collection of numbers q’(j,k) such that q’(j)=q(j), jS, and a collection of positive numbers (j), jS, summing to unity, such thatthen q’(j,k) are the transition rates of the time-reversed process, and (j), jS, is the equilibrium distribution of both processes.

  25. Flows and NetworksPlan for today (lecture 2): • Questions? • Birth-death process • Example: pure birth process • Example: pure death process • Simple queue • General birth-death process: equilibrium • Reversibility, stationarity • Truncation • Kolmogorov’s criteria • Summary / Next • Exercises

  26. Kolmogorov’s criteria • Theorem 1.8:A stationary Markov chain is reversible ifffor each finite sequence of states Notice that

  27. Flows and NetworksPlan for today (lecture 2): • Questions? • Birth-death process • Example: pure birth process • Example: pure death process • Simple queue • General birth-death process: equilibrium • Reversibility, stationarity • Truncation • Kolmogorov’s criteria • Summary / Next • Exercises

  28. Summary / next: • Birth-death process • Simple queue • Reversibility, stationarity • Truncation • Kolmogorov’s criteria • Nextinput / output simple queuePoisson procesPASTAOutput simple queueTandem netwerk

  29. Exercises [R+SN] 1.3.2, 1.3.3, 1.3.5, 1.5.1, 1.5.2, 1.5.5, 1.6.2, 1.6.3, 1.6.4

More Related