Prof. Bob Li

1. Lecture 2 Jan. 23, 2013 Prof. Bob Li

2. Chapter 2. Markov chains Prof. Bob Li

3. 2.1 Formulation and basics Definition. A sequence of discrete r.v. X0, X1, X2, ... is called a markov chain if there exists a matrix such that // P(Xt+1 = j | Xt = i) istime-invariant. = … Prof. Bob Li

4. Memoryless and time-invariant Definition. A sequence of discrete r.v. X0, X1, X2, ... is called a markov chain if there exists a matrix such that // P(Xt+1 = j | Xt = i) istime-invariant. // Memoryless: Only thenewestknowledge Xt counts. Prof. Bob Li

5. Memoryless and time-invariant Definition. A sequence of discrete r.v. X0, X1, X2, ... is called a markov chain if there exists a matrix such that // P(Xt+1 = j | Xt = i) istime-invariant. // Memoryless: Only thenewestknowledge Xt counts. Note. A discrete-time stochastic process = a sequenceof r.v. Prof. Bob Li

6. Memoryless and time-invariant Definition. A sequence of discrete r.v. X0, X1, X2, ... is called a markov chain if there exists a matrix such that // P(Xt+1 = j | Xt = i) istime-invariant. // Memoryless: Only thenewestknowledge Xt counts. Terminology. • Any possible value of Xtis called a state of the markov chain. • Pij is called the transition probability from state i to state j. • The matrix  is called thetransition matrix of the markov chain. It is a kk matrix when there are k ( ) states.

7. 0.5 0.4 0.1 0.3 0.4 0.3 0.1 0.2 0.7 sunny rainy cloudy To From sunny cloudy rainy Example of 3-state weather model A simple model of weather assumes that the weather on any day depends only upon the weather of the preceding day. This model can be described as a 3-state markov chain with the transition matrix The transition diagram (or transition graph): Prof. Bob Li

8. 0.5 0.4 0.1 0.3 0.4 0.3 sunny rainy cloudy To From Row sum = 1 Row sum = 1 Row sum = 1 0.1 0.2 0.7 sunny cloudy rainy Row sum = 1 in transition matrix From the transition equation Pij = P(Xt+1= j | Xt= i), the ith row of the transition matrix is the conditional distribution of Xt+1 given that Xt = i. Hence the row sum must be 1 for all i. Column sum  1 Prof. Bob Li

9. Free 1-dim Random Walk At each game, the gambler wins $1 with the probability p and loses $1 with the probability q = 1p. Let Xt be the total net winning after t games. Thus the states are integers and Modeling by a markov chain, the transition matrix and transition graph are: j = i i

10. - 2 1 0 0 0 0 0 0 é ù K ê ú - ê ú 1 q 0 p 0 0 0 0 K ê ú ê ú 0 0 q 0 p 0 0 0 K ê ú ê ú 1 0 0 q 0 p 0 0 K ê ú ê ú K K K ë û Random Walk with an absorbing state In the same gamble as before, the gambler starts with $2. He has to stop gambling if the net winning reaches $2. In other words, P-2,-2 = 1 and P-2,j= 0 for all j  2 The transition matrix and diagram for the markov chain are: ij2 1 0 1 2 3 4 Prof. Bob Li

11. Independence of PartialMemory P(X3= j | X2= i, X1= i1, X0= i0) = Pij To calculate the probabilities for X3, if we know X2, then we can throw away the knowledge about both X1 and X0. // Independent of either full or partial memory. Only the newest knowledge counts. Example. P(X3= j | X2= i, X0= i0) = Pij Proof. Conditioning on X1, P(X3= j | X2= i, X0= i0) Homework. Prove that P(X3= j | X2= i, X1= i1) = Pijby conditioning on X0. // Conditioning on X1 = Pij

12. Independence of Partial Memory (cont’d) Example. P(X3= j | X1= i, X0= i0) = ? Answer. Conditioning on X2, P(X3= j | X1= i, X0= i0) // Conditioning on X2 = The ijth entry in 2 = P(X3= j | X1= i) // Special case of Chapman-Komogorov equation below Conclusion: Only the newest knowledge counts. Prof. Bob Li

13. Transition Equation in Matrix Form Conditioning on Xt, we have where the summation is over all states i. In the matrix form, a row vector represents a distribution among states and a transition means the multiplication by  from right-hand side. The above equation becomes ,where the denotes the distribution of Xt. Prof. Bob Li

14. Whether example. The notation of the row vector in this example is simply Vt = The transition equation in the matrix form becomes 0.5 0.4 0.1 0.3 0.4 0.3 0.1 0.2 0.7 Transition Equation in Matrix Form Prof. Bob Li

15. Chapman-Komogorov Equation Chapman-Komogorov Equation. Vt+n = Vtn. Proof. Iterate the transition equation starting with Vt. Vt+1 = Vt// Multiplication with  represents a transition. Vt+2 = Vt+1 = (Vt) = Vt2 By induction on n, we arrive at the C-K Equation. Corollary. Entries in the matrix n are probabilities of n-step transitions. Proof. Conditioning on Xt, for all j. Thus,Vt+n = VtM, where M is the matrix with the ijth entry being P(Xt+n = j | Xt = i). From the C-K Equation, M = n.

16. Prisoner in Dark Cell • Hole A leads to freedom through a tunnel of a 3-hour journey. • Holes B and C are at the two ends of a tunnel of a 2-hour journey. • Whenever the prisoner returns to the dark cell, he would immediately enter one of the three holes by a random choice. • We want to investigate • when can the prisoner get out • by modeling the problem as a markov chain. Q: What should be the "states"in the markov chain and what should be a transition? Prof. Bob Li

17. Prisoner in Dark Cell (cont’d) States = places; transition = relocation. The problem is “when” to escape. It involves time. We need a transition per unit time, and a convenient unit time is an hour. Label the “hour-points" as F, R, & T. Use these points, together with “cell” and “freedom” as states. Transition matrix F R T Transition graph

18. Prisoner in Dark Cell (cont’d) As before, let Vt denote the distribution of Xt. Then, V0 = (0 1 0 0 0) // Initially the prisoner is in the cell. V1 = V0 = (2/3 0 1/3 0 0) V2 = V1 = (0 2/3 0 1/3 0) = V02 V3 = V2 = (4/9 0 2/9 0 1/3) = V03// 1/3 = P(Free after 3 days) V4 = V3 = (0 4/9 0 2/9 1/3) = V04 V5 = V4 = (8/27 0 4/27 0 5/9) = V05// 5/9 = P(Free after 5 days) ... V = V = (0 0 0 0 1) = V0// 1 = P(Free eventually) Note that V0 means the second row in the matrix  = . In fact, even with any other V0 (i.e., any distribution of the initial state), we would still have V = V = (0 0 0 0 1) = V0 Thus every row in  is (0 0 0 0 1).

19. Prisoner in Dark Cell Q: What should be the "states"? The problem is “when” to escape. It involves time. We need a transition per unit time, and a convenient unit time is an hour. Another convenient unit time is 2 hours. Then, there are just two states, cell and R, while it takes deterministically one hour to go from R to freedom. The problem becomes: • from cell, when can the prisoner reach R? Equivalently we ask: • how many times does the prisoner go The distribution is Geometric0(1/3): R • through the internal tunnel? P(X= t) = (2/3)t(1/3)for all t 0 Prof. Bob Li

20. Node A Node B Node C SlottedAloha multi-access protocol In every timeslot, transmission is successful when there is a unique node actively transmitting. An transmission attempt can be either a new packet or a retransmission. Every backlogged node in every timeslot reattempts transmission with a probability p. The time till reattempt is Geometric1(p): P(time = t) = (1p)t 1pfor all t  1 Note. Geometric0(p): P(time = t) = (1p)tpfor all t  0

21. 0 1 2 3 4 5 Markov model of Slotted Aloha Markov model. Using the number of backlogged nodes as the state of the system, a transition from state k is always to some state j ≥ k−1. • Assumption for the convenience of analysis. • The probability aithat i new arriving packets intend transmission in a timeslot is fixed and is independent of the state. • Every backlogged node in every timeslot reattempts transmission with an independent probability p. Prof. Bob Li

22. 0 1 2 3 4 5 Markov model of Slotted Aloha Markov model. Using the number of backlogged nodes as the state of the system, a transition from state k is always to some state j ≥ k−1. • Assumption for the convenience of analysis. • The probability aithat i new arriving packets intend transmission in a timeslot is fixed and is independent of the state. • The probability bithat i of the backlogged attempt retransmission depends on the state k binomially:

23. Markov model of Slotted Aloha Markov model. Using the number of backlogged nodes as the state of the system, a transition from state k is always to some state j ≥ k−1. • The transition probabilities from state k are: • Pk,k1 = a0b1 // k > 0 in this case; 1 reattempt; no new • Pk,k= a0(1b1) //  1 reattempt; no new • +a1b0 // no reattempt; 1 new • Pk,k+1 = a1(1b0) // k > 0; ≥ 1 reattempt; 1 new • Pk,k+i= ai for i2 // at least2 new Prof. Bob Li

24. 2.2 Limiting probabilities of Markov chain

25. Formulation of limiting probabilities Definition. In the markov chain X0, X1, X2, ...,let the row vector Vt represent the distribution of Xt. If there exists a row vector V such that • regardless of the initial distribution V0, then the distribution V is called the limiting distribution of the markov chain. Remarks. • The limiting distribution is usually called the “stationary state” by its interpretation asa “random state.” • = V0  for all V0. • Taking V0 = (1 0 0 ... 0), we find V equal to 1st row in the matrix . Similarly, taking V0 = (0 1 0 ... 0), we find V equal to 2nd row in . In the same manner, V is equal to every row in .

26. Example of limiting probability Example. For the prisoner in the dark cell, Prof. Bob Li

27. Ergodicmarkov chain Definition. A finite-state markov chain is ergodic if, for some t 1, all entries in the matrix t are nonzero. // For some particular t, the markov chain can go // from anywhere to anywhere in exactly tsteps. Theorem. If a finite-statemarkov chain is ergodic, then the limiting distribution exists. // Ergodicity is a sufficient condition but, we // shall see that it is not a necessary condition. Proof. Skipped. Prof. Bob Li

28. Example of ergodicmarkov chain Example. A salesman travels between HK and Macau. HKMacau All entries in 2 are positive. Hence this markov chain is ergodic. Prof. Bob Li

29. ( ) ( ) é ù 0 1 é ù n n 1 - 1 1 n P = n lim P = ê ú ê ú 2 2 0 1 ® ¥ ë û n ê ú 0 1 ë û Example of a 2-state markov chain The markov chain is notergodic. Nevertheless, the limiting distribution V = (0 1) exists. Prof. Bob Li

30. Limiting distribution = normalized eigenvector • Since V = V0( ) = V0( ) = V0( ) = V, the limiting distribution is a (row) eigenvector of the transition matrix  with the eigenvalue 1. • V is a probability distribution (sum of entries = 1). // Normalization by 2) is crucial since any scalar multiple // of Vis also an eigenvector with the eigenvalue 1. Prof. Bob Li

31. Example of a 2-state markov chain HK Macau All entries in 2 are positive. Hence this markov chain is ergodic. Since the limiting distribution V is a row eigenvector of  with eigenvalue 1, V  = V Or equivalently, V (  I) = (0 0 … 0) Prof. Bob Li

32. Example of a 2-state markov chain HK Macau … Or equivalently, V (  I) = (0 0 … 0) The eigenspaceof the matrix  with eigenvalue 1 is 1-dimensional. This renders (  I) a singular matrix with a rank 1 less than the full. Write V= (xy). We want to calculate xand y. The matrix equation V  = Vmeans two linearly dependent equations: x/2 + y = x x/2 = y Together with the normalization equation: x + y = 1, we can solve x = 2/3 and y = 1/3. Prof. Bob Li

33. Color-Blindness Gene Color-blindness is X-genetic. There are two kinds of X genes: X0 = color blind X1 = normal Thus there are three types of female: X0X0 = color blind X0X1 = normal // X1 gene dominates X0 X1X1 = normal and two types of male: X0Y = color blind X1Y = normal Question. In a “stationary” society, a survey shows that approximately 10% of the male population is color blind. However, the survey also finds too few color blind females to make a percentage estimate. What can we do?

34. 0 . 05 0 . 54 0 . 405 é ù ê ú 2 = Õ ê ú 0 . 03 0 . 36 0 . 63 ê ú ê ú 0 . 005 0 . 14 0 . 855 ë û Color-Blindness Gene (cont’d) Solution. From the percentage of color blind males, we assert the 1 : 9 ratio between the X0 gene and the X1 gene. Then, model the problem by the mother-to-daughter transition matrix : Daughter X0X0 X0X1 X1X1 X0X0 Mother X0X1 X1X1 From the nature of the problem, we know the markov chain is ergodic. In fact, all entries in 2 are positive.

35. Limiting distribution of a 3-state chain Write the limiting distribution as V= (x y z). We want to calculate x, y, and z. The matrix equation V  = V means 3 linearly dependent equations: 0.1x + 0.05y = x 0.9x + 0.5y + 0.1z = y 0.45y + 0.9z = z They are worth only 2 equations, because the eigenspace of the matrix  with eigenvalue 1 is 1-dimensional. The 3 equations together with the normalization equation x + y + z = 1 yields V = (0.01 0.18 0.81). Conclusion. In this “stationary” society, approximately 1% of the female population is color blind. Meanwhile, another 18% carries the color-blind gene. Prof. Bob Li

36. State 0 State 1 Example of Oscillation • The markov chain is not ergodic. // but ratherperiodic • In fact, there is no limiting distribution. Let V0 = (1 0). Then Vt alternates between (1 0) and (0 1). • Nevertheless, there exists the eigenvector (0.5 0.5) of the transition matrix  with the eigenvalue 1. Prof. Bob Li

37. Ergodicity, limiting distributions, eigenvector Summary of theorems for finite-state markovchains. Ergodicity// All entries in the matrix t are nonzero for some t 1. • The limiting distributionVexists. • // regardless of the initial distribution V0. • Eigenvalue 1 of the transition matrix with 1-dim eigenspace // Rank of the matrixI is full minus1(dim of eigenspace = 1). // Limiting distr. = unique normalized eigenvector with eigenvalue 1 • Eigenvalue 1 of the transition matrix // The matrixI is a singular.  det(I) = 0 Q. When there is no limiting distribution, 1 may or may not be an eigenvalue. What happens then? See the next example.

38. p p 0 1 2 3 1p 1p A gambler wants to go home. A gambler in Macau needs $3 to return to HK but has only $2. He gambles $1 at a time, and wins with probability p.  = The matrix Ihas the rank 2, while the full rank is 4. (1 0 0 0) and (0 0 0 1) are two linearly independent eigenvectors of eigenvalue 1. However, there is no limiting distribution. The probability of eventually going to the state $3 instead of $0 depends on the initial distribution V0. For the said gambler, V0 = (0 0 1 0). Prof. Bob Li

39. 2.3 Expected time & terminal probabilities Prof. Bob Li

40. Expected time from dark cell to freedom The transition graph Freedom is the only terminal state.

41. Calculation methods Multiplicity: Given the initial state = cell, let the frequency of entering the internal tunnel be MT, which is Geometric0(1/3) distributed. Thus, E[Waiting time for freedom | initial state = cell] = E[2MT+3] = 2EMT+3 = 2*2+3 = 7 Waiting time: Calculation by markov chain. For every state S, define the conditional r.v. TS = (Waiting time for freedom | initial state = S) and adopt the abbreviation eS = ETS. We have the following matrix equation. This matrix equation means conditioning on the outcome of the 1st step. Using “mean of sum is sum of means.”

42. Waiting time (cont’d) In other words, we have the following system of linear equations: From these, we can solve for eS for all states S. // Even though we only wanted to compute ecell, the // markov-chain computation gives eS for all states S. It turns out that eR = 1, eF = 2, ecell = 7, eT = 8 // Mean of sum is sum of means.

43. Example: Linear programming Minimizect x // ct= (c1 … cn), nonnegative // xt= (x1 … xn) subject to A  x = b // bt= (b1 … bm); A is an mn matrix of rank m // so that there are m linear equations and x0 The feasibility region is an (n–m)-dim polytope in the n-dim space. The optimal xis at an extreme point of this region and has at least n–m components equal to 0. There are extreme points. Label them from 1 to N by the increasing value of the objective function ct x.

44. Simplex algorithm The simplex method moves from an extreme point to a better extreme point at each iterative step. How many steps are needed? Assumption for the convenience of performance analysis: When the algorithm is at the Lth point, the next extreme point will equally likely be any one among the (L–1)st, (L – 2)nd, … , 1st. Thus,

45. Efficiency of simplex algorithm Let Tibe the number of transitions to go from state i to state 1. In particular, T1 = 0. E[Ti] = ijE[Ti | next state is j] =(1+E[Tj]) Shifting the index, Prof. Bob Li

46. Efficiency of simplex algorithm(cont’d) Take the difference between the last two equations. // By telescoping 1 // Let i+1 = // Stirling’s formula: // n!  (2n)1/2 (n/e)n

47. Efficiency of simplex algorithm(cont’d) Write c = n/m. Therefore, under the said assumption, the number of simplex-method iterations from the worst start is only Numerical example. If n = 8000 and m = 1000, then c = 8,

48. Toss a coin till HH appears Q: Average waiting time till two consecutive H’s = 4? // Fair coin Ans: Not 4. It’s 6 instead. HH 1/2 H 1/2 1/2 HT 1/2 T Prof. Bob Li

49. Toss a coin till HH appears Q: Average waiting time till two consecutive H’s = ? // Fair coin Ans: Not 4. It’s 6 instead. HH 1/2 H 1/2 1/2 HT 1/2 T

50. Pattern occurrence in coin tossing Q. Expected waiting time for the consecutive pattern HTHT = ? // A solid arrow = Head; a dotted = Tail. Probability of every arrow is ½. For every state S, define the conditional r.v. TS = (waiting time for HTHT | initial state = S) and write eS = ETS. Conditioning on the first step, we have the following equation: Terminal state