ST3236: Stochastic Process Tutorial 4

ST3236: Stochastic ProcessTutorial 4 TA: Mar Choong Hock Email: g0301492@nus.edu.sg Exercises: 5

Question 1 An urn initially contains a single red and single green ball. A ball is drawn at random, removed and replaced by a ball of the opposite color and this procedure repeats so that there are always two balls in the urn. Let Xnbe the number of red balls in the urn after n draws, with X0 = 1. Specify the transitions probabilities for MC {X}.

Question 1 Case (Xn=0): Both balls are green. One ball will certainly be replaced with red after a ball is drawn. P(Xn+1 = 1 | Xn = 0) = 1, P(Xn+1 = 0 | Xn = 0) = 0, P(Xn+1 = 2 | Xn = 0) = 0 Case (Xn=2): Both balls are red. One ball will certainly be replaced with green after a ball is drawn. P(Xn+1 = 1 | Xn = 2) = 1, P(Xn+1 = 0 | Xn = 2) = 0, P(Xn+1 = 2 | Xn = 2) = 0

Question 1 The transition matrix is given as follows:

Question 2 Find the mean time to reach state 3 starting from state 0 for the MC whose transition probability matrix is

Question 2 Let T = min{n : Xn= 3} and vi = E(T | X0 = i). The mean time to reach state 3 starting from state 0 is v0. We apply first step analysis.

Question 2 Therefore, v0 = 1 + 0.4v0 + 0.3v1 + 0.2v2 + 0.1v3 v1 = 1 + 0v0 + 0.7v1 + 0.2v2 + 0.1v3 v2 = 1 + 0v0 + 0v1 + 0.9v2 + 0.1v3 v3 = 0 Solving the equations, we have v0 = 10.

Question 3 • Consider the MC with transition probability matrix • Starting in state 1, determine the probability that • the MC ends in state 0 • (b) Determine the mean time to absorption.

Question 3a Let T = min{n : Xn= 0 and Xn= 2}, ui= P(XT= 0|X0 = i) and vi= E(T|X0 = i). u0 = 1 u1 = 0.1u0 + 0.6u1 + 0.3u2 u2 = 0 we have u1 = 0.25.

Question 3b Let T = min{n : Xn= 0 and Xn= 2}, ui= P(XT= 0|X0 = i) and vi= E(T|X0 = i). v0 = 0 v1 = 1 + 0.1v0 + 0.6v1 + 0.3v2 v2 = 0 we have v1 = 2.5.

Question 4 • Consider the MC with transition probability matrix • Starting in state 1, determine the probability that • the MC ends in state 0 • (b) Determine the mean time to absorption.

Question 4a Let T = min{n : Xn= 0 and Xn= 2}, ui= P(XT= 0|X0 = i) and vi= E(T|X0 = i). u0 = 1 u1 = 0.1u0 + 0.6u1 + 0.1u2 + 0.2u3 u2 = 0.2u0 + 0.3u1 + 0.4u2 + 0.1u3 u3 = 0 we have, u1 = 0.3810, u2 = 0.5238 Starting in state 1, the probability that the MC ends in state 0 is u1 = 0.3810

Question 4b Let T = min {n : Xn= 0 and Xn= 2}, ui= P(XT= 0|X0 = i) and vi= E(T|X0 = i). v0 = 0 v1 = 1 + 0.1v0 + 0.6v1 + 0.1v2 + 0.2v3 v2 = 1 + 0.2v0 + 0.3v1 + 0.4v2 + 0.1v3 v3 = 0 we have v1 = v2= 3.33.

Question 5 A coin is tossed repeatedly until two successive heads appear. Find the mean number of tosses required. [Hint: Let Xnbe the cumulative number of successive heads. Then the state space is 0,1,2 before stop]

Question 5 – Method 1 Let Yn be the outcome {H, T} of each toss and (Yn-1, Yn) denotes the sample point for the sucessive tosses. There are 4 possible sample points.

Question 5 – Method 1 The transition probability matrix is

Question 5 – Method 1 Let videnote the mean time to each state 3 starting from state i. By the first step analysis, we have the following equations: v0 = 1 + 0.5v0 + 0.5v1 v1 = 1 + 0.5v2 + 0.5v3 v2 = 1 + 0.5v0 + 0.5v1 v3 = 0 Therefore, v0 = 6 (v1 = 4, v2 = 6)

Question 5 – Method 2 Let Xnbe the cumulative number of successive heads. The 3-state state space now is {0,1,2}. Example:

Question 5 – Method 2 Case (Xn = 0) :Previous two tosses are tails Or a head followed by tail. Note: P(Head) = P(Tail) = 0.5 P(Xn+1 = 1 | Xn = 0) = P(Head) = 0.5, P(Xn+1 = 0 | Xn = 0) = P(Tail) = 0.5, P(Xn+1 = 2 | Xn = 0) = 0 Case (Xn = 1): A tail is followed by a head P(Xn+1 = 2 | Xn = 1) = P(Head) = 0.5 P(Xn+1 = 0 | Xn = 1) = P(Tail) = 0.5 P(Xn+1 = 1 | Xn = 1) = 0

Question 5 – Method 2 Case (Xn = 2): Previous two tosses are heads. We make this state absorbing. P(Xn+1 = 1 | Xn = 2) = 0 P(Xn+1 = 2 | Xn = 2) = 1, P(Xn+1 = 0 | Xn = 2) = 0

Question 5 – Method 2 The transition probability matrix is

Question 5 – Method 2 Let videnote the mean time to each state 2 starting from state i. By the first step analysis, we have the following equations: v0 = 1 + 0.5v0 + 0.5v1 v1 = 1 + 0.5v0 + 0.5v2 v2 = 0 Thus, we have, v0 = 6, v1 = 4

ST3236: Stochastic Process Tutorial 4