ST3236: Stochastic Process Tutorial 3

ST3236: Stochastic ProcessTutorial 3 TA: Mar Choong Hock Email: g0301492@nus.edu.sg Exercises: 4

Question 1 • A markov chain X0,X1,…on state 0, 1, 2 has the transition • probability matrix • and initial distributions, p0 = P(X0 = 0) = 0.3, • p1 = P(X0 = 1) = 0.4 and p2 = P(X0 = 2) = 0.3. • Determine P(X0 = 0, X1 = 1, X2 = 2) and draw the state- • diagram with transition probability.

Question 1 P(X0 = 0,X1 = 1,X2 = 2) = P(X0 = 0)P(X1 = 1 | X0 = 0)P(X2 = 2 | X0 = 0,X1 = 1) = P(X0 = 0)P(X1 = 1 | X0 = 0)P(X2 = 2 | X1 = 1) = p0 x p01 x p12 = 0.3 x 0.2 x 0 = 0.

Question 2 • A markov chain X0,X1,…on state 0, 1, 2 has the transition • probability matrix • Determine the conditional probabilities • P(X2 = 1,X3 = 1|X1 = 0) and P(X1 = 1,X2 = 1|X0 = 0).

Question 3 • A markov chain X0,X1,…on state 0, 1, 2 has the transition • probability matrix • If we know that the process starts in state X0 = 1, determine • probability P(X0 = 1,X1 = 0,X2 = 2).

Question 3 P(X0 = 1,X1 = 0,X2 = 2) = P(X0 = 1)P(X1 = 0| X0 = 1)P(X2 = 2 | X0 = 1,X1 = 0) = P(X0 = 1)P(X1 = 0| X0 = 1)P(X2 = 2 | X1 = 0) = p1 x p10 x p02 = 1 x0.3 x 0.1 = 0.03

Question 4 • A markov chain X0,X1,…on state 0, 1, 2 has the transition • probability matrix

Question 4a Compute the two-step transition matrix P(2). Note: Observe that the rows must always sum to one for all transition matrices.

Question 4b What is P(X3 = 1|X1 = 0)? P(X3 = 1|X1 = 0) = 0.13 In general, P(Xn+2 = 1 | Xn = 0) = 0.13 for any n.

Question 4c What is P(X3 = 1|X0 = 0)? Note that: Thus, P(X3 = 1|X0 = 0) = 0.16 In general, P(Xn+3 = 1 | Xn = 0) = 0.16 for any n.

Question 5 • A markov chain X0,X1,…on state 0, 1, 2 has the transition • probability matrix • It is known that the process starts in state X0 = 1, determine • probability P(X2 = 2).

Question 5 • Note that: • P(X2 = 2) = P(X0 = 0) x P(X2 = 2 | X0 = 0) • +P(X0 = 1) x P(X2 = 2 | X0 = 1) • +P(X0 = 2) x P(X2 = 2 | X0 = 2) • = p0p02 + p1p12 + p2p22 • = 1 x p12 = 0.35

Question 6 • Consider a sequence of items from a production process, with each item being graded as good or defective. • Suppose that a good item is followed by another good item with probability and by a defective item with probability 1-. • Similarly, a defective item is followed by another defective item with probability and by a good item with probability 1-. • Specify the transition probability matrix. • If the first item is good, what is the probability that the first defective item to appear is the fifth item?

Question 6 Let Xnbe the grade of the nth product. P(Xn+1 = g | Xn= g) = , P(Xn+1 = d | Xn= g) = 1 - P(Xn+1 = d | Xn= d) = , P(Xn+1 = g | Xn= d) = 1 -  Thus, the transition probability matrix is

Question 7 The random variables 1, 2, ... are independent and with common probability mass function Set X0 = 0 and let Xn= max {1, 2, ... }. • Determine the transition probability matrix for the MC {Xn}. • Draw the state-diagram associated with transition probability

Question 7 Observe: X0 = 0, X1 = max {X0, 1}, X2 = max {X1, 2}, … Xn= max {Xn-1, n} Hence, Xn recursively compares the previous maximum and the current input to obtain the new maximum.

Question 7 The state space is S = {0, 1, 2, 3} P(Xn+1 = 0 | Xn= 0) = P(n+1 = 0)= 0.1 P(Xn+1 = 1 | Xn= 0) = P(n+1 = 1)= 0.3 P(Xn+1 = 2 | Xn= 0) = 0.2 P(Xn+1 = 3 | Xn= 0) = 0.4 P(Xn+1 = 1 | Xn= 1) = P(n+1 = 0) + P(n+1 = 1) = 0.1 + 0.3 = 0.4 P(Xn+1 = 2 | Xn= 1) = 0.2 … P(Xn+1 = 2 | Xn= 2) = 0.1 + 0.3 + 0.2 = 0.6 … P(Xn+1 = 3 | Xn= 3) = 0.1 + 0.3 + 0.2 + 0.4 = 1 … P(Xn+1 = j | Xn= i) = 0 if j < i. (Cannot Happen!)

Question 7 The transition probability matrix is

ST3236: Stochastic Process Tutorial 3