200 likes | 460 Views
Introduction to Discrete-Time Markov Chain. Motivation. many dependent systems, e.g., inventory across periods state of a machine customers unserved in a distribution system. excellent. good. fair. bad. time. Motivation. any nice limiting results for dependent X n ’s?
E N D
Motivation • many dependent systems, e.g., • inventory across periods • state of a machine • customers unserved in a distribution system excellent good fair bad time 2
Motivation • any nice limiting results for dependent Xn’s? • no such result for general dependent Xn’s • nice results when Xn’s form a discrete-timeMarkov Chain 3
Discrete-Time, Discrete-State Stochastic Process • a stochastic process: a sequence of indexed random variables, e.g., {Xn}, {X(t)} • a discrete-time stochastic process: {Xn} • a discrete-state stochastic process, e.g., • state {excellent, good, fair, bad} • set of states {e, g, f, b} {1, 2, 3, 4} {0, 1, 2, 3} • state to describe weather {windy, rainy, cloudy, sunny} 4
Markov Property • a discrete-time, discrete-state stochastic process possesses the Markov property if • P{Xn+1 = j|Xn= i, Xn−1 = in−1, . . . , X1 = i1, X0 = i0} = pij, for alli0, i1, …, in1, in, i, j, n 0 • time frame: presence n, future n+1, past {i0, i1, …, in1} • meaning of the statement: given presence, the past and the future are conditionally independent • the past and the future are certainly dependent 5
One-Step Transition Probability Matrix • pij 0, i, j 0, 6
Example 4-1 Forecasting the Weather • state {rain, not rain} • dynamics of the system • rains today rains tomorrow w.p. • does not rain today rains tomorrow w.p. • weather of the system across the days, {Xn} 7
Example 4-3 The Mood of a Person • mood {cheerful (C),so-so (S), or glum (G)} • cheerful today C, S, or G tomorrow w.p. 0.5, 0.4, 0.1 • so-so today C, S,or G tomorrow w.p. 0.3, 0.4, 0.3 • glum today C, S, or G tomorrow w.p. 0.2, 0.3, 0.5 • Xn: mood on the nth day, such that mood {C, S, G} • {Xn}: a 3-state Markov chain (state 0 = C, state 1 = S, state 2 = G) 8
Example 4.5A Random Walk Model • a discrete-time Markov chain of number of states {…, -2, -1, 0, 1, 2, …} • random walk: for 0 < p < 1, • pi,i+1 = p = 1 − pi,i−1, i = 0, 1, . . . 9
Example 4.6A Gambling Model • each play of a game a gambler gaining $1 w.p. p, and losing $1 o.w. • end of the game: a gambler either broken or accumulating $N • transition probabilities: • pi,i+1= p = 1 − pi,i−1, i = 1, 2, . . . , N − 1; p00= pNN= 1 • example for N = 4 • state: Xn, the gambler’s fortune after the n play {0, 1, 2, 3, 4} 10
Limiting Behavior of a Positive Irreducible Chain • cost of a visit • state 1 = $5 • state 2 = $8 • what is the long-run cost of the above DTMC? 0.1 0.9 0.2 0.8 2 1 12
Limiting Behavior of a Positive Irreducible Chain • j = fraction of time at state j • N: a very large positive integer • # of periods at state j j N • balance of flow • j N i (i N)pij j= i ipij 13
Limiting Behavior of a Positive Irreducible Chain • j = fraction of time at state j • j= i ipij • 1= 0.91 + 0.22 • 2= 0.11 + 0.82 • linearly dependent • normalization equation: 1+ 2= 1 • solving: 1= 2/3, 2= 1/3 C 0.1 0.9 2 1 0.2 0.8 14
Limiting Behavior of a Positive Irreducible Chain • 1 = 0.752 + 0.013 • 3 = 0.252 • 1 + 2 + 3 = 1 • 1 = 301/801, 2 = 400/801, 3 = 100/801 1 0.25 3 1 2 0.75 0.99 0.01 15
Limiting Behavior of a Positive Irreducible Chain • an irreducible DTMC {Xn} is positive there exists a unique nonnegative solution to • j: stationary (steady-state) distribution of {Xn} 16
Limiting Behavior of a Positive Irreducible Chain • j = fraction of time at state j • j = fraction of expected time at state j • average cost • cj for each visit at state j • random i.i.d. Cj for each visit at state j • for aperiodic chain: 17
Limiting Behavior of a Positive Irreducible Chain • 1 = 301/801, 2 = 400/801, 3 = 100/801 • profit per state: c1 = 4, c2 = 8, c3 = -2 • average profit 1 0.25 3 1 2 0.75 0.99 0.01 18
Limiting Behavior of a Positive Irreducible Chain • 1 = 301/801, 2 = 400/801, 3 = 100/801 • C1 ~ unif[0, 8], C2 ~ Geo(1/8), C3 = -4 w.p. 0.5; and = 0 w.p. 0.5 • E(C1) = 4, E(C2) = 8, E(C3) = -2 • average profit 1 0.25 3 1 2 0.75 0.99 0.01 19