1 / 65

Chapter 3 Stopping time & martingales

Chapter 3 Stopping time & martingales. Prof. Bob Li. 3.1 Stopping time & Wald’s Equation. Prof. Bob Li. Sum of randomly many i.i.d. Theorem 1 . Let X 1 , X 2 , ... , X n , ... be i.i.d. and N  0 be a discrete random variable that is independent of all X n . Then,

aoife
Download Presentation

Chapter 3 Stopping time & martingales

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Chapter 3Stopping time & martingales Prof. Bob Li

  2. 3.1Stopping time & Wald’s Equation Prof. Bob Li

  3. Sum of randomly many i.i.d. • Theorem 1. Let X1, X2, ... , Xn, ... be i.i.d. and N  0 be a discrete random variable that is independent of all Xn. Then, • // Think of N as the number of customers to a shop • // and Xn as the amount spent by the nth customer. • Proof. • // Conditioning on N • // N is independent of all Xn. • // Mean of sum is sum of means. • = EX1 EN

  4. Waiting time in die rolling Q. Roll a die repeatedly until the side “4” shows up. How many rolls does it take on the average? Let N  1 represent the waiting time for “4”. Calculative Solution. P(N = n) = (5/6)n1(1/6) // Geometric1(1/6) distribution EN = 6 Intuition speculates that EN is simply the reciprocalof this probability 1/6. Prof. Bob Li

  5. Waiting time in die rolling Q. Roll a die repeatedly until the side “4” shows up. How many rolls does it take on the average? Intuitive solution. Let Xn be the characteristic r.v. for the event of getting “4” on the nthroll. Thus X1, X2, ... , Xn, ... are i.i.d. and EXn = P{Xn = 1} = 1/6. Intuition speculates that EN is simply the reciprocalof this probability 1/6. Could Theorem 1 in the above apply so that EN EX1 = EN / 6 • No, because the theoremrequires the independence of N from all Xn. • So, is this intuitive speculation incorrect or is Theorem 1 too restrictive? • The verdict: Theorem 1 is too restrictive and we shall water down the required independence between N and Xn. • This calls for the concept of stopping time.

  6. Stopping time • Definition. Consider a random sequence X1, X2, ... , Xn, ... A nonnegative integer r.v. N is called a stopping time (or stopping rule) of this sequence if • For every n, the event {N n} is independent of the r.v. Xn. • // That is, the characteristic r.v. of the event {N n} is indep. of Xn. • // Equivalently, the events {N n} and {Xn x} are indep for all x. • Intuition. Xnrepresents the outcome of the nth experiment. • Right before the nth experiment, one has to decide whether to stop before seeing how Xnturns out. • The decision, though, may depend on the past knowledge X1, X2, ... , Xn1. Prof. Bob LI

  7. Examples and counterexamples • Examples of stopping times • Roll a die until the side “4” appears. • Gamble $1 at a time at a casino until midnight or losing all money. • Gamble $100 at a time at a casino until there is only $100 left. • Examples of nonstopping times • Gamble $100 at a time at a casino until right before losing the last $100. • Gamble $1 at a time at a casino until right before my luck turns bad. Prof. Bob LI

  8. Stopping time • For every n, the event {N n} is independent of the r.v. Xn. • The event, however, may depend on the past knowledge X1, X2, ... , Xn1. • Q. Can the event {N n} depend on the remoter future Xn+1, Xn+2, …? • Not really. Suppose the event {N n} depends on Xn+1. This event is a prerequisite to event {N n+1}, which has to be independent of Xn+1. A contradiction. • Alternative definition. A nonnegative integer random variable N is called a stopping time of this sequence X1, X2, ... , Xn, ... if • For every n, the event {N n} is independent of the r.v. Xn, Xn+1, Xn+2, … • // That is, the decision on when to stop is causal. Prof. Bob LI

  9. Abraham Wald Wald’s equation Theorem (Wald's Equation). Let X1, X2, ... , Xn, ... be i.i.d. with a stopping time N with P(N < ) = 1. If EN <  or EX1 0, then // The condition P(N < ) = 1 clarifies the meaning of Proof. Let Ynbe the characteristic r.v. of the event {N n}. • Observe that: • because XnYn = Xn when n N • = 0 otherwise • N is a stopping time.  Ynand Xn are independent. •  E[XnYn] = EXn EYn • … • This removes the randomness in the upper bound of the summation.

  10. Wald’s equation (cont’d) // Yn = characteristic r.v. of event {N n}. Hence <  or EX1 0. // Mean of sum is sum of means. // Xn and Yn are indep. // Identical distribution // <  or EX1 0 = EX1 EN

  11. Gambler’s ruin at casino Q. A gambler starts with $100 and intends to keep betting $1 on ODD at the roulette until he loses all. How long does this process take on the average? Answer. Let Xn be the net winning on the n-th bet. Then, X1, X2, ... , Xn, ... are i.i.d. and Xn= Thus, EXn= P(Xn= 1)  P(Xn= 1) = 1/19. Let the random variable N represent the number of bets till losing all. Clearly, N is a stopping time for the process X1, X2, ... , Xn, ... By Wald’s equation, Hence EN = 1900 < . Prof. Bob LI

  12. Symmetric random walk is not positive recurrent Theorem. Symmetric 1-dim free random walk is not positive recurrent. // Previously a rough proof was provided. Proof. Let Yn be the winning on the n-th game in fair gamble. This defines i.i.d. // The symmetric 1-dim free random walk is represented // by the markov chain X0, X1, ... , where Xn = j≤nYj. Let T01 be the waiting time for state 1 given X0 = 0. Clearly T01 is a stopping time for the i.i.d. sequence. If ET01were finite, Wald's Equation would have yielded the following contradiction: 1 = E [ ] = ET01 EY1 = ET01 0 = 0 This forces ET01= . Similarly, ET10 =  = ET1,0. Thus ET00 = 1 + ET10 / 2 + ET1,0 / 2 =  // Condition on the first move

  13. A board game Each move advances the token by a number of steps determined by rolling a die. Let pn = P{ever landing on square #n}. Thus, p1 = 1/6 p2 = 1/6 + 1/36 = 7/36 … Prof. Bob Li

  14. For what n is pnthe largest? By conditioning on the first roll, we arrive at a recursion: pn= 1k6 P(1st roll = k)  P(ever landing on n | 1st roll = k) = 1k6 P(ever landing on n | 1st roll = k) /6 = (pn1+ … + pn6) /6 That is, every pnis the average of its 6 immediate predecessors, n  1. // The above recursion is a 6th-order difference equation. // The 6 boundary conditions: p0= 1, p-1 = p-2 = p-3 = p-4 = p-5 = 0 // In this course, we see difference equations quite often. Prof. Bob Li

  15. For what n is pnthe largest? The largest among p1 to p6 is p6. Because every probability mass pn is the average of its six immediate predecessors, an easy induction finds pn<p6 for all n > 6.

  16. limn pn = ? Intuitive solution. As n gets very large, the value of pn should have little correlation with n. Thus, limnpnexists. The average value per roll of the die is 7/2. In the long run, 2 out of every 7 squares should be landed. Therefore limnpn= 2/7. Below, we articulate the intuition in rigor byWald’s Equation. Homework.If we roll two dice instead of just one, show that limn pn = 1/7. Prof. Bob LI

  17. To articulate the intuition in rigor Denote • Yn= characteristic r.v. of the event {Ever landing on n} //EYn = P{Yn= 1} = pn • Tk = waiting time until the square k is passed = Y1 + Y2 + … + Yk+ 1 // Count landed squares. • Xn= outcome the nth roll k+1  X1 + X2 + … + k+6 // Count both landed and passed squares. Clearly Tk is a stopping time for the process X1, X2, … , Xn, … E[X1 + X2 + … + ] = ETk EX1 // Wald's Equation = (EY1+ … + EYk + 1)7/2// Mean of sum is sum of means. = (p1 + … + pk+ 1)7/2 k+1  (p1 + … + pk+ 1)7/2 k+6 or  • p1+ p2 + … + pk • 2+5/k 25/k k 7 7 p1 + p2 + … + pk limn pn = limk= 2/7 // Law of large number k

  18. 3.2 Martingales Prof. Bob LI

  19. Definition of a martingale Definition. A process X1, X2, ... , Xn, where n , is called a martingaleif, for all k, E|Xk| <  and E[Xk+1 | Xk = xk, … , X1 = x1] = xk // Some literature abbreviates this as E[Xk+1 | Xk, … , X1] = Xk Example. Xk = Net cumulative winning after k games in fairgamble. Opposite concept. When you gamble on casino roulette, E[Xk+1 | Xk = xk, … , X1 = x1] <xk Such a process X1, X2, ... , Xn, ... is then called a “super-martingale,” anironical term coined by Doob. Prof. Bob LI

  20. History of martingale The word martingalecame from a middle-age French game. It was first related to the concept of probability through a betting system in the 18th century. • Paul Levy(1886~1971) formulated the mathematical concept. • The intuitive fact of • E[Net gain in fair gamble upon stopping] = 0 • is the Martingale Stopping Theorem, first formulated as • by Joseph Doob. Prof. Bob LI

  21. Martingale Stopping Theorem Theorem. If N is a stopping time for a martingale {Zn}n1, then EZN= EZ1, provided one of the following conditions is satisfied: (i) There exist a and b with a < Zn <b whenever n N. (ii) N is bounded. (iii) EN <  and there exists M < such that E[|Zn+1− Zn| |Z1, …, Zn]< M. Proof. Omitted. Prof. Bob LI

  22. The need of the condition(i), (ii) or (iii) The the condition (i), (ii) or (iii) in the Martingale Stopping Theorem is not superfluous because of the following example. Abel initially bets $1 on Head in fair coin-toss. Upon any winning, he would parley on the next coin toss. The process continues untilhe loses. // Equivalently, the opponent is a progressive gambler. Let Xk = cumulative net winning after k games in this fair gamble. Thus X0 = 0, X1, ... , Xn,...form a martingale. The stopping time N is Geometric1(1/2) distributed, and EN= 2 < . Abel’s net winning upon stopping is XN = 1 deterministically. Hence EXN ≠ EX0. The Martingale Stopping Theorem does notapply! Prof. Bob LI

  23. Martingale Stopping Thm  Wald’s equation Wald's Equation.Let X1, X2, ... , Xn, ... be i.i.d. with a stopping time N. If EN < , then Proof by Martingale Stopping Theorem. WriteXn = XnEX1. // Neutralize the mean of Xn Think of Xnas fair gamble. A martingale is given by successive accumulation of gains: 0, X1, X1 + X2, …, X1 + X2 + … + Xn, ... From the Martingale Stopping Theorem, 0 = E[X1 + X2 + … + XN] = E[X1 + X2 + … + X N]  EN EX1 Prof. Bob LI

  24. Gambler’s random walk A gambler in Macau needs $n for boat fare to go home but has only $i. So he gambles. The bet is $1 and the probability of winning is p every game. Denote pi= P{Will reach $n | starting at $i}. Recall thatlong calculation process by markov chain yields: We now derive this by a martingale with almost no calculation.

  25. Instant calculation by martingale The random walk is not a martingale. Labels of states are in an arithmetic progression. Deploy a geometric progression instead with the common ratio r= = odds per game. Prof. Bob LI

  26. Instant calculation by martingale The random walk is not a martingale. Labels of states are in an arithmetic progression. Deploy a geometric progression instead with the common ratio r= = odds per game. By symmetry,we may assume that p < q. $rk1, the loss is $(rkrk1) From $rk to $rk+1, the gain is $(rk+1rk) = $r(rkrk1) The random walk becomes a martingale X0 = ri, X1, ..., Xn, ...

  27. Instant calculation by martingale The time T till the gambler reaches either end is a stopping time. From the Martingale Stopping Theorem, ri=EXT= (1 pi)1 + pi rn  Prof. Bob LI

  28. Possible applications of this martingale r = = odds Financial engineering.Take rin this martingaleto be the rate of interest, inflation, depreciation, … Information engineering.rcan be, for instance, the attenuation of signal strength. Prof. Bob LI

  29. Lecture 5 Feb. 20, 2013 Prof. Bob Li

  30. Expected waiting time We have solved by either markov chain or martingale. Let W= waiting time from state i till reaching either end = a stopping time of the random walk process Q:EW = ? Prof. Bob LI

  31. Expected waiting time • Without loss of generality, we shall assume that p< q. • // Symmetric random walk is treated as the limiting case when p ½. • Up to the stopping time W, the random walk is an i.i.d. process. • One way to convert an i.i.d. process into a martingale is by neutralizing the mean of every r.v. in the process. • // Recall this technique in the proof of: • // Martingale Stopping Theorem Wald’s Equation • In terms of gambling, a compensation for the gambler in the amount of $(qp) per bet turns the random walk into a martingale. Prof. Bob LI

  32. Expected waiting time • The total compensation up to the stopping time is (qp)W. • By the Martingale Stopping Thm, the expected net gain upon stopping is 0. • E[pi n + (1pi ) 0 + (qp)W] = i • n+ (qp) EW= i EW Prof. Bob LI

  33. Betting strategy pertaining to martingale concept In the NBA championship, a “best 4 out of 7” series is played between Miami and Dallas. Assume that every game is a toss-up and a friendly bookie accepts bets at even odds on every single game. Your boss hands you $10,000 to place bets, and you must return $20,000 to him if Miami wins the series. What betting strategy lets you accomplish this mission? Solution. The game-by-game result of the series can be described by a binary tree where every branching point represents the difference between two outcomes of a game. Let M denote the characteristic r.v. of the eventthat Miami eventually wins the series. This event would cost you $20,000. Initially, your net$ amount is represented by the r.v. 10000  20000M Prof. Bob LI

  34. Betting strategy pertaining to martingale concept • Solution. … If, for example, the initial bet is $3,000 on Miami, then your net fortune after one game becomes the conditional r.v. • 13000  20000(M | Miami wins first game) or • 7000  20000(M | Miami loses first game) • In general, your net fortune, given any partial outcome of the series, is represented by the conditional r.v. • An amount of cash  20000(M | Given the partial outcome) • Again, this conditional r.v. depends on your betting strategy. • Upon stopping of the championship series,the conditional r.v. of M becomes deterministic: • (M | Miami wins series)= 1 deterministically • (M | Miami loses series) = 0 deterministically

  35. Betting strategy pertaining to martingale concept • Solution. … That is, your fortune upon stopping is: • An amount of cash  20000 if Miami wins series • Just an amount of cash otherwise • The strategy is to assure that every possible stopping scenario is nonnegative. • First, we calculate the right amount, say, $J of initial bet. Your fortune after one game will then be • 10000 + J  20000(M | Miami wins first game) or • 10000  J  20000(M | Miami loses first game) • If you J > 3125 (resp. < 3125), than the second (resp. first) between these two conditional r.v. would have negative mean. Once your fortune has a negative mean, there would be no way to ensure nonnegative outcome in all subsequent ending scenarios as the gamble is fair. Prof. Bob LI

  36. Betting strategy pertaining to martingale concept Solution. … This forces J = 3125, which gives 0 mean to both conditional r.v. In a similar fashion, you can calculate the right amount to bet after Miami has won (resp. lost) the first game. It is not hard to see that the only way to ensure nonnegative mean in all scenarios is to have 0 mean in all scenarios. // This is similar to martingale stopping theorem. And so on. Prof. Bob LI

  37. 3.3 Martingales stopping of the gambling team Prof. Bob LI

  38. A pattern with non-binary non-uniform probabilities • Roll a special die repeatedly, where • P{a} = 1/2 • P{b} = 1/3 • P{c} = 1/6

  39. A pattern with non-binary non-uniform probabilities • Roll a special die repeatedly, where • P{a} = 1/2 • P{b} = 1/3 • P{c} = 1/6 • The average waiting time for the pattern aba= ? S.-Y. R. Li, “A martingale approach to the study of occurrence of sequence patterns in repeated experiments,” Annals of Probability, v.8, pp 1171-1176, 1980.

  40. A pattern with non-binary non-uniform probabilities • Roll a special die repeatedly, where • P{a} = 1/2 • P{b} = 1/3 • P{c} = 1/6 • The average waiting time for the pattern aba= • Howis this calculated and why so? • 14 S.-Y. R. Li, “A martingale approach to the study of occurrence of sequence patterns in repeated experiments,” Annals of Probability, v.8, pp 1171-1176, 1980.

  41. a b a Howto calculate average waiting time? Formula. The average waiting time for the pattern aba is 3 2+ P{a} 1+ P{b} P{a} Calculating 3 bits: 1= 2= 3= 1 0 a b a 1

  42. Howto calculate average waiting time? Formula. The average waiting time for the pattern aba is 1 Worth 12 0+ 1/2 1+ Worth 2 1/3 = 14 1/2 1= 2= 3= 1 0 1

  43. Justification by an artificial casino A gambler arrives with $1 and bets on the pattern aba by rolling the special die: • If a appears on the 1st roll, he receives $2 in total. • Then, parlay on the 2nd roll. • If b appears, he receives $6 in total. • Then, parlay on the 3rd roll. • If a appears again, he receives $12 in total. Game over. This isfair gamble! Prof. Bob Li

  44. Gambling teamenters the artificial casino Prof. Bob Li

  45. Gambling teamenters the artificial casino = 1 = 2 Y1 = a Prof. Bob Li

  46. Gambling teamenters the artificial casino = 1 = 0 = 0 = 2 Y1 = a Y2 = c Prof. Bob Li

  47. Gambling teamenters the artificial casino = 1 = 1 = 1 = 1 = 0 = 0 = 1 = 0 = 2 = 0 = 2 = 2 = 6 = 12 = 0 Y1 = a Y2 = c Y3 = b Y4 = a Y5 = a Y6 = b Y7 = a Total receipt = 12 + 0 + 2 regardless of the outcome of die rolling. Net gain of the team upon stopping = 12 + 0 + 2 – N, where the r.v. N represents the waiting time for the pattern aba. Because this is fair gamble, 0 = E[Net gain upon stopping] = 14 – EN EN= 14

  48. Average waiting time for a pattern Formula. The average waiting time for the pattern aba is 3rd last gambler has $12 in the end 1 0+ 1/2 Last gambler has $2 in the end 1+ 1/3 = 14 1/2 Prof. Bob Li

  49. Generalizing the example m m-1+ P{Y=bm} m-2+ … 1 + P{Y=b2} P{Y=b1} Formula. For a pattern B = b1b2…bm, calculate j as before. Then, The average waiting time for B = Prof. Bob Li

  50. The correlation operator  Example. The average waiting time for the pattern aba is 1 0+ 1/2 1+ 1/3 abaaba = = 14 1/2 1= 2= 3=

More Related