1 / 43

n -Player Stochastic Games with Additive Transitions

n -Player Stochastic Games with Additive Transitions. Frank Thuijsman János Flesch & Koos Vrieze Maastricht University. European Journal of Operational Research 179 (2007) 483–497. Outline. Model Brief History of Stochastic Games Additive Transitions Examples. Finite Stochastic Game.

alder
Download Presentation

n -Player Stochastic Games with Additive Transitions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. n-Player Stochastic Gameswith Additive Transitions Frank ThuijsmanJános Flesch & Koos VriezeMaastricht University European Journal of Operational Research 179 (2007) 483–497 Center for the Study of Rationality Hebrew University of Jerusalem

  2. Outline • Model • Brief History of Stochastic Games • Additive Transitions • Examples Center for the Study of Rationality Hebrew University of Jerusalem

  3. Finite Stochastic Game as = (a1s , a2s ,  , ans) joint action rs(as) = (r1s(as), r2s(as),  , rns(as)) rewards ps(as) = (ps(1|as), ps(2|as),  , ps(z|as)) transitions 1 s z • Infinite horizon • Complete Information • Perfect Recall • Independent and Simultaneous Choices rs(as) ps(as) Center for the Study of Rationality Hebrew University of Jerusalem

  4. 3-Player Stochastic Game F 3 0, 0, 0 0, 0, 0 N L 2 R (2/3, 0, 0, 1/3) (1/3, 0, 1/3, 1/3) 0, 0, 0 0, 0, 0 0, 0, 0 0, 0, 0 T (1, 0, 0, 0) (2/3, 0, 1/3, 0) 1 (0, 1/3, 1/3, 1/3) (1/3, 1/3, 0, 1/3) 0, 0, 0 0, 0, 0 1 B (2/3, 1/3, 0, 0) (1/3, 1/3, 1/3, 0) 1, 3, 0 0, 3, 1 3, 0, 1 (0, 1, 0, 0) (0, 0, 1, 0) (0, 0, 0, 1) 2 3 4 Center for the Study of Rationality Hebrew University of Jerusalem

  5. Strategies general strategy i : N×S ×H→Xi (k, s, h) →Xis Markov strategy fi : N×S→Xi (k, s) →Xis stationary strategy xi : S→Xi (s) →Xis opponents’ strategy -i , f-i and x-i mixed actions Center for the Study of Rationality Hebrew University of Jerusalem

  6. Rewards -Discounted rewards (with 0 << 1)  is() = Es((1-) k k-1Rik) Limiting average rewards  is() = Es(limK→K-1Kk=1Rik) Center for the Study of Rationality Hebrew University of Jerusalem

  7. MinMax Values -Discounted minmax vis = inf -isup  i is() Limiting average minmax vis = inf -isup  i is() Highest rewards player i can defend Center for the Study of Rationality Hebrew University of Jerusalem

  8. -Equilibrium  = (i)iN is an -equilibrium if  is(i, -i) ≤is() +  for all i, for all i and for all s. Center for the Study of Rationality Hebrew University of Jerusalem

  9. Question Any -equilibrium? 0, 0 (0, 0, 0, 1) 0, 0 3, -1 0, 0 0, 0 2,1 (0, 0.5, 0.5, 0) (1, 0, 0, 0) (0, 1, 0, 0) (0, 0, 1, 0) (0, 0, 0, 1) 2 1 3 4 Center for the Study of Rationality Hebrew University of Jerusalem

  10. Highlights from (Finite)Stochastic Games History Shapley, 1953 0-sum, “discounted” Everett, 1957 0-sum, recursive, undiscounted Gillette, 1957 0-sum, big match problem Center for the Study of Rationality Hebrew University of Jerusalem

  11. Highlights from (Finite)Stochastic Games History Fink, 1964 & Takahashi, 1964 n-player, discounted Blackwell & Ferguson, 1968 0-sum, big match solution Liggett & Lippmann, 1969 0-sum, perfect inf., undiscounted Center for the Study of Rationality Hebrew University of Jerusalem

  12. Highlights from (Finite)Stochastic Games History Kohlberg, 1974 0-sum, absorbing, undiscounted Mertens & Neyman, 1981 0-sum, undiscounted Sorin, 1986 2-player, Paris Match, undiscounted Center for the Study of Rationality Hebrew University of Jerusalem

  13. Highlights from (Finite)Stochastic Games History Vrieze & Thuijsman, 1989 2-player, absorbing, undiscounted Thuijsman & Raghavan, 1997 n-player, perfect inf., undiscounted Flesch, Thuijsman, Vrieze, 1997 3-player, absorbing example, undiscounted Center for the Study of Rationality Hebrew University of Jerusalem

  14. Highlights from (Finite)Stochastic Games History Solan, 1999 3-player, absorbing, undiscounted Vieille, 2000 2-player, undiscounted Solan & Vieille, 2001 n-player, quitting, undiscounted Center for the Study of Rationality Hebrew University of Jerusalem

  15. Additive Transitions ps(as) = ni=1is pis(ais) pis(ais) transition probabilities controlled by player i in state s istransition power of player i in state s 0 ≤is ≤1 and iis = 1 for each s Center for the Study of Rationality Hebrew University of Jerusalem

  16. Example for 2-PlayerAdditive Transitions ps(as) = ni=1is pis(ais) 21 = 0.7 p21(1)=(1, 0, 0) p21(2)=(0, 1, 0) 11 = 0.3 (1, 0, 0) (0.3, 0.7, 0) p11(1) = (1, 0, 0) p11(2) = (0, 0, 1) (0.7, 0, 0.3) (0, 0.7, 0.3) 1 Center for the Study of Rationality Hebrew University of Jerusalem

  17. Results • 0-equilibria for n-player AT games (threats!) • 0-opt. stationary strat. for 0-sum AT games • Stat. -equilibria for 2-player abs. AT games • Result 3 can not be strengthened, neither to 3-player abs. AT games, nor to 2-player non-abs. AT games, nor to give stat. 0-equilibria Center for the Study of Rationality Hebrew University of Jerusalem

  18. The Essential Observation Additive Transitions induce a Complete Ordering of the Actions If ais is “better” than bis against some strategy, Then ais is “better” than bis against any strategy. Center for the Study of Rationality Hebrew University of Jerusalem

  19. “Better” Consider strategies ais and bis for player i If, for some strategya-iswe have t Sps(t | ais , a-is ) vit ≥ t Sps(t | bis , a-is ) vit Then for all strategies b-iswe have t Sps(t | ais , b-is ) vit ≥ t Sps(t | bis , b-is ) vit Center for the Study of Rationality Hebrew University of Jerusalem

  20. Because …. If t Sps(t | ais , a-is)vit ≥ t Sps(t | bis , a-is)vit Then ist Sps(t | ais) vit + j  ijst Sps(t | a-js) vit ≥ ist Sps(t | bis) vit + j  ijst Sps(t | a-js) vit Center for the Study of Rationality Hebrew University of Jerusalem

  21. which implies that …. ist Sps(t | ais) vit + j  ijst Sps(t | b-js) vit ≥ ist Sps(t | bis) vit + j  ijst Sps(t | b-js) vit And therefore t Sps(t | ais , b-is)vit ≥ t Sps(t | bis , b-is)vit Center for the Study of Rationality Hebrew University of Jerusalem

  22. “Best” The “best” actions for player i in state s are those that maximize the expression t Sps(t | ais) vit Center for the Study of Rationality Hebrew University of Jerusalem

  23. The Restricted Game Let G be the original AT game and let G* be the restricted AT game, where each player is restricted to his “best” actions. Center for the Study of Rationality Hebrew University of Jerusalem

  24. The Restricted Game Now v*i≥ vi for each player i. In G* : t Sps(t | a*s) v*it = v*isi, s, a*s In G : t Sps(t | bis , a*-is ) vit < visi, s, a*-is , bis Center for the Study of Rationality Hebrew University of Jerusalem

  25. The Restricted Game If x*iyields at least v*i in G*, then x*iyields at least v*i in G as well. Center for the Study of Rationality Hebrew University of Jerusalem

  26. Ex. 1: 2-Player Absorbing AT Game 0.5 (1, 0, 0) (0, 1, 0) 0, 0 0, 0 (1, 0, 0) (1, 0, 0) (0.5, 0.5, 0) -3, 1 -1, 3 0, 0 0, 0 0.5 (0.5, 0, 0.5) (0, 0, 1) (0, 0.5, 0.5) (0, 1, 0) (0, 0, 1) 2 1 3 Center for the Study of Rationality Hebrew University of Jerusalem

  27. Ex. 1: 2-Player Absorbing AT Game NO stationary 0-equilibrium 0, 0 0, 0 (1, 0, 0) (0.5, 0.5, 0) -3, 1 -1, 3 0, 0 0, 0 (0.5, 0, 0.5) (0, 0.5, 0.5) (0, 1, 0) (0, 0, 1) 2 1 3 B, L T, L T, L T, R B, R -1, 3 0, 0 0, 0 -3, 1 -2, 2 Center for the Study of Rationality Hebrew University of Jerusalem

  28. Ex. 1: 2-Player Absorbing AT Game 1-/2 /2 stationary -equilibrium with  >0 0, 0 0, 0 0 (1, 0, 0) (0.5, 0.5, 0) -3, 1 -1, 3 0, 0 0, 0 1 (0.5, 0, 0.5) (0, 0.5, 0.5) (0, 1, 0) (0, 0, 1) 2 1 3 equilibrium rewards ≈ ((-1-, 3-), (-3,1), (-1,3)) Center for the Study of Rationality Hebrew University of Jerusalem

  29. Ex. 1: 2-Player Absorbing AT Game non-stationary 0-equilibrium Player 1: B, T, T, T, …. Player 2: R, R, R, R, …. 0, 0 0, 0 (1, 0, 0) (0.5, 0.5, 0) -3, 1 -1, 3 0, 0 0, 0 (0.5, 0, 0.5) (0, 0.5, 0.5) (0, 1, 0) (0, 0, 1) 2 1 3 equilibrium rewards ((-2, 2), (-3, 1), (-1, 3)) Center for the Study of Rationality Hebrew University of Jerusalem

  30. Ex. 2: 2-Player Non-Absorbing AT Game NO stationary -equilibrium with  >0 0, 0 p q 1- q (0, 0, 0, 1) 0, 0 3, -1 0, 0 0, 0 2,1 1- p (0, 0.5, 0.5, 0) (1, 0, 0, 0) (0, 1, 0, 0) (0, 0, 1, 0) (0, 0, 0, 1) 2 1 3 4 p > 1 - q > 0 p <  q = 0 q > 0 Center for the Study of Rationality Hebrew University of Jerusalem

  31. Ex. 2: 2-Player Non-Absorbing AT Game non-stationary 0-equilibrium Player 1: T, B, B, B, …. Player 2: R, R, R, R, …. 0, 0 (0, 0, 0, 1) 0, 0 3, -1 0, 0 0, 0 2,1 (0, 0.5, 0.5, 0) (1, 0, 0, 0) (0, 1, 0, 0) (0, 0, 1, 0) (0, 0, 0, 1) 2 1 3 4 equilibrium rewards ((2, 1), (0, 0), (3, -1), 2, 1)) Center for the Study of Rationality Hebrew University of Jerusalem

  32. Ex. 2: 2-Player Non-Absorbing AT Game alternative 0-equilibrium Player 1: T, T, T, T, …. Player 2: L, R, R, R, …. 0, 0 (0, 0, 0, 1) 0, 0 3, -1 0, 0 0, 0 2,1 (0, 0.5, 0.5, 0) (1, 0, 0, 0) (0, 1, 0, 0) (0, 0, 1, 0) (0, 0, 0, 1) 2 1 3 4 equilibrium rewards ((2, 1), (2, 1), (3, -1), 2, 1)) Center for the Study of Rationality Hebrew University of Jerusalem

  33. Ex. 3: 3-Player Absorbing AT Game F 3 0, 0, 0 0, 0, 0 N L 2 R (2/3, 0, 0, 1/3) (1/3, 0, 1/3, 1/3) 0, 0, 0 0, 0, 0 0, 0, 0 0, 0, 0 T (1, 0, 0, 0) (2/3, 0, 1/3, 0) 1 (0, 1/3, 1/3, 1/3) (1/3, 1/3, 0, 1/3) 0, 0, 0 0, 0, 0 1 B (2/3, 1/3, 0, 0) (1/3, 1/3, 1/3, 0) 1, 3, 0 0, 1, 3 3, 0, 1 How to share 4 among three people if only few solutions are allowed? (0, 1, 0, 0) (0, 0, 1, 0) (0, 0, 0, 1) 2 3 4 Center for the Study of Rationality Hebrew University of Jerusalem

  34. Ex. 3: 3-Player Absorbing AT Game F 3/2, 1/2, 2 3, 0, 1 N L R (2/3, 0, 0, 1/3) (1/3, 0, 1/3, 1/3) 0, 0, 0 0, 1, 3 4/3, 4/3, 4/3 2, 3/2, 1/2 T (1, 0, 0, 0) (2/3, 0, 1/3, 0) (0, 1/3, 1/3, 1/3) (1/3, 1/3, 0, 1/3) 1/2, 2, 3/2 1, 3, 0 1 B (2/3, 1/3, 0, 0) (1/3, 1/3, 1/3, 0) 1, 3, 0 0, 1, 3 3, 0, 1 (0, 1, 0, 0) (0, 0, 1, 0) (0, 0, 0, 1) 2 3 4 Center for the Study of Rationality Hebrew University of Jerusalem

  35. Ex. 3: 3-Player Absorbing AT Game F 3/2, 1/2, 2 3, 0, 1 3, 0, 1 N L R 1/3 * 2/3 * 0, 0, 0 0, 1, 3 0, 1, 3 4/3, 4/3, 4/3 2, 3/2, 1/2 T 1/3 * 1 * 2/3 * 1/2, 2, 3/2 1, 3, 0 1, 3, 0 B NO stationary -equilibrium 1/3 * 2/3 * Center for the Study of Rationality Hebrew University of Jerusalem

  36. Ex. 3: 3-Player Absorbing AT Game F 3/2, 1/2, 2 3, 0, 1 N L R 1/3 * 2/3 * 0, 0, 0 0, 1, 3 4/3, 4/3, 4/3 2, 3/2, 1/2 T 1/3 * 1 * 2/3 * 1/2, 2, 3/2 1, 3, 0 B non-stationary 0-equilibrium 1/3 * 2/3 * Player 1 on B: 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, …. Player 2 on R:0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, …. Player 3 on F:0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾,…. Center for the Study of Rationality Hebrew University of Jerusalem

  37. Ex. 3: 3-Player Absorbing AT Game F 3/2, 1/2, 2 3, 0, 1 N L R 1/3 * 2/3 * 0, 0, 0 0, 1, 3 4/3, 4/3, 4/3 2, 3/2, 1/2 T 1/3 * 1 * 2/3 * 1/2, 2, 3/2 1, 3, 0 B equilibrium rewards (1, 2, 1) 1/3 * 2/3 * Player 1 on B: 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, …. Player 2 on R:0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, …. Player 3 on F:0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾,…. Center for the Study of Rationality Hebrew University of Jerusalem

  38. Results • 0-equilibria for n-player AT games (threats!) • 0-opt. stationary strat. for 0-sum AT games • Stat. -equilibria for 2-player abs. AT games • Result 3 can not be strengthened, neither to 3-player abs. AT games, nor to 2-player non-abs. AT games, nor to give stat. 0-equilibria Center for the Study of Rationality Hebrew University of Jerusalem

  39. ? frank@math.unimaas.nl Center for the Study of Rationality Hebrew University of Jerusalem

  40. GAME VER Center for the Study of Rationality Hebrew University of Jerusalem

  41. GAME VER Center for the Study of Rationality Hebrew University of Jerusalem

  42. GAME VER Center for the Study of Rationality Hebrew University of Jerusalem

  43. GAME VER Center for the Study of Rationality Hebrew University of Jerusalem

More Related