Worlds with many intelligent agents

1. Worlds with many intelligent agents An important consideration in AI, as well as games, distributed systems and networking, economics, sociology, political science, international relations, and other disciplines

2. Multi-agent Systems, or “Games” Worlds with multiple agents, somtimes called “games”, are VASTLY more complicated than worlds with single agents. They’re also more interesting and fun! There are many different aspects to learn about: • Representations (we will consider two, but there are MANY) • Evaluation metrics (I’ll show you a few standard ones, but again there are MANY) • Inference and planning (we will talk about a few techniques) • Learning (We won’t really cover this) • Communication (We won’t directly cover this) • Computational Complexity (I’ll mention this in passing) • Applications (We’ll look at several examples, but there are TONS)

3. Example Game (Prisoner’s Dilemma) Student A and Student B have been naughty, cheating on an exam. Their teacher finds some evidence, and sends them to the dean’s office, where the dean grills them individually to find out more. Outcome: A rats out B A says nothing B rats out A B says nothing

4. Example Game (Prisoner’s Dilemma) Definitions: the set of choices available to each agent is called their action set, denoted Actions(A) or Actions(B). Each square in the grid is called an outcome. Mathematically, let a be an action in Actions(A), and b in Actions(B). O(a, b) is the outcome square. OA(a, b) is the reward for A, and OB(a, b) is the reward for B. This grid is called a normal formrepresentation of the game. Outcome: A rats out B A says nothing B rats out A B says nothing

5. Quiz: Best outcome? Which outcome is the best, and why? Outcome: A rats out B A says nothing B rats out A B says nothing

6. Answer: Best outcome? The answer depends. Let’s look at a few definitions of “best”. Outcome: A rats out B A says nothing B rats out A B says nothing

7. Definition: Dominant Strategy Definition: A strategy (or pure strategy) for player A (ditto for B) is a selection of one of the actions from Actions(A). Definition: A’s strategy a ∊ Actions(A) is dominant if ∀b ∊ Actions(B), ∀a’≠a∊ Actions(A) . OA(a, b) > OA(a’, b) Another way of saying this is: a is a dominant strategy for A if it is better than any other strategy a’ for A, no matter what strategy B chooses.

8. Quiz: Dominant Strategy Definition: A’s strategy a ∊ Actions(A) is dominant if ∀b ∊ Actions(B), ∀a’≠a∊ Actions(A) . OA(a, b) > OA(a’, b) Does A have a dominant strategy for the game below? What is it? What about B? Outcome: A rats out B A says nothing B rats out A B says nothing

9. Answer: Dominant Strategy Definition: A’s strategy a ∊ Actions(A) is dominant if ∀b ∊ Actions(B), ∀a’≠a∊ Actions(A) . OA(a, b) > OA(a’, b) Does A have a dominant strategy for the game below? What is it? What about B? Dominant strategy for A! A rats out B A says nothing Dominant strategy for B! B rats out A B says nothing

10. Definition: Pareto Optimal Definition: An outcome o is called pareto optimal if there is no other outcome o’ such that all players prefer o’ to o. Notice: pareto optimality can lead to lots of optimal outcomes, not just one. That’s because o’ is considered better than o only if ALL players prefer o’ to o.

11. Quiz: Pareto Optimal Definition: An outcome o is called pareto optimal if there is no other outcome o’ such that all players prefer o’ to o. Quiz: which outcome(s) is (are) pareto optimal? Outcome: A rats out B A says nothing B rats out A B says nothing

12. Answer: Pareto Optimal Definition: An outcome o is called pareto optimal if there is no other outcome o’ such that all players prefer o’ to o. Quiz: which outcome(s) is (are) pareto optimal? Outcome: A rats out B A says nothing B rats out A B says nothing

13. Definition: (Nash) Equilibrium Definition: An outcome O(a, b) is called a Nash Equilibrium (or just equilibrium) if • ∀a’≠a∊ Actions(A) . OA(a, b) >= OA(a’, b) and • ∀b’≠b∊ Actions(B) . OB(a, b) >= OB(a, b’) Another way of saying this is that an outcome is an equilibrium if each player is as happy with this outcome as any other available option, assuming the other player sticks with his equilibrium strategy. Notice that this means an outcome is stable, in the sense that both players have no incentive to change their strategy.

14. Quiz: (Nash) Equilibrium Definition: An outcome O(a, b) is called a Nash Equilibriumif • ∀a’≠a∊ Actions(A) . OA(a, b) >= OA(a’, b) and • ∀b’≠b∊ Actions(B) . OB(a, b) >= OB(a, b’) Which outcome(s) is (are) a Nash equilibrium? Outcome: A rats out B A says nothing B rats out A B says nothing

15. Answer: (Nash) Equilibrium Definition: An outcome O(a, b) is called a Nash Equilibriumif • ∀a’≠a∊ Actions(A) . OA(a, b) >= OA(a’, b) and • ∀b’≠b∊ Actions(B) . OB(a, b) >= OB(a, b’) Which outcome(s) is (are) a Nash equilibrium? Outcome: A rats out B A says nothing B rats out A B says nothing

16. Answer: (Nash) Equilibrium One reason that the prisoner’s dilemma is famous: The only outcome that is a Nash equilibrium is also the only outcome that is NOT a pareto optimum. This is not true in all games; it depends on the rewards! For the prisoner’s dilemma, it means that the only stable solution is probably the worst possible solution for the players. (It’s sort of a depressing example.) A rats out B A says nothing B rats out A B says nothing

17. Quiz: The game of Chicken (a.k.a. “Hawk-Dove”, “snow-drift”) What are the dominant strategies, pareto optima, and Nash equlibria for the Chicken game? Player 1 Swerve Straight Swerve Player 2 Straight

18. Answer: The game of Chicken (a.k.a. “Hawk-Dove”, “snow-drift”) What are the dominant strategies, pareto optima, and Nash equlibria for the Chicken game? Player 1 Nash Equilibrium Pareto Optimum No dominant strategies! Swerve Straight Swerve Player 2 Straight

19. Quiz: Coordination Games What are the dominant strategies, pareto optima, and Nash equlibria for each of the games below? Left Right Party Home Left Party Right Home Pure coordination game Drive on which side of the road? Party Home Stag Hare Party Stag Home Hare “Battle of the Sexes” game “Stag Hunt” game

20. Nash Equilibrium Dominant Strategy Pareto Optimum Answer: Coordination Games What are the dominant strategies, pareto optima, and Nash equlibria for each of the games below? Left Right Party Home Left Party Right Home Pure coordination game Drive on which side of the road? Party Home Stag Hare Party Stag Home Hare “Battle of the Sexes” game “Stag Hunt” game

21. Humans don’t necessarily play Nash equlibria The “Guess 2/3 of the Average” Game is famous because in experiments with people, they often don’t play Nash equilibrium strategies. (Other games like this are the centipede game and the prisoner’s dilemma.) Thus Nash equilibria aren’t perfect models of human behavior. Let’s try it. Here are the rules to “Guess 2/3 of the average”: • Everyone guess a number between 0 and 100 (integer or real). Once everyone guesses, we’ll compute the average. • The set of people who come closest to 2/3 of the average are the winners (payoff 1), everyone else loses (payoff 0).

22. Nash equilibrium for “Guess 2/3 of the Average” If everyone guesses 100, then 2/3 of the average is 66.7, so 66.7 is the biggest possible result, so everyone has an incentive to guess 66.7 or less, so anything more than 66.7 can’t be a Nash equilibrium. If everyone guesses 66.7, then 2/3 of the average will be around 44.4, so everyone has an incentive to guess lower, so guessing more than 44.4 can’t be a Nash equilibrium. … If everyone guesses 0, then we have a Nash equilibrium (they’d all guess right and win, and there’s no incentive for anyone to change their guess assuming everyone else remains at 0).

23. Constant-Sum Games Player 1 Rock Paper Scissors Notice: if you add up the reward for P1 and P2 in each square, you get 0. The fact that all squares have the same sum makes this a “constant-sum” game. This game is in fact a “zero-sum” game. Rock Paper Player 2 Scissors

24. Constant-Sum Games Player 1 Actually, any constant-sum game can be converted to an equivalent zero-sum game by subtracting a constant from every payoff. They are “equivalent” in that this transformation preserves equilibria, optimal, and dominant strategies. Rock Paper Scissors Rock Paper Player 2 Scissors

25. Constant-Sum Games Player 1 Rock Paper Scissors In constant-sum games, any gain for one player is offset by a loss for the other player. So these are hyper-competitive games. Rock Paper Player 2 Scissors

26. Quiz: Pure Strategy Nash Equilibria Player 1 Rock Paper Scissors Can you find a Nash equilibrium for Rock-Paper-Scissors? Rock Paper Player 2 Scissors

27. Answer: Pure Strategy Nash Equilibria Player 1 Can you find a Nash equilibrium for Rock-Paper-Scissors? No “pure” strategy for either player can lead to a Nash equilibrium for this game! (Our definition of “strategy” is equivalent to “pure strategy”. We’ll talk about unpure strategies next.) Rock Paper Scissors Rock Paper Player 2 Scissors

28. Definition: Mixed Strategy A mixed strategy for player A is a probability distribution over the set of available actions, Actions(A). For instance, for Rock-Paper-Scissors, the distribution P(Rock) = 1/3 P(Paper) = 1/3 P(Scissors) = 1/3 is a mixed strategy. A pure strategy is also a mixed strategy, but with a probability distribution that places all the probability on one outcome. E.g.: P(Rock) = 0, P(Paper) = 1, P(Scissors) = 0.

29. Definition: Mixed Strategy Nash Equilibrium A mixed strategy profile for players A and B is a pair of mixed strategies, distribution PA for player A and distribution PBfor player B. A mixed strategy Nash equilibrium is a mixed strategy profile (PA, PB) such that neither player would gain by choosing a different mixed strategy, assuming the other player’s mixed strategy stays the same.

30. Quiz: Mixed Strategy Nash Equilibria Player 1 Rock Paper Scissors Can you find a mixed-strategy Nash equilibrium for Rock Paper Scissors? (This can be hard in general, but see if you can guess it for this game.) Rock Paper Player 2 Scissors

31. Answer: Mixed Strategy Nash Equilibria Player 1 Rock Paper Scissors The mixed strategy profile where each player has 1/3 probability for each action is a Nash equilibrium. (It’s the only Nash equilibrium for this game.) Rock Paper Player 2 Scissors

32. Nash’s Theorem Theorem (Nash, 1951): Every finite game (finite number of players, finite number of pure strategies) has at least one mixed-strategy Nash equilibrium. Nash, John (1951) "Non-Cooperative Games" The Annals of Mathematics 54(2):286-295. (John Nash did not call them “Nash equilibria”, that name came later.) He shared the 1994 Nobel Memorial Prize in Economic Sciences with game theorists ReinhardSelten and John Harsanyi for his work on Nash equilibria. He suffered from schizophrenia in the 1950s and 1960s, as depicted in the 1998 film, “A Beautiful Mind”. He nevertheless recovered enough to return to academia and continue his research.

33. More on Constant-Sum Games Minimax Theorem(John von Neumann, 1928): For every two-person, zero-sum game with finitely many pure strategies, there exists a mixed strategy for each player and a value V such that: • Given player 2’s strategy, the best possible payoff for player 1 is V • Given player 1’s strategy, the best possible payoff for player 2 is –V. The existence of strategies part is a special case of Nash’s theorem, and a precursor to it. This basically says that player 1 can guarantee himself a payoff of at least V, and player 2 can guarantee himself a payoff of at least –V. If both players play optimally, that’s exactly what they will get. It’s called “minimax” because the players get this value by pursuing a strategy that tries to minimize the maximum payoff of the other player. We’ll come back to this. Definition: The value V is called thevalue of the game. Eg: The value of Rock-paper-scissors is 0; the best that P1 can hope to achieve, assuming P2 plays optimally (1/3 probability of each action), is a payoff of 0.