Advances in Evolutionary Games

1. Advances in Evolutionary Games E. Altman INRIA, France Bionetics Dcember 2010

2. Overview of the talk • Background on Evolutionary Games and Population Games and Examples • Adding time varying states • Adding controlled state transitions • Examples • Mathematical model • Computing Equilibria

3. EGs and PGs in Biology and Engineering • BIOLOGY CONTEXT: Central tool defined by Meynard Smith (1972) for explaining and predicting dynamics of large competing populations with many limited local interactions. EG. • TELECOM CONTEXT: Competition between protocols, technologies. Can be used to design and regulate evolution • ROAD TRAFFIC CONTEXT: Competition between cars over routes. Introduced by Wardrop (1952). PG.

4. Framework • Large population • Several strategies (behavior of individuals). Call all those who use a strategy a subpopulation • EG: Competition between the strategies through a very large number of interactions each involving a small number of individuals typically pairwize interactions • PG: interactions with an infinite number of players

5. Evolutionary Games: Definitions • Evolutionary Stable Strategy (ESS): At ESS, the populations are immuned from being invaded by other small populations (mutations). • ESS more robust than standard Nash equilibrium. • “State” vector X: fractions of users that belong to different populations. Or fractions of strategies in a population • Fitness: J(p,q):= utility when playing pure strategy p and all others play q. J(x,y) fitness when using mixed strategies x and y, resp.

6. Evolutionary Stable Strategy (ESS): p is ESS if for all q there is d(q) s.t. for all 0<e<d(q) J(p,p) >= (1-e) J(q,p) + e J(q,q) Equivalent condition: J(p,p) > J(q,p) or J(p,p) = J(q,p) and J(q,q) < J(p,q)

7. Ex 1: Hawk and Dove Game • Large population of animals. Occasionally two animal find themselves in competition on the same piece of food. An animal can adopt an aggressive behavior (Hawk) or a peaceful one (Dove). D-D: peaceful, equal-sharing of the food. fitness of 0.5 to each player. H-D or D-H: 0 fitness to D and 1 for H that gets all the food no fight

8. HD Game • H-H: fight in which with equal chances to obtain the food but also to be wounded. Then the fitness of each player is 0.5-d, -d is the expected loss of fitness due to being injured.

9. Modeling competition: Generalized HD Game • Generalized game: A11<A22<A12 and A21<A22. • Simple conditions for H to be unique ESS and for mixed ESS

10. Ex 2: Competition between protocols • There are various flow control protocols to regulate traffic in the Internet. • Huge number of file transfers every second • Interactions occur between limited number of connections that use the same bottleneck link • The average speed of transfer, the delay etc depend on the versions of the protocol involved in the interaction

11. Competition between protocols

12. Ex 3: Population Games (PG) in Wireless communications • Cellular network contains many mobiles. One base station (BS) per cell • CDMA: At each time an individual sends a packet it interacts with all mobiles in the same cell • A mobile can transmit with different power levels q1 < … < qK. Higher power is more costly • Objective of : max_k J(k,w) := • where wk is the fraction of mobiles that use qk

13. Replicator Dynamics • Delayed case: present growth rate depends on past fitness K and tau : design parameters. Determine speed of convergence and stability

14. Architecting evolution: impact of K • stability iff K tau< θ. • Oscillations mean no convergence to ESS.

15. Individual States in EG and PG • Different behaviors may be a result of different inherent characteristics – individual states • Example: weather conditions, age, • The individual state can be random • Description through a Markov chain • EG: Local interactions with players chosen at random; their state is unknown • PG: Global interactions, the state can be known

16. Indiv. states in HD Game • The decisions H or D determine whether a fight will occur • There is also a true identity -- Strong or Weak We call this the individual STATE • If there is a fight then the states determine the outcome. • Note: the decision H/D are taken without knowing the state of the other.

17. Indiv. States in Networks • Flow control protocol: large end to end delay slows the protocol and decreases its throughput • Wireless: - the power received may depend on the radio channel conditions - the transmitted power may depend on the energy level of the battery

18. MDEG: Markov Decision EG ASG: Anonymous Sequential G • Each player has a controlled Markov chain (MDP) • A player has finite or infinite life time. It has several interactions each time with another randomly selected player (MDEG) with a large population (ASG) • Each interaction results in an immediate fitness that depends on the actions and states of the players involved • The states and actions of a player determine also the probability distribution of the next state

19. Assumptions, References • A player maximizes the total expected or average fitness • EGaverage fitness: EA & YH IEEE trans Autom Contr, June 2010 (theory) EG total expected fitness: Infocom 2008 (power control) Evolutionary Ecology Research, 2009 (the theory) (EA, YH, R El-Azouzi, H. Tembine) SAG: Jovanovic &Rosenthal, J Math. Econ, 1988 (disc cost) • Assume: The transition probabilities of the MDP of a player depend only on its own actions and states

20. Ex 1 (MDEG): Hawk and Dove game • A bird that looses becomes weaker (less energy) • A very weak bird dies • State: Energy level • Would a weaker bird be more or less aggressive? • If the result of the fight are determined by the energy level then the transitions are determined by states and actions of both birds.

21. Ex 2: (MDEG or ASG): Battery dependent power control • Transmitting at higher power empties faster the battery • A battery with little energy left is not able to support transmissions at high power • The state: remaining energy in the battery • The transitions do not depend on other mobiles

22. Ex 3: channel dependent power control The decision to transmit at power qk may depend on the channel state • Seems “degenerate”: the mobile does not control the transitions • Restriction: discrete power set; if a power level is chosen then the next power cannot differ by more than one unit. • This creates non-trivial transitions. The state = (Channel state, current power level)

23. MDEG: Local interactions • Each local interaction is described by a stochashtic game with partial monitoring • The stochastic game has an equilibrium. The game is equivalent to a matrix game where the pure actions of a player are its pure stationary policies • Allows us to transform the problem into a standard EG with a huge action space (action=pure policy) • We show: equivalence to a polytope game in the space of marginal stationary occupation measures Cardinality: no. of states times number of actions

24. POWER CONTROL ASG Model of Individual player Each player is associated with a MDP with

25. The model for an individual • State of an individual corresponds to the battery level. • Set of actions available at state s: • Qs decreases with the energy: smaller powers are available when the battery has less energy • Transitions: the probability to stay at a state s if q is used is • Recharging: P0N is the probability to move from 0 to N

26. The model: Interactions • Global state: fraction of mobile in each individual state: • Proportion of mobiles using qk at time t is • The Reward: • Stationary policy the ptob to choose qk in s

27. Interactions and System model • In stationary regime:

28. Interactions and System model • The expected reward: • A stationary policy u is an equilibrium if

29. Results (1) • Define the interference of u: • Denote the probability distribution with mass 1 at q

30. Results (2) Threorem. An equilibrium exists within

31. Results (3)

32. Future work • Branching MDPs: a state-action pair of an individual determines the immediate fitness, the transition probabilities and the number of off-springs