Applying the Repeated Game Framework to Multiparty Networked Applications

1. Applying the Repeated Game Framework to Multiparty Networked Applications Mike Afergan July 22, 2005 Joint work with Dave Clark, Rahul Sami and John Wroclawski

2. My Thesis Repeated games can be an important and practical tool for the design of networked applications.

3. Talk Overview • Fundamental Motivations • Background on Repeated Games • Example: Incentive-Based Routing • Research Overview and Concluding Thoughts

4. Initial Assumptions • Networked applications are important • Incentives are a concern for a large class of networked applications. • Routing • Peer-to-Peer Network application developers need tools to build systems robust to user incentives.

5. Properties Fundamental to Networked Applications Property #1:Multiple interacting self-interested parties • Direct communication or shared network • Motivates the use of game theory Property #2:Interactions are repeated. • Causal relationship between one time period and the next • Examples: • ISPs in near identical BGP sessions • Users in similar interactions with similar users (e.g., web, wireless, P2P) Suggests that the repeated context should be considered to use game theory effectively.

6. Repeated Games are Important • Repeated games are a well-studied area of game theory. • The outcome of the repeated game can significantly differ from the outcome of the one-shot game. • However, most relevant prior work considers only the one-shot game. This research is the first to consider repeated games as a tool for networked applications.

7. RepeatedGame Theory Networked Applications A Practical Fit • Importantly, in each example we derive practical results • These practical results stem from further relationships between networked applications and repeated games.

8. Repeated Games are Practical Property #3:Networked applications face multiple constraints • Example Constraints: • Need to realize system objectives • Cost, privacy, shared network • Impact of Constraints: • May not be able to realize a one-shot solution • Provides explanation for real-world phenomena Repeated games work well with practical models of networked applications.

9. Repeated Games are Practical Property #4:Actions in Networked Applications Are Highly Parameterized • Parameter value is important • More interestingly, parameter granularity is also important • In repeated games, the granularity of the action qualitatively impacts the equilibrium • The freedom permitted can be a first order concern

10. Properties Fundamental to Networked Applications These four properties apply to a large class of networked applications Repeated games are important and practical MultipleParties Repeated Dynamics Constraints Parameterized Repeated games are an important and practical tool for the design of networked applications.

11. Areas of Contribution Exposition of Thesis • Introduce the concept of using repeated games • Demonstration of a fundamental relationship between repeated games and networked applications • Present approaches and techniques Application to Important Networked Problems 1. Inter-ISP Relationships with User-Directed Routing (Chapter 3) 2. Design of Incentive-Based Routing Systems (Chapter 4) 3. Application-Layer Multicast Overlays (Chapter 5) Later in this talk, I will present #2 in depth.

12. Talk Overview • Fundamental Motivations • Background on Repeated Games • Example: Incentive-Based Routing • Research Overview and Concluding Thoughts

13. One-Shot Prisoner's Dilemma P1 Static EquilibriumOutcome In the one-shot game, (D,D) is the outcome of the unique Nash Equilibrium.

14. $$$ $+$+$+ $+ $ + S or Repeated Prisoner's Dilemma Example Strategy: 1. Play C 2. If the other player defects, play D forever P1 Outcome ofthe RepeatedGame Key Takeaway: The equilibrium of the repeated game may differ from the equilibrium of the corresponding one-shot game.

15. $+$+$+ $+ $ + S $$$ or Sample Analysis • Parameterized by discount factor () • Patience Factor (infinite game) • Probability of game ending (finite game with unknown horizon) • Example: Strategy  is an equilibrium of the game iff: (Playing  forever)  (One-time “defect”) + (Resulting payoffs) • “Play C forever. If other plays D, play D forever” is an equilibrium iff: P1    ½

16. Repeated Equilibria Under General Conditions • “Folk theorem” results show the feasibility of a large set of potential outcome payoffs • Repeated equilibria feasible under a variety of practical assumptions: • Imperfect Information [Green-Porter ’84, Fudenberg-Levine-Maskin ’94] • Players of different horizons [Fudenberg-Levine ’94] • Anonymous random matching [Ellison ’93] In practice, this means many repeated outcomes are possible under a broad class of restrictions.

17. Talk Overview • High Level Argument • Background on Repeated Games • Specific Example: Incentive-Based Routing • Problem Overview • The Problem of Repeated Dynamics • Finding Key Protocol Parameters • Generalizing the Results • Summary • Research Overview and Concluding Thoughts

18. PriceA A PriceB s B t PriceC C The ContextIncentive-Based Interdomain Routing • Architecture Overview • Routes as goods • Applied specifically and deployed incrementally • Well-motivated by: • Economic realities of today’s Internet • Increasingly prevalent technology (User-Directed Routing) [A., Wroclawski ’04] • This talk does not defend such an architecture.

19. Protocol Design Question • We consider a single competitive interchange • Our Question: How should one design a protocol for conveying pricing information for routes? s t

20. Our Analytical Framework:Repeated Games • Routing is inherently a repeated process • The outcome of the repeated game can differ qualitatively from that of the one-shot game Our research is the first to consider routing as a repeated game.

21. Our ContributionsPractical Conclusions Although routing is repeated, important properties of prior models do not hold in the repeated setting. We find newfound importance for several parameters • The length of the protocol period • The granularity of the unit-of-measure (e.g., Mbps, MBps, or Gbps) • The width of the price field These provide practical insight for protocol designers. It is possible to upper-bound prices using these parameters. This helps designers (to the extent desired) control the uncertainty presented by the repeated game.

22. Talk Overview • High Level Argument • Background on Repeated Games • Specific Example: Incentive-Based Routing • Problem Overview • The Problem of Repeated Dynamics • Finding Key Protocol Parameters • Generalizing the Results • Summary • Research Overview and Concluding Thoughts

23. s t Problem of Repeated Routing • An interconnect is • A repeated game • Between a small number of players (ISPs) The repeated game may cause artificially higher prices • Standard pricing technique: Strategyproof Mechanisms • Truthtelling is at least as good as any other strategy • Benefits: Reduced strategizing and potential oscillation • Standard mechanism: Vickrey-Clark-Groves (VCG) • Feigenbaum, Papadimitriou, Sami, and Shenker (FPSS ’02) show how to apply this to an Internet-like network efficiently

24. A 10 10 s B t2 t1 1 1 1 1 Applying VCG to a Network [FPSS ’02] Each node, i, on the Least Cost Path (LCP) paid: pi = (LCP avoiding i) – LCP + ci

25. A 10 10 s B t2 t1 1 1 1 1 Applying VCG to a Network [FPSS ’02] Each node, i, on the Least Cost Path (LCP) paid: pi = (LCP avoiding i) – (LCP) + ci Example: s -> t1: A is paid (10 + 1) – (1 + 1) + 1 = 10 s -> t2: B is paid (10 + 1) – (1 + 1) + 1 = 10 In the one-shot game, this is strategyproof.

26. A 10 10 s B t2 t1 1 1 1 1 The Repeated Version In the repeated game A and B could both bid 20: A is paid (10 + 20) – (1 + 20) + 20 = 29 B is paid (10 + 20) – (1 + 20) + 20 = 29 Conclusion #1: Although Internet routing is a repeated setting, the VCG mechanism (and thus the FPSS implementation) is not strategyproof in the repeated routing game.

27. Questions • What determines the equilibrium price? • What can be done to control, bound, or influence prices (if so desirable)?

28. Talk Overview • High Level Argument • Background on Repeated Games • Specific Example: Incentive-Based Routing • Problem Overview • The Problem of Repeated Dynamics • Finding Key Protocol Parameters • Generalizing the Results • Summary • Research Overview and Concluding Thoughts

29. A Full Model of Routing We: • Prove that particular parameters may significantly impact price • Formally analyze that impact (by looking at the derivatives) given a model with: • Repeated interactions • Asynchronous interactions • Heterogeneous networks • Multi-hop paths and multiple destinations • Confluent (BGP-like) routing • Large class of strategies This talk focuses on a simple model: Repeated Incentive Routing Game (RIRG) • Intuition and analysis is similar for more general models • Will later briefly discuss generalizations (more details in thesis)

30. Repeated Incentive Routing Game (RIRG): Topology Direction of Traffic • A particular interchange: • Single Source • Single Destination • Multiple homogenous networks offering connectivity • Networks compete for traffic on price (Bertrand competition) • Route is the market good Strategic Player s t …

31. RIRG: Key Assumptions • Key Assumption #1: The game is played via a networked protocol. • Protocol runs in a series of synchronized rounds (of length d) • There is a minimum bid granularity size (b). • Key Assumption #2: The game is not infinite. • Players only know length in expectation (D) • Note: D and d define : = 1- d/D • Additional Assumptions that can be Relaxed • Traffic is fixed • Networks have fixed per unit cost • Networks have infinite capacity • Minimum bid becomes common knowledge • Traffic is splittable FPSS-like network

32. RIRG: Play of the Game In each round: • All N players announce their bids • Traffic is evenly split among the provider(s) with the lowest price • Provider is paid for the volume of traffic at the price bid (1st price auction) • Key Decision:In each round, each network can either: • Try to be the low-price provider • Split the market with other firms at a higher price

33. Equilibrium Notion • The potential strategy space is quite large • An equilibrium notion refines the strategy space • Subgame perfect equilibrium (SPE) is natural and standard for repeated games • A strategy  is subgame perfect if i)  is a Nash equilibrium for the entire game and ii)  is a Nash equilibrium for each subgame.

34. Price Matching • For the purposes of this talk, I will focus on Price Matching (PM) Strategies • Informally: “Bid the lowest price seen in the prior period” • Results generalize, for example: • “Match price and then raise later” • “Punish by doubling initial deviation” Price Matching Strategy: • At t0, offer p* • For all t>t0, pi = p* is the largest p such that PM is SPE

35. (1) (2) p* is the maximum p such that the inequality holds. Defining Price MatchingSolving for p* • One Stage Deviation Principle (Abridged): •  is subgame perfect if and only if no player can gain by deviating from  in a single stage and conforming to  thereafter.

36. Solving for Equilibrium (2’) (3) (4) Theorem: In the RIRG, the unique equilibrium price from Price Matching is:

37. Deriving Practical Intuition Theorem: When playing Price Matching: where d is the length of the protocol period. Conclusion #2: A longer period may lead to lower prices

38. $$$ $+$+$+ $+ $ + S or “A longer period may lead to lower prices” Lowering price leads to: Higher payoffs later Big payoff now $ $ $ $ $ 1sec Period of protocol 1 month

39. More benefit to deviating Longer time before competitors react Lower prices “A longer period may lead to lower prices” Longer protocol period

40. More Practical Intuition • Theorem: When playing PM: where b is the minimum bid size. • “Minimum bid size” is not a protocol parameter. • But: • Unit-of-measure (Megabits, Megabytes, Terabits) • Width of price field (number of bits in protocol) are protocol parameters Conclusion #3: A wider price field and a more granular unit of measure may reduce price.

41. Sensitivity to Parameters • Observations: • Sensitivity to delta is large, especially in the relevant range • Impact of b is qualitative, not just precision

42. Result Summary • Example Takeaways: • Using Megabytes instead of Megabits can lead to lower prices. • A system that runs faster may lead to higher prices. A priori, some of these parameters seem benign or at most only having impact as “rounding error.”

43. Constraining Prices • Sensitivity to parameters means: • This insight must be considered • They can help “solve the problem” of the repeated dynamics (to the extent desirable) • Theorem: For all >0, there exists protocol parameter settings such that pR pS + , where: • pR is the equilibrium price in the repeated game • pS is the equilibrium price of the stage game.

44. Talk Overview • High Level Argument • Background on Repeated Games • Specific Example: Incentive-Based Routing • Problem Overview • The Problem of Repeated Dynamics • Finding Key Protocol Parameters • Generalizing the Results • Generalizing the Strategy Space • Generalizing the Game • Multiple Destinations and Confluent Flows • Heterogeneous Costs • Summary • Research Overview and Concluding Thoughts

45. Proportional Punishment (PP) Strategies • Price Matching has two weaknesses: • Prices never rise • Punishment limited to matching price • Proportional Punishment • Strategies are SPE • Punishment is bound by some constant k • Class is very large (perhaps too large) • If is a one-stage deviation from h when playing  at t0, then for PPk iff:

46. Match then Raise Price Matching Punish by Doubling p-k(p-p’) Visualizing PPk Price p p’ Time t0

47. Analyzing PPk • Theorem 3: For any PPk, the maximal price obtained by  is bound by Further, this bound is tight. • Other results follow similar to the simple Price Matching case • Impact of b and  • Bounds on pR

48. A c t1 s B c t2 A More General ModelMultiple Destinations and Confluent Flows • Multiple Destinations • Multiple goods, multiple markets • Provides for cooperation even with confluent flows A wins traffic for t1 B wins traffic for t2

49. A c t1 s B c’ t2 A More General ModelHeterogeneous Networks • Assume c > c’ • Potential for a repeated equilibrium at p*(c’) • Requires that |c – c’| is sufficiently small • Equilibria may involve only a subset of the N players • Does not necessarily imply repeated equilibria • More general graph presents more options • A robust protocol must consider such conditions A wins traffic for t1 B wins traffic for t2

50. Talk Overview • High Level Argument • Background on Repeated Games • Specific Example: Incentive-Based Routing • Problem Overview • The Problem of Repeated Dynamics • Finding Key Protocol Parameters • Generalizing the Results • Summary • Research Overview and Concluding Thoughts