Towards Adaptation in Complex Agent Negotiations & Markets

1. Towards Adaptation in Complex Agent Negotiations & Markets Ryszard Kowalczyk (Acknowledgments: Jakub Brzostowski, Eduardo Gomez, Ingo Mueller, Tino Schlegel) Swinburne Centre for Information Technology Research Faculty of Information and Communication Technologies Swinburne University of TechnologyMelbourne, Australia

2. Agenda • Introduction and motivation • Adaptation in complex agent negotiations • Prospects  partner selection • Opponents  predictive negotiation • Co-negotiators  compound negotiations • Adaptation in agent e-markets • Participants  coalition formation • Market mechanism  market optimisation • Emergence in markets • Conclusions

3. Introduction • Negotiation – a form of distributed decision-making that involves multiple parties interacting in order to find an agreement on some matter • Economics, Management Science, Game Theory • AI (DAI): DPS & MAS (distributed search, decision theories, logic) • Mechanism design • Objects & protocols (common) • Decision-making models (typically private) • Agent strategies – individual decision making to satisfy (maximise) agent’s negotiation objectives • Competitive (self-interested) vs collaborative (social welfare) • Decisions according to strategies given mechanism and participants’ behaviour • Outcome results from interplay of participants’ decisions/strategies • E-markets – “an electronic exchange where sellers & buyers communicate & conduct business over the Internet” virtual place for negotiations

4. Motivation • Significant progress in agent-based negotiation, typically considering: • Bilateral or multilateral negotiation (e.g. auctions, CNP, ICNP,…) • Negotiation with all or a set number of participants • Multiple negotiations (e.g. related auctions) • Mostly predefined “static” strategies (fine tuning; incl. time&behaviuor-dependent tactics) • Some adaptation (e.g. strategy selection, rules, CBR) and learning (e.g. Bayes, GA, Q-learning) • Real-world negotiations can involve complex settings • E.g. required in open, large-scale and dynamic environments such as Web and Grid • Many different & changing participants (pot. unknown & large #) • Inter-related (joint, nested and compound) negotiations • many-one-to-one-many (M:1:1), many-one-to-many (M:1:N) • Multi-round, coalition-based, etc • Different negotiation and market mechanisms • In addition to limited info, no centralised control/coordination, … Is adaptation needed? Can it help?

5. Adaptation in Agent Negotiations & Markets • Adaptation to prospects (e.g. limit a number of negotiations) • Select most prospective partners (e.g. similar offerings; top-down) • Adaptation to opponents (e.g. opponent behaviour) • Select (from predefined strategies) or adapt (predictive strategy) • Adaptation in multiple/compound negotiations (e.g. M:1:N) • Decompose global utility functions for sub/co-negotiating agents • Collectively adapt individual utility functions in multiple rounds • Adaptation in markets (closed and open) • Coalition formation (e.g. bottom-up aggregations) • Adaptive market optimisation • Adapt agent strategies/behaviour to market mechanisms • Emergent self-organisation in markets

6. Selection of Negotiation Partners (Brzostowski & Kowalczyk, 2005) • Most current approaches engage all agents in negotiations • Possible waste of time/resources for unsuccessful negotiations • Unfeasible if a large number of potential negotiation partners • More importantly - desirable to negotiate with agents with whom a higher chance of successful negotiation and better agreements • Most prospective negotiation partners selection (one or many) based on their behaviour in past negotiations • The extended inference rule in CBR: “the more similar are the situations the more probable/possible that the outcomes are similar” • Prediction of each potential partner negotiation capability (probability or possibility distributions) based on similarity of situations S, outcomes P and their utility  • Expected utility with partner = aggr. distribution & agent utility • Order (&select) partners according to max expected utility

7. Selection of Negotiation Partners (cont.) (Brzostowski & Kowalczyk, 2005) • Experiments • Different history sizes (nc≤40) generated by negotiations between 6 client and 6 provider agents (positional bargaining with random parameters) • A randomly chosen client negotiate (100 times for each nc) with the providers and simultaneously perform the predictions • Probabilistic and possibilistic expected utility • Results • Good approximation of possibility distribution (~utility function) • Negotiation outcome prediction vs actual outcome (possibilistic) • Statistically significant utility gain vs random (5-50agents, 800runs)

8. Adaptation to Opponents (Brzostowski & Kowalczyk, 2006) • Most current approaches • “Static” (e.g. tit-for-tat, behaviuor-dependent tactics - need fine tuning) • Adaptation (e.g. strategy selection, rules, CBR – heuristics or history) • Offline learning (e.g. Bayesian, GA, Q-learning – need history) • Online learning (e.g. nonlinear regression – simple tactics) • Adaptive negotiation with online prediction of complex strategies • Non-linear regression analysis of mixed time-dependent and behaviour-dependent tactics • Forecast opponent responses to our potential offers • Based on the forecast determine our offers to maximise utility gain • Adaptation at each negotiation step • Higher precision and utility gain

9. Adaptation to Opponents (cont.) (Brzostowski & Kowalczyk, 2006) Sample encounters • Opponent models (4 time + behaviour dependent) • Polynomial/exponential + absolute/relative tit-for-tat • Models’ parameters (6) estimated by regression analysis recursively (opponent & our past offers) • Model with the best fit used to assess sequences of our potential future offers • Choose the next offer that maximises gain at the end of negotiation (according to our deadline and predicted opponent deadline) • Experiments ST = (C U L U B) x {a, r} x (S U M U L) • 7 time tactics in 3 groups (Concede, Linear, Boulware) • 2 behaviour tactics (Absolute & Relative TFT) • 9 weights in 3 groups (Small, Medium, Large) • Utility gain: adaptive against staticST strategies BLA BMA BSA LLR LMR LSR Utility

10. Utility Decomposition in CompoundNegotiations • Compound multi-agent negotiations • Agents negotiate with multiple providers of atomic services to form a compound service • Global preferences/utility functions specified for the compound service (e.g. over total cost & time) • Individual agents need utility functions for negotiated atomic services (each over cost & time) • So, individual utility functions need to be derived from global utility function • Some challenges • Only global utility function specified • No prior information about the pattern and number of atomic services (and negotiation agents) • Inverse problem – no unique decomposition!

11. Utility Decomposition in CompoundNegotiations (cont.) (Brzostowski & Kowalczyk, 2006) • Related work: Specific solution for time/utility function decomposition (H. Wu et al) • Domain specific (allocation of computing tasks to threads) & single attribute (execution time) • Five heuristic decompositions (no validation nor sound foundations) • Generic utility decomposition based on principles of fuzzy set projection • Initial decomposition (first round) • Derive the single-service (multi-attribute) utility functions fromthe global (compound) utility function as fuzzy projections • Easy/efficient calculation based on the proof that“projection of utility function in the case of attributes montone in the sense of Pareto is equal to boundary function”(proof and algorithm in the paper) • Adaptation in subsequent rounds (in progress) • Based on relative gain in previous rounds • Based on relative cooperative behaviour of opponentsobserved in previous rounds (progress of offers) • Different metrics currently under investigation • Lower/pessimistic bound? (projection is optimistic) Agent 2 Multi-attribute Agent 1 opponents our individual agents

12. Coalition Formation in Markets (Mueller & Kowalczyk, 2006) • Considerable interest in agent coalition formation • Self-interested or collaborative agents for stable, longer-term and optimal coalitions • Most approaches assume a priori knowledge • Value of coalition and a set of possible coalitions • Agents (their capabilities) are known to the central coalition leader • Usually exponential computational complexity • Multiple short-term coalitions in response to opportunity (bottom-up aggregation options) • Agents seek coalitions in response to complex advertisements • Compositions emerge through agent interactions • Coalitions offer aggregated capabilities (similar) • Coalitions may further negotiate with the requestor • All or selected by the requestor • Strategy/preferences and gain sharing (in progress) • Agreed/negotiated during the coalition formation • Negotiated within coalition during external negotiation (~sub-contracting)

13. Coalition Formation in Markets (cont.) (Mueller & Kowalczyk, 2006) • Composite service request scenario • Individual agents can only provide atomic services • Can participate in multiple coalition processes • Coalition formation = interaction + individual DM • Asynchronous communication (with timeouts) • Coalition request conversation • Candidate voting conversation • Leave coalition conversation • Decentralized (but simple) decision making • Advertise own service if willing to contribute • Ask candidates to join a coalition (min message counter; unknown pref. over rejected) • If asked, join larger coalition (& leave smaller) • Experiments & results • 19 test scenarios (2-20 services per coalition; 3 agents per service; 1000 runs per scenario) • Able to form multiple complete coalitions • Formation time proportional to coalition size • Stable coalitions (leave actions << situations)

14. Adaptive Market Optimisation (Gomes & Kowalczyk, 2007) • Many market mechanisms to exchange goods/services • Auctions, negotiations, double auctions, iterative price adjustment, … • Focus on market mechanism design  create an environment (structure and rules) where it is in the agents’ individual interests to act in a way that also benefits the market as a whole • Given market mechanism – how to optimise the individual agents’ behaviour to maximise individual and market (social) gain? • Reinforcement learning to optimise agents’ behaviour • Adapt strategies/decision functions to market mechanisms • Self-interested participants (ind. decision functions → social welfare?) • Sample market mechanism - Iterative Price Adjustment (IPA) • Market adjusts prices pi based on demand xi for resources Ri • Agents make decisions based on their demand functions x = f (p) (note: no explicit utility) • Market clears when demand = supply (available resources)

15. Adaptive Market Optimisation (cont.) (Gomes & Kowalczyk, 2007) • Reinforcement learning (Q-learning) with utility-aware agents for market-based resource allocation (IPA) • No unique mapping of utility functions into demand function • Agents learn optimal demand functions given utility functions • Experiments • Self-interested agents with 2 utility functions (price and memory) • Simultaneously learn demand functions L (4 settings x 106 runs) • Then 49 IPA market runs LxL, LxS, SxS (S defined by hand) • Results • Agents can learn meaningful demand functions L (trend) • L gain good individual and higher social utilities (vs S) • Further work • Learn against mixed L & S • How much learning is enough • Other market mechanisms

16. Emergent Self-organising Agent Resource Allocation (Schlegel & Kowalczyk, 2006) • Services need to use resources and communicate • Typically resources are pre-allocated to services • May constrain QoS capability of services • Possibly high communication and coordination costfor highly communicative services in distributed compositions • Current resource allocation mechanisms • Centralised (global info about all services and resources) • Distributed (e.g. market-based needs a central facilitator; agent-based uses extensive info exchanged for coordination) • Agent-based decentralised approach without inter-agent communication • No central control authority, resource management layer, coord. communication • Optimised resource allocation in dynamic service/resource environment • Minimize communication and coordination overhead between services/resources

17. Self-organising Agent-based Resource Allocation (Schlegel & Kowalczyk, 2007) • Based on inductive reasoning & bounded rationality principles (B. Arthur: El Farol) • Randomised simple predictors (resource load) • Info on previous loads (allocated servers only) • Experiments • 10 predictors per agent rand from 32 • Predictors use history t <10 of load y • n-cycle predictor • n-mean predictor • n-linear regression • n-distribution predictor (random) • n-mirror predictor • 3 servers, 9000 units shared resources • Server 1 and 2 capacity changes • 750 agents allocating services • Random resource consumption [1, 45] • Random exec. time [1, 15]; between [0, 30] Server2 Server1 Server3 Total

18. Conclusions • Real-world negotiations and e-markets can involve complex settings, e.g: • Many different & changing participants (pot. unknown & large #) • Inter-related (joint, nested and compound) negotiations • Different negotiation/market mechanisms and participants • Then adaptation is useful (required?) • Adaptation to prospects (e.g. most prospective partners, markets) • Adaptation to opponents (e.g. on-line predictive strategies) • Adaptation in multiple/compound negotiations (e.g. decompose and adapt individual utility functions) • Adaptation in markets (e.g. coalition formation, adaptive market optimisation, emergence) • Early promising results – more work needed!

19. Towards Adaptation in Complex Agent Negotiations & Markets Thank you! Q&A

20. References • J. Brzostowski and R. Kowalczyk (2005). On Possibilistic Case-based Reasoning for Selecting Partners for Multi-Attribute Agent Negotiation. AAMAS 2005, Utrecht, The Netherlands, 25-29 July 2005, pp. 273-279 • Ingo Mueller, Ryszard Kowalczyk and Peter Braun (2006). Towards Agent-based Coalition Formation for Service Composition. IEEE/WIC/ACM IAT’06, 18-22 December 2006, Honk Kong • J. Brzostowski and R. Kowalczyk (2006). Predicting partner's behaviour in agent negotiation. AAMAS'2006, Hakodate, Japan, May 2006. pp. 355-361 • Jakub Brzostowski and Ryszard Kowalczyk (2006). Adaptive negotiation with on-line prediction of opponent behaviour in agent- based negotiations. IEEE/WIC/ACM IAT’06, 18-22 Dec, Honk Kong • Jakub Brzostowski and Ryszard Kowalczyk (2006). On utility decomposition in compound multi-agent negotiations. Group Decision and Negotiation (GDN2006), June 25–28, Karlsruhe, Germany • Eduardo Gomes and Ryszard Kowalczyk (2007). Reinforcement Learning with Utility-aware Agents for Market-based Resource Allocation. AAMAS'2007, Hawaii (submitted) • T. Schlegel, P. Braun and R. Kowalczyk (2006). Towards Autonomous Mobile Agents with Emergent Migration Behaviour. AAMAS'2006, Hakodate, Japan, May 2006. pp. 585-592 • T. Schlegel and R. Kowalczyk (2007). Towards Self-organising Agent-based Resource Allocation in a Multi-Server Environment. AAMAS'2007, Hawaii (submitted)