1 / 31

Trading Agent Competition: Performance Evaluation

Trading Agent Competition: Performance Evaluation. Presented by Brett Borghetti borg@cs.umn.edu 22 March 2006. Think about this. You own a small business You make a bunch of strategic decisions/plans/policies Your 1 st quarter net profit is $100,000 Which choices helped?

bradley-albert
Download Presentation

Trading Agent Competition: Performance Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Trading Agent Competition: Performance Evaluation Presented by Brett Borghetti borg@cs.umn.edu 22 March 2006

  2. Think about this • You own a small business • You make a bunch of strategic decisions/plans/policies • Your 1st quarter net profit is $100,000 • Which choices helped? • Which choices hurt? • Can your decisions be examined independently? • How do you improve next quarter?

  3. The Situation • We sometimes have to make our plans and policies before their execution • We don’t know fully what the market will do next quarter (uncertainty) • We are in competition with other businesses/entities who may act to thwart our plans

  4. A Solution • Repeat (until good enough): • Predict the effects of our choices offline • Adjust our choices to optimize outcome • Execute our plans • Measure the effectiveness of our choices online

  5. Presentation Overview • TAC-SCM Overview • Current analysis methods • New methods • Future Research

  6. What is TAC-SCM? • Simulation of a market supply chain • Agent is the computer manufacturer • Buys parts from suppliers in auction • Manage assembly line/production schedule • Reverse Auction to sell computers • Ship computers to customers • Six agents compete: maximize profit TAC-SCMCURRENT ANALYSIS NEW METHODS FUTURE RESEARCH

  7. TAC-SCM Interaction TAC-SCMCURRENT ANALYSIS NEW METHODS FUTURE RESEARCH

  8. Game Flow Diagram

  9. Complexity: Beyond human-in-the-loop capability Compete with 5 other agents selling computers Real time: 15 sec/day x 220 days Auctions (normal and reverse for all transactions) 8 parts suppliers with production capacity changing daily 16 different computer types to build in 3 price classes 100s of Customers with varying demand and reserve prices Price probing, future purchase decisions . . . . . Small market: Agents have large impact on each other Explicit Competition – PROFIT! Learning other’s habits & patterns and out-thinking them Information denial / Decision perturbation TAC - Why is it Interesting? TAC-SCMCURRENT ANALYSIS NEW METHODS FUTURE RESEARCH

  10. UMN MinneTAC Design • Component-based architecture • Procurement – Purchase parts from suppliers • Production – Manages the production line • Sales – Interacts with customers to make sales • Shipping – plans customer shipping schedule • Repository – centralized data storage / accesors • Oracle – decision assistance evaluators TAC-SCMCURRENT ANALYSIS NEW METHODS FUTURE RESEARCH

  11. Design pros and cons • Lower module coupling = good design • More simultaneous developers • Easier to maintain • Self interest vs. Common good • Causality – which components responsible for a good or bad decision? • How do we analyze and improve our global performance? TAC-SCMCURRENT ANALYSIS NEW METHODS FUTURE RESEARCH

  12. Current Analysis Methods • Run offline simulations and tweak components to optimize profit • CPU intensive (1 hour per game) • Statistical significance => many games • Competition is limited • Causal analysis is complicated TAC-SCMCURRENT ANALYSIS NEW METHODS FUTURE RESEARCH

  13. New Analysis Methods • What if we could measure performance of components inside of the agent? • We could directly compare performance between two components of the same type against the same TAC market dataset • We could reduce the number of games required to show correlations / relative performance • We could more rapidly determine which ‘tweaks’ actually have an effect on game outcome TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

  14. Challenges of Measuring • Which metrics are actually correlated with profit? • How do we assign sharing of credit or blame? • How do we account for the varying market conditions while taking measurements over multiple games? • How do we simulate various competitive environments offline? TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

  15. Methodology - Overview • Controlling the market conditions • Control Randomness • Control market supply / demand situation • Measuring component performance • Create metrics • Determine if metric is correlated with profit • Assign component responsibility TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

  16. Controlling Randomness • Re-design server to allow deterministic / replayable games • Three types of random processes: • Server variables (customer/supplier) • Agent-dependent variables • Dummy agent variables • Each process gets its own seed • Eliminates race conditions in replays • Allows some process true randomness while others replay TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

  17. Market Manipulation Agents • Goal – develop a way of manipulating supply and demand conditions during a simulation to observe how competitive agents respond • Method – Build TAC agents that are not concerned with their own profit, but rather with absorbing/releasing market share TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

  18. Market Manipulation Agents • Market Relief Agent • Accepts and fulfils no customer RFQs • Purchases no parts from suppliers • Result: Reduces demand on suppliers and reduces supply to customers • Market Pressure Agent • Makes more promises to customers than regular agent could handle • Buys more parts from suppliers than regular agent should • Result: Increases demand on suppliers and causes customer demand to go down TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

  19. Measuring Component Performance • Create suite of metrics to measure: • Replacement costs when a part is sold • Storage costs of parts/computers • Late penalties • Wasted production cycles • Remaining inventory at end of game • … TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

  20. Measuring Causality • How do we assign responsibility? • For example: Why was the item late? • Didn’t ship the product? • Didn’t make the product? • Didn’t have the parts? TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

  21. Implementing Metrics • Allow for easy creation of new metrics • Serialize game information • Evaluations can then be made offline • Enables us to experiment in finding metrics that are correlated with profit. • But how do we even know if a metric is correlated with profit? • Large amount of variability in each game • Need a large sample size, which takes time TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

  22. Results to date • We have some preliminary data regarding how the manipulation agents cause the other agents to behave under various market conditions TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

  23. Performance Results: Market Relief Agent vs Dummy Agents TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH Note the scale of this graph

  24. Performance Results: Market Relief Agent vs Dummy Agents TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH • Unexpected benefits! • MRAs can reveal undesireable traits/logic flaws in an agent

  25. Performance Results: Market Pressure Agent vs MinneTAC TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

  26. Performance Results: Market Pressure Agent vs MinneTAC TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

  27. Performance Results: Market Pressure Agent vs MinneTAC TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

  28. Performance Results: Market Pressure Agent vs Competition TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

  29. Conclusions • We’ve created some new tools for measuring offline performance • Replayable games • Market Condition Manipulation • Embedded Metrics Collection • Started choosing what metrics contain information allowing profit prediction TAC-SCMCURRENT ANALYSISNEW METHODS FUTURE RESEARCH

  30. Future Work • Improve Market Manipulation agents • Make competition modeling more realistic • Find additional metrics that have a better correlation to overall profit • Better off-line prediction of on-line performance • Use metrics to guide development of better components • Leads to better profit performance [build to the metric] • Use on-line metrics to make live strategic decisions • Live ‘tuning’ of components if they begin to underperform • Selection of ‘pinch-hitter’ components in certain market conditions TAC-SCMCURRENT ANALYSISNEW METHODSFUTURE RESEARCH

  31. Acknowledgement / Info Special thanks to: • Eric Sodomka • Dr. Maria Gini • Dr. John Collins • UMN TAC team More Info at • MinneTAC website • www.cs.umn.edu/tac • SICS website • www.sics.se/tac

More Related