1 / 7

Testbeds and the Real World

Testbeds and the Real World. Breaking it Down. Closed Testbeds better for testing optimization methods and Open Testbeds the real world better for testing your model’s assumptions, especially value Opponent Modeling? Conceivable in both worlds. Closed Testbed Objectives.

laszlo
Download Presentation

Testbeds and the Real World

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Testbeds and the Real World

  2. Breaking it Down • Closed Testbeds • better for testing optimization methods • and Open Testbeds • the real world • better for testing your model’s assumptions, especially value • Opponent Modeling? • Conceivable in both worlds

  3. Closed Testbed Objectives • Closed Testbeds should be: • Reproducible (if not statistically significant) • Have an objective • Hard enough (not solved the same year) • Low barriers of entry for multiagent researchers

  4. Models for Solving Problems vs Competitions • Models for the real world it helps to solve optimally • Models for competitions should be more general

  5. Human Models • Should we bring real world data into these competitions without conflicting with the strategic aspects of the problem (e.g. stock market data)? • Human players would be cool • Many players at once is hard to get • Mechanical Turk • Many players at once is hard to get • Even one human player makes a difference

  6. Open Problems • Can we have competitions incorporate learning utilities? • Can modeling opponents really be handled in the real world?

  7. Negative Aspects • Teaching terrorists • Teaching spammers

More Related