120 likes | 236 Views
International Planning Competition Series. Derek Long University of Strathclyde, Glasgow, UK. Competition Format. Biennial event organised by different chairs Multiple tracks to showcase different technologies Beyond STRIPS: Temporal/metric/uncertainty/learning
E N D
International Planning Competition Series Derek Long University of Strathclyde, Glasgow, UK
Competition Format • Biennial event organised by different chairs • Multiple tracks to showcase different technologies • Beyond STRIPS: Temporal/metric/uncertainty/learning • Benchmark problems specified by organisers and delivered to competitors “unseen” • Evaluation according to “pre-specified” criteria, selected by organisers
The Good PDDL inside • Common format inputs promoting development and interchange of benchmarks • Excitement and fun factor (so far...) • Multiple organisers setting different agendas • Growth in strength of empirical evaluation of planners • Availability of more systems with improved robustness and common input formats
The Bad • Finding a continuing stream of organisers.... • Rewards shrinking while demands increasing • Splintering of the competition into many small tracks • Maintaining relevance • Community division into the “competition folk” and the rest • Competition becoming an end in itself?
The Ugly • Self-regulation of competitors • Relaxed timing of distribution of competition materials versus retrieval of results • Poorly specified evaluation criteria • Soundbite reportage: false perceptions and wrong lessons
“Research” versus “Applications” • Competitions are not applications • They form a bridge: • Accessible, well specified problems • Closer to research interests • Repeatable environment, open infrastructure • Do competitions have a more important role when applications are still hard to find?
Interest and Access • Entry bar increases • Perceived relevance falls • applications gap begins to fill? Local optimisation: tweaking what works
Motivations • Pushing research by setting challenges • Encourage jumps out of local optima • Taboo search? • Careful winning criteria • Special prizes? • Pushing engineering quality by rewarding robustness • Rewards for good software tools and encouragement/requirement to open source • Education and training: • encourage student participation and lower entry bar • Showcasing • entertainment value, broad appeal, relevance
Lessons for Competitions • Stay fresh • Catch the mood of the moment • Build up resources for future organisers • Identify long-term organisational goals • Be aware of the dangers of competitions becoming • “an application” • “an institution”
What works? PDDL inside • Common format inputs promoting development and interchange of benchmarks • Excitement and fun factor (so far...) • Multiple organisers setting different agendas Drew McDermott Fahiem Bacchus Maria Fox and Derek Long HakanYounes, Michael Littman Stefan Edelkamp and Jeorg Hoffmann Alfonso Gerevini, Alessandro Saetti, PatrikHaslum BlaiBonet, Robert Givan MalteHelmert, Minh Binh Do, IoannisRefanides Dan Bryce, Olivier Buffet, Alan Fern, RoniKhardon