What Good is a Scheduling Competition? - Insights from the IPC

1. What Good is a Scheduling Competition? - Insights from the IPC Terry Zimmerman Carnegie Mellon University, Robotics Institute 5000 Forbes Avenue, Pittsburgh, PA wizim@cs.cmu.edu

2. The International Planning Competition: An exemplar for a scheduling competition? • Close (and growing closer) relationship between planning & scheduling • How would a scheduling competition distinguish itself? (a variety of ‘scheduling domains’ have been featured in IPC events) • Same question relative to the various other competitions (e.g. Benedetti, Pecora, Policella 2007) • 5 competitions held to date (1998, 2000, 2002, 2004, 2006) • Young enough: initial competition setup/design issues in recent memory • Old enough: Good examples of what worked / didn’t work • Learning curve over initial years is of interest • It’s the only computational competition I have experience with…

3. Stated general goals of the IPC • analyzing and advancing the planning state-of-the-art • providing new benchmarks and a representation formalism to aid planner comparison and evaluation • emphasizing new research issues and directions • promoting applicability of planning technology • (disseminating as much performance data as possible to the community)

4. Overview of the International Planning CompetitionsFirst IPC: 1998 Pittsburgh, PA Major Focus: • Non-temporal ‘classical’ planning only • No explicit resource modeling or metric values Domain Language extensions & Domains PDDL introduced, 6 domains Competitors: 5 planners entered Blackbox, STAN, HSP, IPP, SGP -all but HSP are Graphplan based Results: No clear-cut winner. ‘Big’ plans: 30-40 steps, Max solution sizes >100 steps

5. Overview of the International Planning CompetitionsSecond IPC: 2000 Breckenridge, CO Focus: • largely ‘classical’ planning, limited metric values • 2 tracks: 1) Fully automated 2) Hand tailored Minor refinement of PDDL, 5 domains Competitors: 17 planners entered Blackbox, FF, STAN, AltAlt, MIPs, HSP2, IPP, PropPlan, GRT, TokenPlan, SHOP, TALplanner, PbR, SystemR, BDDPlan, CHIPS Results: Fully automated> Top performers vary by domains – FF, STAN, MIPs, HSP2, GRT scale over the 5 domains Hand-tailored> TALplanner dominates (scaled to 500 blocks, ~1.5s), SHOP often gets shorter length plans Blackbox, FF, STAN, AltAlt, MIPs, HSP2, IPP, PropPlan, GRT, TokenPlan, SHOP, TALplanner, PbR, SystemR, BDDPlan, CHIPS

6. Overview of the International Planning CompetitionsThird IPC: 2002 Toulouse, Fr. Focus: • extension to temporal planning • extension to numeric constraints & fluents • 2 tracks: 1) Fully automated 2) Hand tailored Extended PDDL to support temporal & numeric features Competitors: 14 planners entered

7. Overview of the International Planning CompetitionsFourth IPC: 2004 Whistler, B.C Focus: • development of benchmark domains close to applications and diverse in structure • optimal planners separated from sub-optimal • Introduced uncertainty (probabilistic action effects) Limitation: fully observable domains, discrete distr. • 2 tracks: 1) Deterministic 2) Probabilistic PDDL extended for both tracks: Deterministic –Derived predicates, Limited exogenous events Probabilistic –Created PPDDL: effects of actions may have discrete outcome probs & probabilistic initial state literals Domains: 7 for deterministic track (2 replays from IPC-3), 8 for prob. track Competitors: 19 deterministic planners: Optimal –BFHSP, CPT, HSP*-a,Optiplan, SemSyn, SATPLAN-04, TP4-04 Sub-optimal - CRIKEY, FAP, Fast Downward, Fast Diag. Downward, LPG-TD, Macro-FF, Marvin, Optop, P-MEP, Roadmapper, SGPlan, Tilsapa, YAHSP, FF, MIPS, & LPG from IPC-3 also run where capable. 10 probabilistic planners: mGPT, Purdue-Humans, Classy, FF-rePlan, NMRDPP, ProbaPOP, FCPlanner, CERT

8. Overview of the International Planning CompetitionsFifth IPC: 2006 English Lakes, U.K Focus: 2 major tracks- Deterministic: fully deterministic & observable (previously called "classical" planning).  Subtracks- Optimal Satisficing (sub-optimal) Non-deterministic: 2 subtracks: 1) Conformant planning: nondeterministic problems for which planners must produce a contingency-safe and linear solution. 2) Probabilistic planning: Focus on real-time decision making not complete policies. PDDL extended for both tracks: Deterministic –Derived predicates, Limited exogenous events Probabilistic –Created PPDDL: effects of actions may have discrete outcome probs & probabilistic initial state literals Domains: 7 for deterministic track (2 replays from IPC-3), 8 for prob. track

9. IPC Goals“analyzing and advancing the planning state-of-the-art” Visibility for the competition’s focal problems/ algorithms/tracks will likely increase across the diverse & broad scheduling community • Relative maturity of planning / scheduling • Planning advances have been driven more by perceived need to expand model expressiveness • Scheduling advances have come more from imminent and immediate applications

10. IPC Goals“providing new benchmarks and a representation formalism to aid planner comparison and evaluation” There are many existing scheduling benchmarks –But tend to be ‘classical’ in that they don’t include breadth of constraints found in practical apps. There is no existing broadly recognized scheduling domain description language (i.e. no ‘SDDL’) • Like PDDL creation of an SDDL may facilitate comparisons of scheduling paradigms across diverse problems • Differences: modeling of resources as 1st class objects Shed light on relative strengths of planning/scheduling approaches to similar problems: Translate the several IPC ‘scheduling’ domains into SDDL….

11. IPC Goals“emphasizing new research issues and directions” • tasks with uncertain durations and/or outcomes • scheduling/ rescheduling to keep pace with execution • distributed or multi-agent scheduling • trade-offs in schedule robustness vs. quality/utility

12. IPC Goals“promoting applicability of planning technology” (arguably) Demonstrating and promoting applicability is a larger concern at this time for planning than scheduling

13. IPC Goals“disseminating as much performance data as possible to the community” IPC experience: few competing systems are effective, let alone dominate, across many domains and tracks --Even more likely to be the case for scheduling systems, at least in early competitions. Must motivate competition to generate this data Performance visibility improves with a successful competition

14. The tally: So expect recruitment for Scheduling Competition committees to get underway shortly (!)