Modern Planning Techniques

1. Modern Planning Techniques Jőrg Hoffmann Albert-Ludwigs-University Freiburg, Germany

2. Popular Modern (since 1995) Domain-Independent Classical (mostly STRIPS and ADL) Planning Techniques Jőrg Hoffmann Albert-Ludwigs-University Freiburg, Germany

3. Foreword “Pre-historical” faster « Optimal  1995 “Graphplan-based” « Sub-optimal  2000 “Heuristic-based”

4. PLANSAT vs. ID PLANMIN • PLANSAT: Is a given planning task solvable? • PLANMIN: Is a given planning task solvablewithin d steps? • Iterative Deepening PLANMIN: decide PLANMIN for d = 1, 2, 3, … until plan is found • Note 1: PLANSAT isPSPACE-complete. PLANMIN isNP-complete if d is in unary encoding. (for propositional STRIPS [Bylander 94]) • Note 2: ID PLANMIN requires proving unsolvability for all d < k if shortest plan has length k

5. Modern Planners ID PLANMIN Graphplan [Blum&Furst,95] Blackbox [Kautz&Selman,96] IPP [Koehler et.al.,97] SGP [Anderson&Weld,98] STAN [Fox&Long,98] TGP [Smith&Weld,99] R-IPP [Koehler&Hoffmann,99] TPSYS [Garrido et.al.,02] TP4 [Haslum&Geffner,01] Greedy PLANSAT LPG [Gerevini et.al.,02] Unpop [McDermott,96] HSP-r [Bonet&Geffner,99] HSP [Bonet&Geffner,98] HSP2 [Bonet&Geffner,00] GRT [Refanidis&Vlahavas,99] FF [Hoffmann,00] Mips [Edelkamp,00] AltAlt[Kambhampati et.al.,00] STAN4 [Fox&Long,00] RePOP [Nguyen&Kambhampati,01] SAPA [Do&Kambhampati,01] Metric-FF [Hoffmann,02] VHPOP [Younes&Simmons,02]

6. We Observe: • Graphplan and its successors basically do ID PLANMIN, the ``heuristic approaches’’ do a greedy form of PLANSAT • The former technique must be able to prove unsolvability efficiently, the latter technique not, in solvable examples • By doing ID PLANMIN, one finds an optimal (provably shortest) plan • Proving unsolvability requires very different techniques than, greedily, trying to find a plan

7. Outline • Modern Planners (since 1995) • Part I: ID PLANMIN • Reachability • Regression Search • Non-directional Search • Part II: Greedy PLANSAT • Ignoring Delete Lists • Local Search Topology • Numeric and Temporal Extensions • Summary

8. ID PLANMIN • For increasing bound d = 1,2,3 ... prove task unsolvable; stop when (optimal) plan is found • Two key notions: • Parallel action application allows us to find plans within fewer decision iterations • Admissible heuristics allow us to safely prune (potentially large) fragments of the search space

9. Parallel Action Application • Two actions can be applied „in parallel“ if their outcome does not depend on the order of execution • Drive T1 A B, Drive T2 C D • Actions a and a'interfere if: • del(a)  (add(a')  pre(a')) \neq , or • del(a')  (add(a)  pre(a)) \neq  • Allowing sets of non-interfering actions at each step in the plan can make plans shorter => potentially fewersearch iterations

10. Admissible Heuristics • A heuristic is a function h:S N0 that estimates the remaining cost of a search state sS: the cost h*(s) of the cheapest path from s to a solution state h is admissible if h(s) is always  h*(s) • If we are searching with a cost bound d, and h is admissible, we can prune s if g(s) + h(s) > d where g(s) is the cost of the path from s0 to s (this is known as Branch-and-Bound)

11. Part I: ID PLANMIN • k-Reachability • Value Iteration • k-Planning Graphs • Regression Search • Reachability Pruning • Heuristic Updates • Systems, and Open Questions • Non-directional Search • To Encode… • …or Not To Encode • Systems, and Open Questions

12. Reachability Generally: „reachability“ concerns things that have to do with the difficulty of reaching something (eg. a set of facts) from the initial state (the opposite notion is „relevance“, which concerns things to do with reaching something from the goals) • There are admissible approximations of the distance of a fact set to the initial state • By r*(F), for a fact set F, we denote the minimal number of parallel action applications needed to achieve F

13. k-Reachability Approximate the difficulty of reaching F by the difficulty of reaching the hardest k-subset of F • Need a notation: • given a fact set F, a fact set F' is the regression of F through A, F‘ = reg(F,A), if A is a (non-redundant) action set s.t. the parallel application of A achieves F, and F' is the union of aA‘s preconditions • reg(F): all pairs (F',A) s.t. F' is the regression of F through A • we allow Noop-f actions: f => f (convenient trick to model non-affected facts)

14. k-Reachability, ctd. rk(F) = 0 if F  I rk(F) = min(F',A)reg(F) 1+rk(F') if |F|k, FI rk(F) = maxF'F,|F'|=k rk(F') if |F| > k • [Haslum&Geffner 00] • We have rk(F)  r*(F) • Polynomial in task size but exponential in k • So far only r1 and r2 have been made effective

15. Value Iteration • Compute rk values for all F, |F| = k: • For all F, |F| = k: set rk(F) := 0 if F  I, else rk(F) :=  • for all F',A,F, |F| = k, s.t. (F'‚A)reg(F) and rk(F')<: • setrk(F) := min( rk(F), 1+rk(F') ) • iterate until no more changes occur • rk for F,|F| > k ismax. rk of F‘s k-subsets • [Haslum&Geffner 00] • Here, breadth-first style updates; any order of value updates possible

16. Example, k = 1 • Logistics-style task: T P1 P2 • Value iteration fragment: • at(T Left), at(P1 Left), at(P2 Right): 0 • Load T P1 Left, Drive T Left Right • in(P1 T), at(T Right): 1

17. Example, k = 2 T P1 P2 • at(T Left), at(P1 Left), at(P2 Right): 0 • Load T P1 Left and Drive T Left Right interfere! • Load T P1 Left, Noop-at(T Left) • {in(P1 T), at(T Left)}: 1 • Noop-in(P1 T), Drive T Left Right • {in(P1 T), at(T Right)}: 2

18. Value Iteration, ctd. • How to represent rk? • Naive approach: keep one integer per k-tuple • Better: represent  as absence, ie. keep only the reached tuples in memory • Better (?): see next slide

19. k-Planning Graphs Fi and Ai --- „fact- and action-layers“ eFi and eAi --- sets of „exclusion relations“ • F0 := I, A0 := actions whose precondtions are in F0 • eF0 := , eA0:= all {a, a‘} in A0 that interfere • Fi+1 := facts that appear in adds of Ai • Ai+1 := actions whose preconditions appear in Fi+1, without a subset in eFi+1 • eFi+1:= all minimal k-subsets of Fi+1 s.t. any achievingaction set Ain Aihas a subset in eAi • eAi+1:= all {a, a‘} in Ai+1 that interfere, and all minimal k-subsets of Ai+1 s.t. the preconditions have a subset in eFi+1

20. Example, k = 2 T P1 P2 • 0: at(T Left), at(P1 Left), at(P2 Right) • Load T P1 Left and Drive T Left Right interfere! • Noop-at(T Left), Load T P1 Left, Drive T Left Right • 1: at(T Left), in(P1 T), at(T Right) • in(P1 T), at(T Right) exclusive! • Noop-in(P1 T), Drive T Left Right • 2: in(P1 T), at(T Right)

21. k-Planning Graphs, ctd. • Planning graphs are a version of breadth-first value iteration where rk(F) is represented as the index i of the first layer in which F appears without an exclusion relation in eFi • This representationis useful for search implementation, see later • Complicated explanation of a simple thing? • Maybe, but understanding matters from a general point of view is useful for seeing open directions (some more on these later)

22. .bbl • P. Haslum, H. Geffner, Admissible Heuristics for Optimal Planning. AIPS’00. • A. Blum, M. Furst, Fast Planning Through Planning Graph Analysis. IJCAI’95. AIJ’97. • R. Brafman, Reachability, Relevance, Resolution and the Planning as Satisfiability Approach. IJCAI’99. JAIR’01. • P.Haslum, H.Geffner, Heuristic Planning with Time and Resources. ECP’01.

23. SearchSchemes • Search Scheme (S, s0, br, Sol): • space of all search states S • start state s0S • branching rule br: S  ((S)) (maps a state to a set of branching points; each branching point is a set of successor search states) • solution states Sol S (s.t. below each branching point there are the same solutions) (s.t. a plan can be easily extracted from s0 ...  sol) • Often:transition-cost function c:S x S  N0, andpath-cost function g(s0 ... s)(summed up transition costs) Search scheme does not specify search algorithm!

24. (Bounded) Regression Search • Search Scheme (S, s0, br, Sol) where • S == space of all fact sets F(at a time 1  t  d) • s0 == original goal set (at d) • br(s) == a single branching point, containing F‘s regression through all possible action sets (at t – 1) • Sol == fact sets contained in original initial state • c(s,s'): uniform 1, cost bound d bound on plan length Without pruning technique, wemust explore all d-step possibilities to decide about solvability

25. Reachability Pruning • Heuristic h: S  N0; admissible: h(s)  h*(s) • Given an admissible heuristic h, prune a search state s if g(s) + h(s) > d • Here: • search state is subgoal F after g(F) regression steps from the goal • remaining cost is F‘s distance r*(F) from I • admissible heuristic is rk(F) The number of steps made on the path to F, plus the reachability estimate to F, exceeds d => need not consider this path to F

26. Reachability Pruning, ctd. • Also known as: g(s) + h(s) > d (d-t) + rk(F) > d rk(F) > t ie. cost bound is exceeded iff F can‘t be reached at t • The reason why we know that F is „ at t “ is that we start from the bound d • Planning graph implementation: • „ rk(F) > t “ is represented as „ F is not in Ft, or F contains an exclusion in eFt “ • such F are never even generated as resp. action set will not be in At, or contain an exclusion in eAt

27. Heuristic Updates • If search beneath sfailed then: The real remaining cost h*(s) must be greater than d - g(s) => We have h(s)  d – g(s) < h*(s) => We can update h(s)s.t. h(s) > d – g(s) => When h is a function into N0: h(s) := d – g(s) + 1 (this is known as „transposition tables“)

28. Heuristic Updates, ctd. • If search beneath F fails then: rk(F) := d - g(F) + 1 • By transitivity of reachability: rk(F') := d - g(F') + 1 for all F' F • Planning graph implementation: • „Memoization“: store F at t(= d – g(F))as new exclusion into eFt • Originally, restricted use of these „higher-order“ exclusions(|F|might be greater than k)

29. Systems • Graphplan: depth-first bounded regression search with r2 heuristic computed by and represented as 2-planning graph, heuristic updates represented asexclusion learning („memoization“) • Numerous extensions, eg. STAN, IPP, SGP, TGP, R-IPP, TPSYS, ... • HSPr*: depth-first bounded regression search with r2 heuristic computed by value iteration in a naive integer table, heuristic updates in a limited-size transposition table HSPr* can easily be modified to use heuristic search algorithms such as A*, best-first, ... (see Part II)

30. Open Questions • Can (k>2)-reachability be made effective? • For what kinds of domains? • By „targeted“ use of value iteration/exclusion reasoning? • Other admissible reachability heuristics? • Based on relaxations (see Part II) other than k-subsets? • Ignoring delete lists? Ignoring preconditions? • Use k-relevance and search forwards instead? • [Brafman 99,01; Parker 99]

31. .bbl • A. Blum, M. Furst, Fast Planning Through Planning Graph Analysis. IJCAI’95. AIJ’97. • P. Haslum, H. Geffner, Admissible Heuristics for Optimal Planning. AIPS’00. • P. Haslum, H. Geffner, Heuristic Planning with Time and Resources. ECP’01. • A. Reinfeld, T. Marsland, Enhanced Iterative-Deepening Search. IEEE Trans. On Pattern Analysis and Machine Intelligence. • A. Junghanns, J. Schaeffer. Domain-Dependent Single-Agent Search Enhancements. IJCAI’99. • J. Hoffmann, J. Koehler, A New Method to Index and Query Sets, IJCAI’99. • S. Kambhampati, Planning Graph as a (Dynamic) CSP: Exploiting EBL, DDB, and other CSP Search Techniques in Graphplan. JAIR’00.

32. .bbl, ctd. • D. Long, M. Fox, Efficient Implementation of the Plan Graph in STAN. JAIR’99. • J. Koehler, B. Nebel, J. Hoffmann, Y. Dimopolous, Extending Planning Graphs to an ADL Subset. ECP’97. • C. Anderson, D. Smith, D. Weld, Conditional Effects in Graphplan. AIPS’98. • J. Koehler, Planning under Resource Constraints. ECAI’98. • D. Smith, D. Weld, Temporal Planning with Mutual Exclusion Reasoning. IJCAI’99. • R. Brafman, Reachability, Relevance, Resolution and the Planning as Satisfiability Approach. IJCAI’99. JAIR’01. • E. Parker, Making Graphplan Goal-Directed. ECP’99.

33. Non-Directional Search • Search Schemes(S, s0, br, Sol) where • S == space of all sets of commitments • s0 == empty set • br(s) == branching points are the open decisions, each decision contains a set of possible additional commitments • Sol == consistentsets of commitmentswith no open decisions • c(s,s'): here,uniform 0; maybe 1 if commitment introduces an action • Consistency checks correctness of plan, via constraints • Constraint propagation: state can be pruned if inconsistency occurs (related to branch-and-bound, see later)

34. Branching on Actions, Bound d • Search Scheme (S, s0, br, Sol) where • S == space of all sets of commitments of the form execute/don‘t execute action aat time 0  t  d - 1 • s0 == empty set • br(s) == branching points are the un-committed action/time pairs, each point contains in/outcommitments • Sol == consistentcommitments for all action/time pairs • c(s,s'): here, uniform 0; maybe 1 if commitment is „execute“ a at t • Constraints check that goals/preconditions are achieved, and that no interfering actions are executed in parallel

35. Non-Directional Search, ctd. • What‘s the advantage? • unlike in regression (or forward) search, we have a choice of branching points • often, looking at few branching points (the most „critical“ ones) suffices to decide solvability • „branching heuristics“ become (one) important technique • Remember: with an explicit reference to all time points (up to the bound), we‘re in NP... We can encode into other NP-problems, and use solution methods for these: eg., SAT-solvers

36. SAT Encodings • One possibility: • variables a(t) for all actions a and 0  t  d – 1 • a1(d-1)  ...  am(d-1) where a1, ..., am are all achievers of a goal • a(t)  a1(t-1)  ...  am(t-1) where a1, ..., am are all achievers of a precondition of a • a(t)  a'(t) where a and a' interfere == branching on actions, constraints are clauses „Davis Putnam“ == depth-first, propagating „unit constraints“

37. Unit Propagation • (Search) commitment a(t): • bya(t)  a'(t),new commitmenta'(t) • by a(t)  a1(t-1)  ...  am(t-1), new commitment ai(t-1) if all \neq i out already • (Search) commitment ai(t): • by a1(d-1)  ...  am(d-1),new commitmentaj(d-1)if t = d, and all \neq i, j out already • by a(t)  a1(t-1)  ...  am(t-1), new commitment aj(t-1) if a(t) in, and all \neq i, j out already • by a(t)  a1(t-1)  ...  am(t-1), new commitment a(t) if all \neq i out already

38. SAT Encodings, ctd. • Many other possible encodings, in particular: compute k-reachability, include heuristic information into encoding by ways of „exclusion relations“ => more pruning power by redundant clauses => has successfully been done for 2-reachability • To encode...(has the advantage that): • many powerful techniques in existence (clause learning, randomized restarts, higher-order resolution, ...) • highly optimized implementations ready for use • progress in SAT can translate directly to progress in planning

39. …or Not to Encode „Anything that we can do within the SAT encoding, we can also do without the SAT encoding“ • Searching directly has the advantage that: • the solver „knows what it does“ • and can thus use planning-specific techniques/heuristics • Some implementations ([Baioletti et al 00; Rintanen 98]) similar to DP in existence: • branch on actions/on actions and facts • do some form of constraint propagation

40. ...or Not to Encode, ctd. • Another existing implementation: [Hoffmann & Geffner 03] 1. branch on actions 2. in every search state s, re-compute a 2-planning graph under the commitments in s 3. use a (relevance, criticality) guided branching heuristic • Note: • 2. and 3. are planning-specific techniques • corresponds to a version of branch-and-bound where the admissible heuristic to s is r2s(G) -- prune state if that is greater than d

41. Systems • Blackbox: configurable system implementing various SAT encodings (including 2-planning graph based), and incorporating various SAT solvers (including non-DP ones) • DPPlan: build 2-planning graph, branch on actions and facts, various propagation rules • „R“: compute 2-invariants, branch on actions, various propagation rules • BBG: branch on actions, re-compute r2s(G), prune node if r2s(G) > d

42. Open Questions • What are good branching heuristics in non-directional planning? • How to customise SAT techniques for the planning context? • „from left to right“: transport planning-specific techniques into the SAT solvers • „from right to left“: transport techniques used in SAT solvers to the planning context

43. .bbl • H. Kautz, B. Selman, Planning as Satisfiability. ECAI’02. • H. Kautz, B. Selman, Pushing the Envelope: Planning, Propositional Logic, and Stochastic Search. AAAI’96. • H. Kautz, B. Selman, Unifying SAT-based and Graph-based Planning. IJCAI’99. • M. Ernst, T. Millstein, D. Weld, Tradeoffs in Automatic SAT-Compilation of Planning Problems. IJCAI’97. • M. Davis, G. Logemann, D. Loveland, A Machine Program for Theorem Proving. Communications of the ACM’62. • E. Giunchiglia, A.. Massarotto, R. Sebastiani, Act, and the Rest will Follow: Exploiting Determinism in Planning as Satisfiability. AAAI’98.

44. .bbl, ctd. • M. Baioletti, S. Marcugini, A. Milani, DPPlan: an Algorithm for Fast Solutions Extraction from a Planning Graph. AIPS’00. • J. Rintanen, A Planning Algorithm not Based on Directional Search. KR’98. • J. Hoffmann, H. Geffner, Branching Matters: Alternative Branching in Graphplan. ICAPS’03. • J. Rintanen, Numeric State Variables in Constraint-based Planning. ECP’99. • R. Brafman, Reachability, Relevance, Resolution and the Planning as Satisfiability Approach. IJCAI’99. JAIR’01. • H. Geffner, Planning as Branch and Bound and its Relation to Constraint-based Approaches. Draft, available from www.