Mohan Sridharan Texas Tech University

1. Planning to See: Hierarchical POMDPs for Joint Planning of Information Processing and Sensing Actions Mohan Sridharan Texas Tech University Jeremy Wyatt and Richard Dearden, University of Birmingham, UK Peter Stone (UT Austin) and Ian Fasel (U Arizona) International Cognitive Vision Workshop

2. The Big Picture • Grand challenge: • Deploy large teams of autonomous robots in the real-world, where they can collaborate with humans over an extended period of time. • Targets: • Adapt focus of sensing and computation to the changes in environment, tasks and team structure. • Leverage minimal real-time feedback from human collaborators. International Cognitive Vision Workshop

3. Motivation • High-fidelity sensors available at moderate costs. • Cameras, laser range finders, tactile sensors etc. • Mobile robots used in real-world applications. • Manual calibration, sensitive to change. • Integrated systems. • Robots that sense and interact with the surroundings. • Uncertainty in sensing/actuation. • Full potential of mobile robots not yet exploited! International Cognitive Vision Workshop

4. Desiderata and Challenges • Real-time/Dynamic response. • Computational constraints. • Reliability. • Noisy sensors and actuators. • Autonomous operation. • Manual sensor calibration, Sensitivity to changes. • Challenges: • Autonomous learning and adaptation. • Processing management. • Multi-robot collaboration. International Cognitive Vision Workshop

5. Sensor Processing Management • Large amount of raw data from multiple sensors. • Several sophisticated processing algorithms. • Processing can vary with the task and environment. • Autonomously tailor processing to the task at hand. • Pose sequencing of sensor processing operators as probabilistic sequential decision making (POMDPs). International Cognitive Vision Workshop

6. Related Work • Planning visual operations: • Image interpretation (POMDP: Darrell 97, MDP: Li et al. IIS03), Image processing (Borg: Clouard et al. PAMI99, Astronomy: Chiens et al. ProcSoft00) • Classical Planning architectures: • Layered architecture (Brooks, RA86), SOAR(Anderson et al. PR04). • Observation Planners: • C-BURIDAN (Draper et al. UAI94), PKS (Petrick and Bacchus, ICAPS04), CP (Brenner and Nebel, PCAR06). • Active Sensing: • Sensor placement (Kreucher et al., SP05), Gaze sequencing (Vogel and de Freitas, ICRA08), Submodularity (Krause et al., JMLR08). • POMDP approximations: • Preference elicitation (Boutilier et al., AAAI02), Q-MDP (ICML95) • Hierarchical planning: • MAXQ (Dietterich, ICML98), Nursebot (Pineau et al. RAS03). • Imposing/learning structure in POMDPs: • FSC (Hansen et al. ICAPS03), DBN (Toussaint et al. UAI08). International Cognitive Vision Workshop

7. Example: Visual Processing Management • Robots and humans jointly reason about objects. • Is there a blue submarine? Where is the red square? • Features: • State not observable, action modifies agent’s belief. • Non-deterministic actions: action effects not reliable. • Computational complexity. • Approach: • plan visual processing:where to look? How to process? International Cognitive Vision Workshop

8. Talk Outline • Visual Processing Management: • Planning using POMDPs – simple example. • Exploit structure – imposing hierarchy (HiPPo). • Other features: occluding objects, approximation error. • Summary and Future Work. International Cognitive Vision Workshop

9. Talk Outline • Visual Processing Management: • Planning using POMDPs – simple example. • Exploit structure – imposing hierarchy (HiPPo). • Other features: occluding objects, approximation error. • Summary and Future Work. International Cognitive Vision Workshop

10. Communication SA Communication SA Communication SA Communication SA Communication Vision Planning Manipulation Binding Coordination Communication SA Communication SA Tabletop Scenario • Focus: • Robot and human jointly manipulate and converse about objects on a tabletop. • Back to blocks-world?  • Dynamic response, reliability. • Components: • Objects segmented to form ROIs, speech commands. • Bind information across different modalities like speech, vision, touch. • Katana arm. International Cognitive Vision Workshop

11. POMDP for one ROI – • States: Cartesian product of individual state vectors. • Actions: visual+”special”. • Observations: red, green, blue, circle, triangle, square, empty, unknown. International Cognitive Vision Workshop

12. POMDP for one ROI – • Transition function. • Observation function. • Reward specification. • Drawback:Exponential state explosion with several ROIs and actions – 25n + 1 states for n ROIs with just two visual actions!! • Approach: Exploit the existing structure. International Cognitive Vision Workshop

13. Talk Outline • Visual Processing Management: • Planning using POMDPs – simple example. • Exploit structure – imposing hierarchy (HiPPo). • Other features: occluding objects, approximation error. • Summary and Future Work. International Cognitive Vision Workshop

14. Hierarchical POMDP Formulation • Proposed solution: Hierarchical Planning in POMDPs – HiPPo  • InfoMax POMDP to chose area of scene. • One POMDP for planning the processing actions on each ROI. • Higher-level POMDP to choose one of the LL-POMDPs at each step. International Cognitive Vision Workshop

15. HiPPo – LL Formulation • Operates on a single ROI. • Key points: • Observation functions learned. • Transition function is an identity matrix, except for special actions and actions that change the state. • Reward function learned. Relative costs and run-time complexity. International Cognitive Vision Workshop

16. HiPPo – HL Formulation • HL-POMDP: • State space: object presence in different combinations of regions. • Action ui means process ROI Ri. • FRi means desired object found in Ri. • Key points: • Observation functions and costs derived from the policy trees of LL-POMDPs. • LL-POMDPs are black boxes that return definite labels (not belief densities). International Cognitive Vision Workshop

17. Estimating OH and RH • Parse LL policy tree, conditioned on the HL state. . . . International Cognitive Vision Workshop

18. Illustrative Example • Consider the scene with two ROIs extracted. • Query: Where are the blue circles? • Available operators: Color, Shape, SIFT. International Cognitive Vision Workshop

19. Example – Where are the Blue Circles? International Cognitive Vision Workshop

20. Example – Where are the Blue Circles? International Cognitive Vision Workshop

21. Example – Where are the Blue Circles? International Cognitive Vision Workshop

22. Talk Outline • Visual Processing Management: • Planning using POMDPs – simple example. • Exploit structure – imposing hierarchy (HiPPo). • Other features: occluding objects, approximation error. • Summary and Future Work. International Cognitive Vision Workshop

23. Overlapping ROIs – Introduction • ROIs often contain multiple objects: • Region-split is an useful operator. • Planning through splits is problematic: • Splitting a region changes the state space – no longer the same POMDP! • Different split action for each basic operator, e.g. split-color: • Each action is a split plus the operator applied to all new regions. International Cognitive Vision Workshop

24. Region Splitting – Planning • Assume a geometric distribution over the number of regions created (2,3,4…). • Assume feature has n possible levels. • Calculate likelihood of generating an interesting ROI: • Intuition: after splitting a ROI, if there is a ROI with the property we want, that is the only interesting ROI. International Cognitive Vision Workshop

25. Region Splitting – Planning • Make the split operator (e.g. split-color) feasible only when ‘multiple’ feature values exist. • Use the transition function: • After split, need to recreate HL-POMDP: • Add to action costs. International Cognitive Vision Workshop

26. Region-split Example – Step 1 International Cognitive Vision Workshop

27. Region-split Example – Step 2 International Cognitive Vision Workshop

28. Region-split Example – Step 3 International Cognitive Vision Workshop

29. Region-split Example – Step 4 International Cognitive Vision Workshop

30. Region-split Example – Step 5 International Cognitive Vision Workshop

31. Experimental Setup • The HL-POMDP and LL-POMDPs are query-specific. • LL-POMDPs for each ROI written in ZMDP format. • Solved using point-based VI [Smith & Simmons, 05] • Generate observation probabilities, costs for HL-POMDP, which is solved in a similar manner. • Experiments on a physical robot interacting with a human on a tabletop. • Performed ~60 queries, multiple trials of each. • Occurrence: “Is there a red cup in the scene?” • Location: “Where is the blue circle?” • Property: “What colour is the box?” • Global Scene: How many green squares are there?” International Cognitive Vision Workshop

32. A ‘Modern’ Planner • Continual Planning (CP) [Brenner & Nebel, 06] provides a solution to this problem. • CP allows actions with non-deterministic effects: • Use these to represent information gathering actions. • Assumes that actions are reliable. • At plan time, the planner asserts that the effect it wants will actually occur: • If the effect doesn’t occur at execution time, replan. • Intuitively: build a contingent plan, but replanning ensures you only build the branches you need. International Cognitive Vision Workshop

33. Joint POMDP vs. HiPPo International Cognitive Vision Workshop

34. Comparison of Planning Time International Cognitive Vision Workshop

35. Comparison of Planning + Execution Time International Cognitive Vision Workshop

36. Reliability Analysis • Modern planners that do not model uncertainty cannot do much better than naïve visual processing. • HiPPo exploits models of action outcomes to provide higher reliability. International Cognitive Vision Workshop

37. Talk Outline • Visual Processing Management: • Planning using POMDPs – simple example. • Exploit structure – imposing hierarchy (HiPPo). • Other features: occluding objects, approximation error. • Summary and Future Work. International Cognitive Vision Workshop

38. Approximation Error Analysis • Policy-caching speeds up computation of models and policies (LL). • Action costs a function of ROI-size: • Approximation errors in value estimation. International Cognitive Vision Workshop

39. Approximation Error – Theoretical Bound • Approximation error depends on discretization. • Experimental values < 0.1 * theoretical bounds. International Cognitive Vision Workshop

40. Related Work • Planning visual operations: • Image interpretation (POMDP: Darrell 97, MDP: Li et al. IIS03), Image processing (Borg: Clouard et al. PAMI99, Astronomy: Chiens et al. ProcSoft00) • Classical Planning architectures: • Layered architecture (Brooks, RA86), SOAR(Anderson et al. PR04). • Observation Planners: • C-BURIDAN (Draper et al. UAI94), PKS (Petrick and Bacchus, ICAPS04), CP (Brenner and Nebel, PCAR06). • Active Sensing: • Sensor placement (Kreucher et al., SP05), Gaze sequencing (Vogel and de Freitas, ICRA08), Submodularity (Krause et al., JMLR08). • POMDP approximations: • Preference elicitation (Boutilier et al., AAAI02), Q-MDP (ICML95) • Hierarchical planning: • MAXQ (Dietterich, ICML98), Nursebot (Pineau et al. RAS03). • Imposing/learning structure in POMDPs: • FSC (Hansen et al. ICAPS03), DBN (Toussaint et al. UAI08). International Cognitive Vision Workshop

41. Summary and Future Work • Visual processing management posed as a planning problem: • Hierarchical planning framework: not a general-purpose planner! • Focus on processing management, not to improve individual operators. • HiPPo models uncertainty: • Reliable and efficient performance. • Learns required model parameters. • Functional architecture: • Automate belief propagation. International Cognitive Vision Workshop

42. Summary and Future Work • Lots of other operators to integrate: Viewpoint Change, Zoom etc • State and action hierarchies. • Learn hierarchy? • From image analysis to scene analysis: • Should I look somewhere else or analyse the ROIs I have? • Relational queries? Action queries? • Object interaction: • Push, poke object to learn object (epistemic) affordances. • Other domains and applications. International Cognitive Vision Workshop

43. That’s all folks  International Cognitive Vision Workshop